HMTT v3: HMTT for DDR3

Features

  • Supports DDR3 UDIMM/RDIMM/LRDIMM Interface on multiple platforms, e.g., Intel, AMD, Loongson
  • Be able to collect full system off-chip memory traces with semantic information, including commercial applications
  • Better signal integrity to support higher memory frequency signal acquisition
  • Works at DDR3-800 properly, partly supports  DDR3-1600
  • Supported memory size can be up to 512GB
  • Uses PCI-Express cable to transport trace data, the peak bandwidth can achieve 8Gbps
  • Uses 5-disk hardware SSD RAID, providing at about 2000MB/s IO write bandwidth and 5TB storage space for offline memory trace
  • Cooperating with binary instrument software toolkit, we can further track memory traces with high level events
  • Supports linux 3.10.93
  • And more important: It is in use now!

Framework

  • Memory Trace Board (MTB)
  • Kernel Synchronization Module (KSMod)
  • Trace packets capture, storage and analysis Toolkit

[framework]

Hardware Design

  • Power supply
  • DDR3 gold-finger
  • DDR3 register
  • 10 GE PHY
  • PCI-Express external cable gen2 x4
  • Xilinx Virtex-6 FPGA
  • Multi-board synchronization module (optional)

The FPGA Physical Block Diagram

[fpga]

HMTT Hardware Board

[board]
HMTT3 illustrate
HMTT 3.0
A working photo for collecting trace from a Intel Xeon E5 V2 Server
HMTT working in a Server

Software Design

Kernel Synchronization Module (KSMod)

  • One module is added to alter the first page’s cache attribution
  • Another module is a more important module which provides the hmtt_printk routine to collect page table information and to store the information in a kernel-user shared buffer
  • One kernel patch for pgtable.h records each update of page table
  • Another kernel patch for entry.S sends identifiers to MTB when kernel-enter and kernel-exit occur (optional)

Trace packets capture and analysis Toolkit

  • Memory Trace Board (MTB) Configuration
  • Trace Dumping, Storage and Replay