HMTT v3: HMTT for DDR3


  • Supports DDR3 UDIMM/RDIMM/LRDIMM Interface on multiple platforms, e.g., Intel, AMD, Loongson

  • Be able to collect full system off-chip memory traces with semantic information, including commercial applications

  • Better signal integrity to support higher memory frequency signal acquisition

  • Works at DDR3-800 properly, partly supports  DDR3-1600

  • Supported memory size can be up to 512GB

  • Uses PCI-Express cable to transport trace data, the peak bandwidth can achieve 8Gbps

  • Uses 5-disk hardware SSD RAID, providing at about 2000MB/s IO write bandwidth and 5TB storage space for offline memory trace

  • Cooperating with binary instrument software toolkit, we can further track memory traces with high level events

  • Supports linux 3.10.93

  • And more important: It is in use now!


  • Memory Trace Board (MTB)

  • Kernel Synchronization Module (KSMod)

  • Trace packets capture, storage and analysis Toolkit


Hardware Design

  • Power supply

  • DDR3 gold-finger

  • DDR3 register

  • 10 GE PHY

  • PCI-Express external cable gen2 x4

  • Xilinx Virtex-6 FPGA

  • Multi-board synchronization module (optional)

The FPGA Physical Block Diagram


HMTT Hardware Board

HMTT3 illustrate

A working photo for collecting trace from a Intel Xeon E5 V2 Server

Software Design

Kernel Synchronization Module (KSMod)

  • One module is added to alter the first page's cache attribution

  • Another module is a more important module which provides the hmtt_printk routine to collect page table information and to store the information in a kernel-user shared buffer

  • One kernel patch for pgtable.h records each update of page table

  • Another kernel patch for entry.S sends identifiers to MTB when kernel-enter and kernel-exit occur (optional)

Trace packets capture and analysis Toolkit

  • Memory Trace Board (MTB) Configuration

  • Trace Dumping, Storage and Replay

  • We had used HMTT3 to collect page table walk access ( All TLB miss access to page table.)