Cleaning up the Mess - ISPASS'26 Talk by Nisa Bostanci and Haocong Luo
Why It Matters
Accurate memory‑system simulators are essential for hardware innovation; correcting these errors restores trust and ensures future research builds on reliable performance data.
Key Takeaways
- •Emulator2 benchmark misconfigured, halving memory channels versus real system.
- •Incorrect cache latency settings caused unrealistically low simulated memory latency.
- •Reproducing original results impossible due to missing source code artifacts.
- •Corrected configuration shows Emulator2 matches real hardware bandwidth and latency.
- •Authors propose four best‑practice guidelines for reliable memory‑system simulation.
Summary
The ISPASS 2026 presentation by Nisa Bostanci and Haocong Luo challenges a recent best‑paper that claimed the open‑source DRAM simulator Emulator 2 dramatically under‑performed real hardware.
The speakers identified three fatal flaws: missing source code prevented replication; the simulation used only eight DDR5 channels while the real system had sixteen, halving bandwidth; and cache latency was set to zero with excessive status registers, yielding unrealistically low latency. Similar statistical misuse was found in the DO framework.
After rebuilding the artifacts and correcting the configuration, their experiments showed Emulator 2 achieving bandwidth within 2 % of the theoretical maximum and latency rising proportionally with load—behaviour matching the physical platform. The authors also released the corrected code, data, and a new Emulator 2.1 branch, and highlighted specific graph comparisons.
The work underscores the need for transparent, reproducible simulation practices. By publishing four concrete best‑practice guidelines, the team aims to prevent misleading results that can steer memory‑system research off course and erode confidence in simulation‑driven design.
Comments
Want to join the conversation?
Loading comments...