ECE Seminar - What Your DRAM Datasheets Are Not Telling You: Experimentally Understanding Read Disturbance and Power Consumption of High Bandwidth Memory Chips

High bandwidth memory (HBM) is a critical component enabling modern AI compute infrastructure. In this talk, we present our state-of-the-art DRAM testing infrastructure and recent cutting-edge experimental studies on read disturbance and power consumption in modern HBM.  First, we describe the DRAM Bender infrastructure that enabled our  discoveries as well as discoveries in other studies (e.g., the RowPress phenomenon). Second, we demonstrate that real HBM chips are vulnerable to read disturbance and that an undocumented on-DRAM-die RowHammer defense mechanism can be bypassed with a specialized access pattern. Third, we describe variable read disturbance (VRD), a recently discovered phenomenon in modern DRAM chips, including HBM chips: a DRAM row’s read disturbance threshold changes significantly and unpredictably over time, making it fundamentally difficult to identify the threshold and thus develop efficient solutions.  

Fourth, HBM can account for a substantial portion (e.g., approximately 30% for an NVIDIA H200 GPU) of the total system power budget in modern machine learning training and inference systems. We present the first comprehensive experimental characterization of power consumed by real HBM chips. Our results have important implications for HBM-based memory system design. Among 12 new findings, we highlight that a carefully-designed power-intensive memory access pattern consumes 35.4% more power than the most power-intensive JEDEC-specified memory access pattern. Thus, to account for the worst-case power consumption, a system designer should not rely solely on JEDEC-specified memory access patterns. Building on our findings, we develop a new HBM power model that predicts the power consumed by 36 real HBM2 stacks across diverse workloads (including a real LLM decoding workload) with significantly better accuracy (8.6% average error) than state-of-the-art power models (20.3% average error).

Data, sources, and paper PDFs for the described works are freely and openly available.

- DRAM Bender: https://github.com/CMU-SAFARI/DRAM-Bender, PDF:

https://arxiv.org/pdf/2211.05838, https://arxiv.org/pdf/2606.13725 

- HBM Read Disturbance:

https://github.com/CMU-SAFARI/HBM-Read-Disturbance, PDF:

https://arxiv.org/pdf/2310.14665

- Variable Read Disturbance PDF: https://arxiv.org/pdf/2502.13075

- HBM Power Characterization Artifacts: https://github.com/CMU-SAFARI/HBM-Power