Modern CPUs hit a memory wall — and here's why data movement now matters more than teraFLOPS

For decades, computing performance was easy to summarize with a few headline specifications. CPUs were sold on clock speed and core count, GPUs on shader throughput and later teraFLOPS, memory on frequency, and storage on sequential transfer rates. Those numbers still have value, but modern processors have become so extraordinarily capable of performing arithmetic that another problem increasingly dictates how much of that theoretical performance can actually be used: getting data where it needs to go, when it needs to be there.

A modern CPU core can execute multiple instructions every clock cycle. A high-end GPU contains thousands of arithmetic units operating in parallel. AI accelerators push matrix multiplication throughput into the petaFLOP range. Yet none of these units can calculate anything useful without operands, instructions, textures, geometry, model weights, intermediate results, or some other form of data.

This creates a fundamental imbalance. Arithmetic throughput has scaled enormously over the years, but moving data is comparatively much more expensive in terms of latency, bandwidth, energy, and physical chip resources. As a result, many modern workloads spend a surprising amount of effort simply trying to keep execution units fed.

That does not mean that every workload is "memory-bound." Far from it, in fact. Performance can still be constrained by compute throughput, instruction dependencies, branch prediction, synchronization, software overhead, storage, networking, or numerous other factors. What has changed is that understanding memory performance has become a critical factor in assessing real-world performance. The gap between raw compute capability and the ability to supply data to those compute units is widening, making memory architecture, cache hierarchies, and data locality key design considerations for modern processors and systems.