Optical fibers instead of switches: a 32 TB memory pool for LLM inference cuts first-token latency 6.6x in simulation

Tens of terabytes of memory, moving at hundreds of gigabytes per second. That is what a large language model serving many users at once wants for its KV cache, the running scratchpad of intermediate values a model keeps so it does not have to re-read and re-process every word of a long conversation from scratch. According to a new preprint from a team at Marvell, no memory tier available today delivers both the capacity and the speed at the same time. Their proposed way around the problem involves rather less electronics than you might expect: a passive bundle of optical fiber with no switch in it at all.

Jing Ding, Yash Nishant, Chandrish Ambati, Jyothsna Kamati and Trung Diep posted their paper to arXiv on 29 July 2026. It has not yet been peer reviewed, so the numbers in it should be read as the authors' own measurements and models rather than independently confirmed results.

The starting point is a set of measurements the team made across GPU systems of several different generations, running various LLaMA models. Two findings frame everything that follows. First, when a model needs an old KV cache entry, fetching it back from the host machine's ordinary memory beats recomputing it, by up to a factor of 100. Recomputation means running the model's attention math over the earlier text all over again, so avoiding it is a large saving. Second, and less encouragingly, host memory only stretches so far. The team found it supports only tens of users at once when those users are holding long contexts. Past that point the cache has to start throwing things away.

That eviction is where the user actually feels the pain. Ask a chatbot a follow-up question and, if its cache for your conversation has been evicted, the system has to rebuild that context before it can produce a single word. The paper describes this as a cliff: performance is fine, then abruptly it is not.

Why a passive fiber shuffle

The obvious fix is memory pooling, and there is already a standard for it. CXL, or Compute Express Link, lets a machine reach out to memory that physically sits somewhere else in the rack and treat it as an extension of its own. On paper, a CXL pool is exactly the missing tier. In practice, the authors argue, three things get in the way of building one at terabyte scale: the latency the electrical switch itself adds, hard limits on how far the copper cables can run, and power consumption that does not scale gracefully as the pool grows.

So the team removed the switch. Their design, which they call the Marvell Photonic Fabric Memory Appliance, is a photonic-CXL hybrid: the CXL protocol still governs how hosts talk to memory, but the interconnect is optical, and what sits between the hosts and the memory is a passive fiber shuffle. "Passive" is the load-bearing word. A shuffle is just a fixed rearrangement of fiber strands that routes every host to every memory endpoint, a full crossbar with no active component in the middle deciding where anything goes. Nothing to add latency, nothing to burn power, and optics carry signals much further than copper.

The configuration they report is 32 terabytes of memory shared across 16 hosts. In emulation, their setup showed more than a 50 percent reduction in latency compared with an electrical CXL pool. In simulation of a multi-turn conversation workload, the kind where users come back with follow-up questions, they report the eviction cliffs disappearing and time-to-first-token improving by 6.6x.

It is worth being clear about what those two words mean. Emulation and simulation are not the same as a running data center. The latency result comes from hardware standing in for the real thing; the 6.6x figure comes from a model of the workload. Both are standard practice for architecture work at this stage, and both are a step away from deployed silicon.