Prince Mario-Max Schaumburg-Lippe: Cerebras Maps a 3-Trillion-Parameter Model Across 59 Wafers

How do you serve a model with about 3 trillion parameters without spending half your time hauling its weights around? Cerebras answered that question in a technical post this week, and the answer is refreshingly physical. You spread the model across 59 of its WSE-3T wafers, park the weights on the chips, and let only the small stuff travel.

The working example is Kimi K3, the open-source model from Moonshot AI with roughly 3 trillion parameters across 93 layers. Cerebras is candid that K3 serves as an illustrative example for the layout. That honesty is welcome. What the company published is a deployment recipe, worked out in public with the arithmetic shown, and the recipe is the interesting part.

Start with one wafer and one hard number

A single WSE-3T wafer holds 44 GB of on-chip memory. That sounds roomy until you price a frontier model. Trillion-parameter systems need terabytes for their weights alone, before a single user request arrives. No one chip holds that. So the question stops being how big your chip is and becomes how sensibly you can divide the work.

Cerebras uses 4-bit weight quantization, the same approach as the Hugging Face implementation, plus an 8-bit KV cache. At 4 bits per weight, 3 trillion parameters come to about 1.5 terabytes of raw payload. Divide that by 93 layers and each layer lands near 16 GB. Sixteen fits inside 44 with room to spare. That spare room is where the design gets clever.

Readers who followed yesterday’s launch coverage of Kimi K3.1 Pro and its open weights promised for November 4 will see why this pairing makes sense. Big open models need serving plans that ordinary teams can actually reason about, and a layer-by-layer map is about as legible as infrastructure gets.

Two kinds of layers, two kinds of wafers

Kimi K3 mixes 69 linear-attention layers with 24 dense layers, and they behave differently. A dense layer stores keys and values for every token in every active request, so it needs cache space. Cerebras gives each dense layer a full wafer and leaves the remaining memory for that cache. A linear-attention layer keeps only a small, fixed-size state per request, whatever the context length. Two of those fit on one wafer, at 32 GB combined.

Do the division and the headline number falls out. The 69 linear layers need 35 wafers. The 24 dense layers need 24. Total, 59 wafers. No layer gets split across chips, which matters more than it might sound. A layer that lives whole on one wafer never has to pause mid-calculation to fetch a missing piece of itself.

An assembly line, not a waiting room

The wafers connect as a pipeline. Wafer 1 runs the first layers and hands its activations to wafer 2, which runs the next layers and hands on again, until the last stage produces the next token. Activations cross between wafers over RDMA over Ethernet at roughly 2 microseconds a hop. The weights stay put. Only the intermediate results move, and they are small.

Your first worry might be idle time. If a token walks the whole line, does each wafer sit around waiting for its turn? No, because the line stays full. While stage 3 works on a token for request A, stage 2 is already working on request B and stage 1 on request C. Each stage can also handle a micro-batch, a small group of requests at once. Smaller micro-batches make each individual request faster. Larger ones raise total throughput. Operators get to choose their trade, which is how it should be.

Cerebras puts the resulting speed at 1,000 to 2,000-plus tokens per second, using an assumed average request size of 256,000 tokens for the example. Long requests are not an edge case in this design. They are the planning assumption.

Concurrency is a memory question

How many requests can run at once? On this architecture, the answer sits in the dense-layer wafers. Every request in flight needs its own KV cache on each of those wafers, so the memory left over after the 16 GB layer sets the ceiling. Requests do not wait for one another to finish. They stream through the pipeline continuously, which is a different feel from batch processing, where your answer waits for the group photo.

Cerebras also names the yardstick it prefers, and it is a fair one. Token throughput per unit of power, measured at the same output speed, tells you more than a raw speed figure alone. The company does not publish that number for this exact layout in the post, and saying so plainly keeps the claim honest. A recipe with stated assumptions beats a benchmark with hidden ones.

The same week brought a practical cousin of this story in transport, where Waymo started driverless rides at Las Vegas airport. Different field, same pattern. The winners lately are the teams that turn a hard system into a routine service.

What to take away if you run models

Three practical points travel well beyond Cerebras hardware. First, keep weights still and move activations; data movement is usually the tax, not the arithmetic. Second, match your layout to your layer types instead of treating a model as one uniform block – the 35-plus-24 split exists because linear and dense layers want different things. Third, plan concurrency from memory outward, because cache space, not compute, will cap you first.

The approach also scales past 3 trillion parameters, at least on paper. Each added wafer contributes both compute and on-chip memory, so a larger model mostly means a longer pipeline. Whether every operator needs a model this size is a separate question, and a healthy one to ask. But teams experimenting with open models now have a concrete map for the day they do.

There is a quiet confidence in publishing the math instead of just the result. Anyone can check the 16 GB per layer, recount the wafers, and argue with the assumptions. That is how infrastructure knowledge spreads. Kimi K3 may be the example, but the lesson is portable – and portable lessons are the ones that end up mattering.

Also published on: