What Kimi K3 asks of the hardware

Moonshot released Kimi K3 on July 16: 2.8 trillion parameters, a 1M-token context window, native vision, and open weights promised by July 27. The announcement is organized around benchmarks, and most of the coverage followed that structure. I found it more useful to read it from the systems side, because at this scale the architectural choices are also decisions about hardware: how many bytes have to move, over which link, how often. The announcement contains more of that information than is usual for a frontier release, and I wanted to work through what it implies before the technical report comes out.

That timing is also the main caveat. The report is not published yet, so this post is grounded in the official blog, the platform docs, the Kimi Linear paper from October 2025 (which introduced the attention mechanism K3 uses, at 48B scale), and two SemiAnalysis threads (one and two) about the hardware consequences. Where I extrapolate shapes that Moonshot has not disclosed, I say so explicitly, and the arithmetic should be read as order-of-magnitude estimates rather than measurements.

The changes K3 introduces sort naturally by which axis of scaling they address, and that is how I have organized the post. Kimi Delta Attention addresses sequence length. Attention Residuals address depth. The move to 16-of-896 expert sparsity addresses width, and the MXFP4 training format cuts across all three. The layout:

• The spec sheet, with error bars: what is confirmed, what is inferred, and how K3 compares to K2 on the dimensions that determine serving cost.
• Sequence length: KDA and the 3:1 hybrid: what a fixed-size attention state changes when the KV cache is the thing that grows with context, and the prefix-caching problem that linear attention introduces.
• Depth: Attention Residuals: a small change to the residual stream with a claimed 25% training-efficiency gain, and where its cost actually lands.
• Width: 896 experts and the interconnect: why serving K3 requires spreading experts across many GPUs, what that costs in communication, and how the training side deals with expert imbalance.
• Numerics: training in MXFP4: what quantization-aware training from the SFT stage changes compared to quantizing after the fact.
• The serving stack: Mooncake, prefix-cache economics, and how the pricing reflects the architecture.
• What lands by July 27: what the open weights will and will not make possible.
• Conclusions.

# The spec sheet, with error bars

Here are the confirmed numbers from Moonshot, side by side with K2, whose technical report is public:

	Kimi K2	Kimi K3
total parameters	1.04T	2.8T
active per token	~32B	~50–60B (reported, not yet official)
experts per MoE layer	384	896
experts activated	8 (+1 shared)	16
activation ratio	~2.1%	~1.8%
total : active	~32×	~50×
attention	MLA, every layer	KDA : Gated MLA, 3:1
context	128K (256K in K2.x)	1M
weights precision	BF16, quantized after training	MXFP4, QAT from SFT onward
modality	text	text + vision + video, one model

Three claims from the announcement frame most of what follows. Moonshot reports roughly 2.5× better overall scaling efficiency than K2, which they attribute to the combination of architecture and training recipe, so it cannot be assigned to any single change. They report up to 6.3× faster decoding at million-token context, which comes from the attention mechanism. And they recommend deploying on supernode configurations of 64 or more accelerators, which comes from the expert count. I will come back to each.

The architecture diagram in the blog answers one question the text leaves open: the repeating block is three KDA sub-blocks followed by one Gated-MLA sub-block, each followed by its own MoE FFN, with Attention Residual connections drawn from earlier blocks:

repeating block:
  [KDA + MoE] -> [KDA + MoE] -> [KDA + MoE] -> [Gated MLA + MoE]
   linear         linear         linear         full attention

This is the same 3:1 hybrid ratio the Kimi Linear paper validated at 48B scale in October. I find it genuinely informative that the ratio survived unchanged: Moonshot committed a frontier training run to a design that had, publicly at least, only been tested two orders of magnitude smaller.

# Sequence length: KDA and the 3:1 hybrid

Full attention keeps the keys and values of every past token and pays for it twice during decoding: memory that grows linearly with context, and per-token attention work that also grows linearly. Kimi Delta Attention belongs to the family of linear attention mechanisms that replace the growing cache with a fixed-size recurrent state. Each head maintains a $d_k \times d_v$ state matrix $S_t$ that is updated once per token and read once per token:

$$S_t = \left(\operatorname{Diag}(\alpha_t) - \beta_t, k_t k_t^{\top}\right) S_{t-1} + \beta_t, k_t v_t^{\top}, \qquad o_t = S_t^{\top} q_t$$

The update is a delta rule: rather than only accumulating $k_t v_t^{\top}$ outer products, it first removes what the state currently predicts for the key $k_t$ and then writes the new value, so the state behaves more like an associative memory that can be corrected than a running sum. What KDA adds over its closest predecessor, Gated DeltaNet, is the forget gate $\alpha_t$: it is a per-channel vector rather than a single scalar per head, so the state can retain some feature directions for a long time while clearing others quickly. The paper spends much of its length on making this efficient on GPUs, with a chunkwise algorithm over a specialized diagonal-plus-low-rank transition structure, and the kernels are public in the FLA repository.

For this post, the property that matters is simple: the state has the same size at token one hundred and at token one million. K3 uses KDA in three out of every four layers, keeping conventional (gated) MLA attention in the fourth so the model retains some exact global lookup capability, which the Kimi Linear ablations found necessary.

It helps to put numbers on what this does to memory, even approximate ones. K3's dimensions are not published, so I will use K2's MLA shape as a stand-in. K2 stores a compressed 576-dimensional latent per token per layer, about 1.1 KB at FP16. A hypothetical 60-layer model using MLA in every layer would carry roughly $576 \times 2 \times 10^6 \times 60 \approx 69$ GB of KV cache per sequence at 1M context. The same model with KDA in 45 of the 60 layers carries about 17 GB of growing cache for the 15 MLA layers, plus KDA states that total a few hundred megabytes and do not grow. That is the "up to 75% KV reduction" from the paper restated in bytes. The saving compounds in a useful way: less KV memory per sequence means more concurrent sequences fit on a GPU, and larger decode batches are exactly what a very sparse MoE needs to run efficiently, for reasons the width section gets into.

Two quieter consequences seem worth spelling out.

Pricing. The API charges the same rates at any context length: $0.30 per million cache-hit input tokens, $3.00 for cache misses, $15.00 for output, with no tiering between short and long contexts. Providers usually tier because with full attention a token processed at position 900,000 genuinely costs more than one at position 1,000. When three quarters of the layers do constant work against a constant-size state, the cost difference between those two tokens shrinks a lot, and flat pricing becomes something the provider can afford to offer. I don't know Moonshot's margins, but the structure of the price list is consistent with the structure of the model.

Prefix caching. Serving stacks lean heavily on the fact that full attention's KV cache is append-only. If two requests share a long prompt prefix, the KV blocks computed for that prefix are valid for both, forever, so an agentic workload that resends a large system prompt on every turn can skip most of its prefill. A KDA state does not have this property. It is a running summary, overwritten in place as tokens arrive, so caching a prefix means checkpointing the state matrices at the prefix boundary, per layer and per head, and restoring them exactly. That is a different data structure with different invalidation behavior than paged KV blocks. The announcement acknowledges this directly, saying KDA "poses new challenges for conventional prefix caching," and Moonshot has contributed an implementation to vLLM that will be released with the model. They report cache hit rates above 90% on coding workloads, and given the 10× price gap between hits and misses, their own serving economics depend on this mechanism working well. It is good news for everyone else that it ships as open source, since every other provider hosting the weights will need it too.

# Depth: Attention Residuals

The second change gets one paragraph in the announcement, and honestly not much more is known. Attention Residuals (AttnRes) modify the residual stream. In a standard transformer, each block adds its output into a running sum and every later block reads that sum; the announcement describes AttnRes as "selectively retrieving representations across depth rather than accumulating them uniformly," and the diagram shows each block drawing weighted connections from the outputs of earlier blocks and from the embedding, not just from the accumulated stream. Moonshot claims roughly 25% higher training efficiency at under 2% additional cost.

The motivation is easier to see if you think about what the residual stream has to do in a very deep model. It is a single $d_{model}$-dimensional vector that every layer writes into, so information from early layers has to survive hundreds of subsequent additions to be usable late in the network. Giving a late layer a direct, gated connection to an early layer's output is a way to route around that, and the "under 2% cost" figure suggests the retrieval is implemented with gates and additions rather than new matrix multiplies of any size.

The cost that does not show up in that 2% is memory residency during training. In a plain transformer, an activation can be freed (or cheaply recomputed) once the next layer has consumed it. If arbitrary later blocks can read block $k$'s output, those activations have consumers far downstream, which complicates activation checkpointing and pipeline-parallel schedules: either more tensors stay live or more recomputation happens. That cost lands on Moonshot's training cluster rather than on anyone's inference deployment, which may be part of why they considered it a good trade. If the technical report includes a clean ablation for the 25% figure, I would expect this idea to be adopted quickly, because a training-efficiency gain of that size at this scale is worth an enormous amount of compute.

# Width: 896 experts and the interconnect

The starting point is the basic asymmetry of MoE models: the routed experts hold most of the parameters but do a small minority of the per-token work, because each token only visits a few of them. (An upcoming post on this blog looks at this asymmetry in detail through vLLM-Moet, a stack that serves a 753B MoE on two consumer GPUs.) K3 pushes the asymmetry further than any open model so far. Each MoE layer has 896 experts of which 16 are active, so a token touches about 1.8% of the parameters, and the ratio of total to active parameters is about 50, up from about 32 in K2. The extra parameters are capacity the model gets during training at modest per-token compute cost. What I want to work through here is what they cost at serving time, because that is where the "64 or more accelerators" recommendation comes from.

Capacity is the first constraint, but not the binding one. At MXFP4's 4.25 bits per weight (32-weight blocks sharing an 8-bit exponent scale, the same block layout covered in the GGUF post), the weights come to

$$2.8\text{T} \times \frac{4.25}{8} \approx 1.5 \text{ TB}.$$

No 8-GPU node holds that with room left for KV cache and activations, so the model is sharded across tens of GPUs regardless of any other consideration.

The binding constraint is bandwidth, and it is a batching argument. During decoding, producing one token means reading the token's active parameters from HBM, roughly 29 GB at MXFP4 for ~55B active. A single stream therefore uses a small slice of the machine while 2.75T parameters sit idle. The standard remedy is batching: with enough concurrent streams, every expert has work every step, and the aggregate read per decode step approaches the full 1.5 TB. Expert parallelism spread wide (one or a few experts per GPU, across the whole cluster, often called WideEP) is what makes that aggregate read fast, because 64 GPUs each streaming their ~23 GB share per step is a parallel read of the whole model. This is the same "decode throughput tracks bytes moved" reasoning as always; the difference at this scale is that the bytes are spread over a cluster instead of a card.

What expert parallelism costs. Once experts live on different GPUs, the FFN becomes a communication pattern. For every MoE layer, each token's hidden vector has to be sent to the GPUs owning its 16 experts (the dispatch), and the 16 outputs have to come back and be summed (the combine):

one MoE layer, experts spread across the domain:

  token's hidden vector (on its home GPU)
      |
      |  dispatch (all-to-all #1):
      |  copy the vector to the GPUs owning its 16 experts
      v
  GPU 0           GPU 1           ...         GPU 63
  [expert FFNs]   [expert FFNs]               [expert FFNs]
      |
      |  combine (all-to-all #2):
      |  16 outputs back to the home GPU, weighted sum
      v
  token's new hidden vector

repeat for every MoE layer, for every token generated

That is two all-to-all exchanges per MoE layer, per forward pass. One of the SemiAnalysis threads counts more than 120 dispatch/combine rounds per forward pass for K3, which is consistent with roughly 60 MoE layers and the 3:1 block structure. A rough envelope, assuming a hidden size around 8K and MXFP8 activations: a token's dispatch is ~8 KB times 16 expert copies, about 128 KB per MoE layer, plus a combine of similar size, times ~60 layers, so on the order of 20–30 MB of activation traffic over the fabric per token per forward pass. With batches of thousands of tokens and many steps per second, this traffic is large enough that the interconnect between GPUs stops being a background detail and becomes one of the two memory systems the model runs on.

This is why the deployment recommendation specifies supernodes. The all-to-all needs to run over a scale-up fabric within a 64+ GPU domain, rather than over the slower scale-out network that connects 8-GPU islands, and the gap between the two is large. SemiAnalysis puts the copper backplane of a GB200/GB300 NVL72 at roughly 18× the bandwidth of a comparable DGX B200 system, and rack-scale domains of that kind (or the equivalents other accelerator vendors are building) are what the WideEP pattern is optimized for.

It also explains the argument in the SemiAnalysis threads I linked at the top. A reasonable first reaction to KDA is that it should reduce networking demand, since it cuts the KV cache that disaggregated serving systems transfer between prefill and decode clusters by up to 10×. The thread's counterpoint is about frequency: that KV transfer happens once per turn, while dispatch/combine happens 120+ times per forward pass, once per output token, for potentially hundreds of output tokens per turn. The attention savings are real, but they apply to a term that was already small next to the expert traffic, and the 896-expert design grows the larger term. Cheaper attention also tends to increase usage, longer contexts and more of them, so the overall demand for interconnect bandwidth goes up rather than down.

The training side of the same problem. At this sparsity, keeping experts evenly loaded stops being a tuning detail and becomes a throughput requirement, because in synchronous training the step time is set by the most loaded GPU. The announcement names its mechanisms. Quantile Balancing derives expert allocation from the quantiles of the router scores directly, replacing the auxiliary balancing loss (and its sensitive weighting hyperparameter) that most MoE recipes use. The expert-parallel training method is described as fully balanced, "with static shapes and no host synchronization on the critical path." Static shapes mean the all-to-all buffers are the same size every step regardless of what the router decided, which avoids reallocation and keeps the step CUDA-graph-friendly; no host synchronization means the GPU never waits on a CPU round-trip to learn where tokens are going. Both are properties you want when the same communication pattern will also run in inference. One term I can't yet explain is "Stable LatentMoE" itself: the "latent" part is not specified anywhere official, and while the name hints at experts parameterized against some shared latent space, that is a guess, and I would wait for the report before building on it.

# Numerics: training in MXFP4

K3 applies quantization-aware training from the SFT stage onward, with MXFP4 weights and MXFP8 activations. Two things make this more interesting than a release note about available precisions.

The first is the choice of format. MXFP4 is the OCP microscaling format, E2M1 elements in blocks of 32 under a shared power-of-two scale, and it is what Blackwell tensor cores and AMD's MI400 generation execute natively. The checkpoint's storage format and the hardware's matrix-multiply datapath are the same thing, so there is no separate dequantization step and, more importantly, no post-hoc calibration of the kind whose failure modes I catalogued in the GGUF post. The model spent its whole post-training life inside the quantization grid it ships in, with the optimizer compensating for the quantization error as it went. Post-training quantization to 4 bits is workable for many models, but at 2.8T parameters, where the weights are 1.5 TB even at 4.25 bits per weight, there is little appetite for shipping a BF16 master copy and hoping the community's quantizations are faithful. Training in the serving format removes the question.

The second is what it means for further compression. Whatever gets released by July 27 is presumably the MXFP4 artifact itself, so anyone wanting a smaller K3 starts from 4.25 bits per weight rather than 16. One relevant data point comes from vLLM-Moet, an open serving stack (the subject of an upcoming post here) that runs 753B–1T MoE models with their experts cut to about 2.25 bits per weight. Its results suggest that pushing below roughly 2.5 bits per weight is only survivable for models with a high ratio of total to active parameters, and K3's ratio of ~50 is the highest of any open model. The caveat is that this is one engineering project, not peer-reviewed research, so the threshold is an observation from a single stack and a handful of models rather than an established rule. With that grain of salt, K3 is at least a plausible candidate for extreme quantization, on this one axis more so than anything released before it.

# The serving stack

The remaining piece of the picture predates K3. Mooncake, Moonshot's serving platform and a best-paper winner at FAST 2025, is a disaggregated architecture organized around the KV cache: prefill and decode run on separate clusters, and cached context is pooled across the otherwise underused CPU memory, SSDs, and NICs of the GPU fleet. The official K3 API runs on it, and the blog credits it, together with the KDA prefix cache, for the >90% cache hit rate on coding workloads.

The same source adds a capacity observation that connects the width section to this pooling design. On whatever domain serves the model, the weights permanently occupy more than 1.5 TB of HBM before the first request arrives, and the context state competes for what is left. At long contexts that remainder is not small even after KDA's reduction: with the K2-shaped estimate from earlier, a single million-token sequence still carries about 17 GB of KV for the Gated-MLA layers, so a few dozen long-context streams hold as much memory as the weights themselves. SemiAnalysis expects the KV cache and KDA states to be offloaded to CPU DDR5 and NVMe even at relatively low concurrency, and the arithmetic supports that. It also means K3 leans on the tier Mooncake was built to manage, so the offload path is less a workaround than a part of the design the serving platform already assumed.

With the earlier sections in place, the pricing becomes easy to interpret. A cache-hit input token is 10× cheaper than a miss because a hit skips prefill compute entirely; it is a fetch of stored KV blocks and KDA states from the Mooncake pool into the decode cluster. KDA makes the fetched object smaller and makes long-context decoding faster, so each decode GPU serves more concurrent streams. Wide expert parallelism over 64+ GPU domains provides the aggregate HBM bandwidth that streaming 1.5 TB of experts per step requires. Each line item on the price list corresponds to a mechanism described in the announcement. I don't think I have seen a frontier release where the connection between the architecture and the serving economics is this easy to trace, and it is part of why I thought the announcement deserved a systems-side reading in the first place.

# What lands by July 27

Since this blog spends most of its time at the small end of the hardware spectrum, it is worth being concrete about what the open weights will make possible, and for whom.

The artifact is around 1.5 TB. It does not fit on a DGX H100 (640 GB of HBM), on an 8×H200 node (1.1 TB), or on anything a person owns. Running it as intended starts at the 64-GPU supernode scale Moonshot recommends. The most aggressive compression recipe demonstrated in public, vLLM-Moet's 2-bit experts with the GPU acting as an expert cache over NVMe, would shrink the experts to roughly 0.8 TB, which is still far outside workstation territory. And it is worth being honest about how hard that recipe is to replicate: it required hand-written kernels (in raw SASS, against a reverse-engineered consumer-Blackwell instruction set) and an inference engine modified end to end, all of which would have to be redone for K3's weight layout. The 896-expert shape is favorable for expert caching (more, smaller experts means a smaller per-step working set relative to the total), so I mention it as an existence proof that these numbers can move, not as something to expect soon.

In the near term, I think the practical value of the release for most readers is in the pieces around the weights rather than the weights themselves. The KDA kernels are public and useful at every scale. The vLLM prefix-caching implementation solves a problem every future linear-attention model will share. The architecture becomes inspectable, and a 2.8T open-weights model is a distillation source regardless of who can serve it. And the recent history here is encouraging: serving a 753B model on consumer hardware sounded unreasonable a year ago, and it has been done. K3 is a harder target, but not a categorically different one.

# Conclusions

What I take away from the announcement, holding the usual reservation that the technical report may revise the picture:

The three architectural changes line up with the three axes along which the model grew, and each one moves a cost to a place Moonshot evidently decided it could afford. KDA caps per-sequence attention state so context length stops being a memory problem, and pays for it with a harder prefix-caching problem that they solved in vLLM. AttnRes helps information cross hundreds of layers, and pays for it in training-time activation memory on their own cluster. The 896-expert MoE buys 2.8T parameters at ~55B per-token cost, and pays for it in scale-up network traffic, which is why the deployment guidance names a minimum interconnect domain rather than just a GPU count.

That last point is the one I find most worth remembering. For dense models, the serving bottleneck was HBM bandwidth on a single device. For a model shaped like K3, it is the all-to-all bandwidth of a 64+ GPU domain, crossed more than a hundred times per forward pass. Efficiency work in one subsystem (attention, here) did not shrink the overall hardware demand; it shifted the load onto the subsystem the architecture chose to lean on more heavily.

And the direction of design seems to have inverted. The traditional flow is model first, infrastructure second: train in BF16, quantize afterwards, let serving frameworks catch up. K3 was trained in its serving number format, its expert count was chosen together with a balancing method designed for the interconnect, and its attention mechanism shipped with the caching code it needs. Whether or not the benchmark numbers hold up, that co-design is the part of this release I expect to age well.