GGUF quantization, bit by bit

Open any GGUF repository on Hugging Face and you get a wall of files:

model-Q2_K.gguf        model-Q4_K_M.gguf      model-Q6_K.gguf
model-IQ1_S.gguf       model-Q4_K_S.gguf      model-Q8_0.gguf
model-IQ2_XXS.gguf     model-Q5_K_M.gguf      model-UD-Q4_K_XL.gguf
model-IQ3_XS.gguf      model-Q5_K_S.gguf      model-MXFP4.gguf

Most people pick Q4_K_M because a table somewhere recommended it, and that choice is usually reasonable. But each of those suffixes names a precise, documented binary layout: a specific arrangement of scales, offsets, packed integers, sign bits, and lookup-table indices, each described down to the byte by a C struct in ggml. Understanding those layouts, and the reasoning behind them, is what this article is about.

We will walk through all of them at the bit level. By the end you should be able to read a quant name and reconstruct what the bytes look like, compute the file size by hand, explain why a "2-bit" model actually costs 2.56 GiB and not 2 GiB, and understand what tools like Unsloth Dynamic and DwarfStar 4 are doing when they mix formats inside one file.

The article is long because it is meant to work as a compendium: it can be read front to back, but each section is also written to stand on its own, so you can come back later and jump directly to the format you need. The layout:

• The rules of the game: what block quantization is, type-0 vs type-1, the notation used throughout, and how bits-per-weight arithmetic works.
• How a quantized matmul actually runs: the part most explanations skip. Weights are not dequantized to float; the dot products happen in integers.
• The legacy quants: Q4_0, Q4_1, Q5_0, Q5_1, Q8_0.
• K-quants: Q2_K through Q6_K, super-blocks, and quantized quantization constants.
• S, M, L: what the mix suffixes really select.
• i-quants: IQ2_XXS through IQ4_XS, where codebooks and sign tricks replace plain integers.
• The one-bit frontier: IQ1_S, IQ1_M, and the ternary formats.
• The importance matrix: the calibration data that makes low-bit quants survivable.
• Quantization as a placement problem: per-tensor mixes, Unsloth Dynamic 2.0, DwarfStar 4.
• The floating-point cousins: MXFP4 and NVFP4.
• Choosing a quant: the practical table.
• Appendix: every format, one table.

A note on scope: this is about weight quantization in the GGUF/ggml ecosystem. KV-cache quantization is a different problem with different constraints; I covered a slice of it in the TurboQuant post.

The rules of the game

Everything in this post is post-training, weight-only, block-wise quantization. Each of those three terms carries a specific commitment:

Post-training: the model was trained in bf16 or fp8, and we compress the finished weights. No fine-tuning, no straight-through estimators (the rounding-aware backpropagation trick that quantization-aware training relies on). Quantization-aware training exists in this ecosystem too: Google ships QAT checkpoints of Gemma, and the ternary formats only make sense for QAT models. The formats themselves, however, don't care how the weights were produced.

Weight-only: activations stay in high precision as they flow through the network. They get temporarily quantized to 8 bits inside the matmul kernels, as we'll see, but nothing below 8 bits touches an activation on the weight path. (The one activation you can choose to store compressed is the KV cache; it is out of scope here, but the same type names apply.)

Block-wise: the most consequential of the three. A weight tensor is chopped into fixed-size blocks (32 or 256 consecutive weights along a row), and each block is quantized independently with its own scale. The block is the unit of everything: the C structs, the file layout, the SIMD kernels.

Before the first formula, let's fix the notation used for the rest of the article. These symbols keep the same meaning in every section:

• $x_i$ is an original weight, as trained; $\hat x_i$ is its reconstruction after quantizing and dequantizing.
• $q_i$ is the small integer code stored for $x_i$ (the "quant").
• $d$ is a block's scale factor (the ggml source calls it delta), and $m$ is a block's offset, typically its minimum.
• $y_i$ is an activation value, and $p_i$ is its 8-bit integer code inside the matmul kernels.
• $sc_j$ and $m_j$ are the quantized scale and quantized min of sub-block $j$ inside a K-quant super-block.
• $a_i$ is the activation that weight $x_i$ gets multiplied by, used when we discuss calibration.

With that in place: the simplest possible scheme stores, per block, one float scale $d$ and one integer code $q_i$ per weight, and reconstructs each weight as

$$x_i \approx d \cdot q_i$$

ggml calls this type-0. Dequantization is one multiply. The alternative adds the offset $m$ per block:

$$x_i \approx d \cdot q_i + m$$

That's type-1. The offset buys asymmetry: if all the weights in a block are, say, positive, a type-0 quantizer wastes half its codes on negative values that never occur, while a type-1 quantizer slides the whole grid onto the data. The price is an extra stored constant per block and an extra term in the dot product, which is cheaper than it sounds, as we'll see. You can read the entire family of formats below as increasingly refined answers to one question: given a budget of N bits per weight, how should you split them between payload (the $q_i$) and metadata (the scales, mins, signs, and indices that give the payload meaning)?

That split is why bits-per-weight (bpw) is never an integer. Take Q8_0, the simplest real format: blocks of 32 weights, each block storing one fp16 scale (2 bytes) plus 32 int8 values (32 bytes). That's 34 bytes per 32 weights:

$$\frac{34 \times 8}{32} = 8.5 \text{ bpw}$$

That is eight bits of payload plus half a bit of metadata, and every format in this post is an entry in the same ledger. For each one, we will look at exactly where the bits go.

One more piece of bookkeeping before we start. GGUF is the container: a binary file with metadata, tensor names, shapes, and offsets. The quantization types are ggml tensor types, and a single GGUF file almost always contains several of them. When a file is called Q4_K_M, that's the general.file_type metadata saying "mostly Q4_K"; the token embedding, the output projection, and a handful of attention tensors are usually something else entirely. We'll get to why.

Also: blocks run along tensor rows, so a row's width must be divisible by the block size. When it isn't (a 300-wide tensor can't be cut into 256-blocks), llama-quantize silently falls back to a compatible type (Q6_K falls back to Q8_0, Q5_K to Q5_1, and so on). If you've ever wondered why one odd tensor in your file has a different type than everything around it, that's usually the reason.

How a quantized matmul actually runs

A common mental model of quantized inference is that the weights are stored compressed and, at inference time, get decompressed back to floats before being multiplied. That model is inaccurate in an important way, and correcting it explains many of the design decisions below.

During decode, LLM inference is memory-bound. The matrix-vector products at batch size 1 are trivial compute; the bottleneck is streaming gigabytes of weights from RAM or VRAM into the cores every single token. Quantization addresses exactly this: fewer bytes per weight means fewer bytes to stream. But if the kernel dequantized everything to fp32 first, the bandwidth win would be partly consumed by conversion work.

So ggml does something better. Every quantized type declares a companion dot-product type: an 8-bit format that the activations get converted into on the fly, once per matmul. For legacy quants that companion is Q8_0 (or Q8_1 for the type-1 variants); for K-quants and i-quants it's Q8_K, which quantizes activations in blocks of 256 with an fp32 scale. Then, for each pair of a weight block (original values $x_i$, scale $d_x$, codes $q_i$) and an activation block (original values $y_i$, scale $d_y$, codes $p_i$), the kernel computes

$$\sum_i x_i , y_i \approx (d_x , d_y) \sum_i q_i , p_i$$

where the inner sum is a pure integer dot product between the weight codes and the activation codes. Integer dot products are an operation modern hardware performs exceptionally well: AVX2's maddubs, ARM's i8mm and sdot, CUDA's DP4A. The float scales get multiplied once per block, not once per element.

Two details are worth examining:

The bsums optimization. Type-1 formats and K-quants with mins introduce an offset term into the dot product. Expanding it (written here with the K-quant sign convention, where the min $m$ is stored positive and subtracted; legacy type-1 adds it instead, and the algebra is identical):

$$\sum_i (d,q_i - m), y_i = d \sum_i q_i,y_i - m \sum_i y_i$$

The second term only needs the sum of the activations in the block, $\sum_i y_i$; it doesn't depend on the weights at all. So Q8_K precomputes partial sums of its quants (16 weights at a time) and stores them in the block (bsums), and the min term costs almost nothing. This is why formats can afford per-sub-block minimums: the arithmetic they add is one multiply per 16 elements.

// This is only used for intermediate quantization and dot products
typedef struct {
    float   d;              // delta
    int8_t  qs[QK_K];       // quants
    int16_t bsums[QK_K/16]; // sum of quants in groups of 16
} block_q8_K;

Q8_1 plays the same role for the legacy type-1 formats, and its declaration shows how far the precomputation idea goes: it dedicates a field to s = d * sum(qs), whose only job is to make the offset term free.

typedef struct {
    ggml_half d;      // delta
    ggml_half s;      // d * sum(qs[i]), precomputed for the offset term
    int8_t qs[QK8_1]; // quants
} block_q8_1;

(These structs, like every struct quoted in this article, come from ggml-common.h, the single header that defines all block layouts for every backend. One simplification is applied throughout: where the source wraps two fp16 fields in a small union for backend convenience, I show them as plain fields.)

Quantization makes decode faster, not just smaller. From the llama.cpp quantize README, Llama-3.1-8B single-stream text generation: F16 runs at 29.2 t/s, Q8_0 at 50.9, Q4_K_M at 71.9, IQ1_S at 79.7. Speed tracks the inverse of bytes moved, sub-linearly at the top where per-block decode work starts to matter. Meanwhile prompt processing sits near 800 t/s for every type, because prefill is compute-bound and everyone pays roughly the same integer-multiply cost. Quantization is fundamentally a bandwidth technique, and its benefits appear wherever bandwidth is the limiting factor.

Keep this in mind for everything that follows: each format below was designed simultaneously as a compression codec and as the operand layout of a SIMD kernel. When a layout looks strange (nibbles from distant positions sharing a byte, scales packed across byte boundaries), the explanation is almost always that the AVX2 or NEON kernel wanted it that way.

The legacy quants: Q4_0, Q4_1, Q5_0, Q5_1, Q8_0

These are the original ggml formats from 2023: block size 32 and deliberately simple. They remain in every backend partly for compatibility, and partly because two of them still have important roles.

Q4_0 is the archetype. Per block of 32 weights: one fp16 scale, then 16 bytes of packed 4-bit codes. (ggml_half in these structs is an IEEE fp16 value.)

#define QK4_0 32
typedef struct {
    ggml_half d;           // delta
    uint8_t qs[QK4_0 / 2]; // nibbles / quants
} block_q4_0;

18 bytes per 32 weights → 4.5 bpw. The codes are unsigned 0..15 with an implicit offset: dequantization is $x = d,(q - 8)$. Quantization is round-to-nearest with the scale chosen so the largest-magnitude weight in the block lands on an end of the range.

How the 32 codes are arranged inside the 16 bytes of qs is a defining characteristic of the format, and it is not the arrangement most people guess. The natural assumption is that consecutive weights share a byte: weight 0 and weight 1 in byte 0, weight 2 and weight 3 in byte 1, and so on. Instead, ggml splits the block in half: byte $j$ holds weight $j$ in its low nibble and weight $j+16$ in its high nibble, so the low nibbles of the 16 bytes carry the first half of the block and the high nibbles carry the second half.

byte:      0    1      2        3        4      ...     17
         [ d (fp16) ][q16|q0][q17|q1][q18|q2]  ...  [q31|q15]
                      hi  lo  hi  lo  hi  lo        hi   lo

The reason is the dequantization kernel. A SIMD implementation loads all 16 bytes into a single vector register; one AND with 0x0F then produces weights 0–15 in order, and one 4-bit right shift produces weights 16–31, also in order. With the consecutive-weight packing, extracting the weights in sequence would require an interleaving shuffle on every block. This is the first concrete instance of the principle stated earlier, that each layout is designed for the kernel that reads it, and the same split-halves nibble pattern reappears in most of the 4-bit and 5-bit formats below. Keep it in mind whenever you write code that touches qs bytes directly.

Q4_1 adds an fp16 min for the type-1 affine version, $x = d,q + m$: 20 bytes, 5.0 bpw. Q5_0 and Q5_1 extend the codes to 5 bits by storing the fifth bit of each weight separately, packed into a 4-byte qh bitfield. This is an early appearance of a technique you will see throughout this article: when a bit width doesn't divide 8, store the "leftover" bits in their own plane. 22 bytes (5.5 bpw) and 24 bytes (6.0 bpw) respectively.

Their declarations show both additions:

typedef struct {
    ggml_half d;           // delta
    ggml_half m;           // min
    uint8_t qs[QK4_1 / 2]; // nibbles / quants
} block_q4_1;

typedef struct {
    ggml_half d;           // delta
    uint8_t qh[4];         // 5th bit of each of the 32 quants
    uint8_t qs[QK5_0 / 2]; // nibbles / quants
} block_q5_0;

Q5_1 combines the two: d, m, qh, and qs in one block.

Then there's Q8_0: fp16 scale plus 32 straight int8 codes, 34 bytes, 8.5 bpw.

#define QK8_0 32
typedef struct {
    ggml_half d;       // delta
    int8_t  qs[QK8_0]; // quants
} block_q8_0;

Q8_0 is the closest thing this ecosystem has to a reference format. Its round-trip error is negligible for LLM weights (perplexity is typically indistinguishable from fp16 at measurement noise level), so it serves as the option for when you don't want to think about quality at all, the standard KV-cache quantization type, the on-the-fly activation format, and the format DwarfStar uses for everything it declines to compress. When a quantized model is evaluated "against baseline", Q8_0 is usually that baseline.

Why did Q4_0's simplicity stop being good enough? One scale per 32 weights means a single outlier degrades its entire block: the scale must stretch to cover the outlier, and the remaining 31 weights are left with only a handful of effective levels. You can fight that with a smaller block (more scales, more overhead) or an offset (Q4_1, which helps with asymmetry but not with outliers). The more durable fix was to become more sophisticated about the metadata, which is exactly what K-quants did.

K-quants: scales all the way down

In June 2023, Iwan Kawrakow landed the k-quants PR, which is still the workhorse family today. It introduced two ideas, both concerning metadata:

Idea one: the super-block. Blocks grow to 256 weights, subdivided into sub-blocks of 16 or 32. Each sub-block gets its own scale (and possibly min), so the format adapts to local structure at a finer granularity than Q4_0, not coarser.

Idea two: quantize the quantization constants. Storing an fp16 scale per 16 weights would cost a whole bit per weight just in scales. So the sub-block scales are themselves quantized (to 4, 6, or 8 bits) relative to one fp16 super-scale per 256-block. For a weight in sub-block $j$, with the notation from earlier, the hierarchy is:

$$x_i \approx \underbrace{d}$${\text{fp16, per 256}} \cdot \underbrace{sc_j}{\text{4-8 bits, per 16/32}} \cdot ; q_i ;-; \underbrace{d_{\min}}{\text{fp16, per 256}} \cdot \underbrace{m_j}{\text{4-8 bits, per 16/32}}

This is metadata describing metadata, and it may sound baroque, but the arithmetic below shows that it is precisely what makes usable 2-bit and 3-bit formats possible.

A third change hides in the quantizer rather than the format: legacy quants just round to nearest, while the K-quant quantizer runs a small search per sub-block, trying candidate scales and keeping the one that minimizes weighted squared error. The file format doesn't reveal this, but it's the hook where the importance matrix will later attach.

One note on reading the structs that follow, because they contain a line that looks like a contradiction: a comment such as "scales, quantized with 6 bits" sitting above a plain uint8_t array. There is no contradiction. C has no 4-bit or 6-bit type, so uint8_t is only the storage granularity, and the quantized scale values are bit-packed into the byte array. The array length is where the arithmetic shows: sixteen 6-bit values need 16 × 6 = 96 bits, which is exactly the scales[12] you will see in Q3_K and Q4_K, with individual values straddling byte boundaries. The bit width of the scales is itself a design knob, tuned per format against the payload it serves: Q2_K spends 4 bits per scale code, Q3_K through Q5_K spend 6, and Q6_K affords plain int8 scales. Six bits instead of eight saves 0.125 bpw and gives up nothing that matters, since 64 scale levels relative to an fp16 super-scale is already fine-grained.

Let's go through the family. QK_K = 256 throughout.

Q2_K: 2.625 bpw

typedef struct {
    uint8_t scales[QK_K/16]; // scales and mins, quantized with 4 bits
    uint8_t qs[QK_K/4];      // quants
    ggml_half d;             // super-block scale for quantized scales
    ggml_half dmin;          // super-block scale for quantized mins
} block_q2_K;

84 bytes per 256 weights: 16 bytes of scales, 64 bytes of qs, and two fp16 super-scales. Drawn as a memory map (widths not to scale, byte offsets on top):

byte:  0             16                                  80      82      84
       ┌──────────────┬───────────────────────────────────┬───────┬───────┐
       │  scales[16]  │              qs[64]               │   d   │ dmin  │
       │ 16 × (scale, │  256 × 2-bit codes, packed 4/byte │ fp16  │ fp16  │
       │  min) pairs  │    in four bit-planes (below)     │       │       │
       └──────────────┴───────────────────────────────────┴───────┴───────┘

The super-block is divided into sixteen sub-blocks of 16 weights, and each sub-block owns one byte of scales: the low nibble is its quantized scale, the high nibble its quantized min.

scales[j], one byte per sub-block j of 16 weights:

  bit:  7          4 3          0
       ┌────────────┬────────────┐
       │  min  m_j  │ scale sc_j │
       └────────────┴────────────┘

The payload is 2-bit codes, $q \in {0,1,2,3}$, four per byte. Their arrangement generalizes Q4_0's split-halves idea: qs is two 32-byte halves, each covering 128 consecutive weights, and within a half each byte carries the codes of four weights spaced 32 apart, one in each 2-bit plane.

qs byte b inside a 32-byte half (each half covers 128 weights):

  bit:  7 6   5 4   3 2   1 0
       ┌─────┬─────┬─────┬─────┐       each cell is the 2-bit code of
       │b+96 │b+64 │b+32 │  b  │       one weight; the four weights of
       └─────┴─────┴─────┴─────┘       a byte sit 32 positions apart

Following one weight through the whole layout makes the addressing concrete:

decoding weight i = 100:
  sub-block   j = 100 / 16          = 6      → scale byte scales[6]
  half          = 100 / 128         = 0      → bytes qs[0..31]
  byte        b = 100 mod 32        = 4      → qs[4]
  bit-plane     = (100 mod 128)/32  = 3      → bits 7..6
  code        q = (qs[4] >> 6) & 3
  weight      x = d * (scales[6] & 0xF) * q  -  dmin * (scales[6] >> 4)

In general form, this is the dequantization from ggml's reference implementation, where sc is the weight's per-sub-block byte. It mixes arithmetic with bit extraction, so C is the honest notation for it:

x = d * (sc & 0xF) * q  -  dmin * (sc >> 4);

Where the bits go, per weight:

component	bits/weight	share
2-bit codes	2.0	76%
sub-block scales+mins (16×8 bits)	0.5	19%
super-scales d, dmin (2×16 bits)	0.125	5%

Four levels per weight is very coarse (every weight in a sub-block is reduced to one of four values), and Q2_K quality reflects that. The format remains usable because the min term recenters each 16-weight group, and because in the standard Q2_K mix the most sensitive tensors are not actually stored as Q2_K (more on mixes below). Its modern relevance: it's one of the two formats DwarfStar 4 uses for DeepSeek's routed experts.

Q3_K: 3.4375 bpw

typedef struct {
    uint8_t hmask[QK_K/8]; // quants, high bit
    uint8_t qs[QK_K/4];    // quants, low 2 bits
    uint8_t scales[12];    // scales, quantized with 6 bits
    ggml_half d;           // super-block scale
} block_q3_K;

110 bytes per 256. The 3-bit codes are split across two planes: qs holds the low 2 bits (64 bytes), hmask holds the third bit (32 bytes), the same technique as Q5_0. Sixteen 6-bit scales pack into 12 bytes, and there is no min: Q3_K is type-0, $x = d \cdot sc \cdot q$ with signed codes in $[-4, 3]$. Dropping the mins is where the budget for the wider 6-bit scales comes from.

Q4_K: 4.5 bpw

Q4_K is the most widely used format in the family: it is the backbone of Q4_K_M, the default choice for most local deployments.

typedef struct {
    ggml_half d;                  // super-block scale for quantized scales
    ggml_half dmin;               // super-block scale for quantized mins
    uint8_t scales[K_SCALE_SIZE]; // scales and mins, quantized with 6 bits (12 bytes)
    uint8_t qs[QK_K/2];           // 4-bit quants
} block_q4_K;

144 bytes per 256 weights: two fp16 super-scales, 12 bytes of packed scale/min metadata, and 128 payload bytes. As a memory map:

byte:  0       2       4          16                                 144
       ┌───────┬───────┬───────────┬───────────────────────────────────┐
       │   d   │ dmin  │ scales[12]│              qs[128]              │
       │ fp16  │ fp16  │ packed 6- │  256 × 4-bit codes, split-halves  │
       │       │       │ bit pairs │    nibbles per 64-weight group    │
       └───────┴───────┴───────────┴───────────────────────────────────┘

Eight sub-blocks of 32 weights, each with a 6-bit scale and a 6-bit min. Sixteen 6-bit values into 12 bytes does not pack cleanly, and the arrangement ggml chose is the most intricate in the K family: sub-blocks 0-3 store their values whole in the low 6 bits of the first eight bytes, while the values of sub-blocks 4-7 are split across the leftover pieces.

scales[12]: eight 6-bit scales s0..s7, eight 6-bit mins m0..m7

  bit:            7   6 5          0
                 ┌─────┬────────────┐
  bytes 0..3     │s4-s7│   s0..s3   │  6-bit scales 0-3 in full, plus
                 │top 2│  (6 bits)  │  the top 2 bits of scales 4-7
                 ├─────┼────────────┤
  bytes 4..7     │m4-m7│   m0..m3   │  6-bit mins 0-3 in full, plus
                 │top 2│  (6 bits)  │  the top 2 bits of mins 4-7
                 └─────┴────────────┘
  bit:            7      4 3      0
                 ┌────────┬────────┐
  bytes 8..11    │ m4..m7 │ s4..s7 │  bottom 4 bits of scales 4-7 in the
                 │ low 4  │ low 4  │  low nibble, of mins 4-7 in the high
                 └────────┴────────┘

A 6-bit value for sub-blocks 4-7 is therefore reassembled from two bytes at decode time; scale 6, for instance, is (scales[10] & 0xF) | ((scales[2] >> 6) << 4). None of the 96 bits goes unused. The payload bytes follow the split-halves nibble pattern from Q4_0, applied per 64-weight group. The ledger:

component	bits/weight	share
4-bit codes	4.0	88.9%
sub-block scales+mins (16×6 bits)	0.375	8.3%
d, dmin	0.125	2.8%

Compare with Q4_0's ledger (4.0 payload + 0.5 scale, the same 4.5 bpw total) and notice what changed: not the budget, the allocation. Q4_0 spends its entire half-bit on one fp16 scale per 32 weights. Q4_K spends the same half-bit on a 6-bit scale and a 6-bit min per 32, plus two fp16 super-scales per 256 to absorb the dynamic range the sub-scales gave up. The cost is identical, the allocation is strictly more adaptive, and the measured quality is better. This comparison captures the central idea of the k-quants.

Q5_K: 5.5 bpw

typedef struct {
    ggml_half d;                  // super-block scale for quantized scales
    ggml_half dmin;               // super-block scale for quantized mins
    uint8_t scales[K_SCALE_SIZE]; // scales and mins, quantized with 6 bits
    uint8_t qh[QK_K/8];           // quants, high bit
    uint8_t qs[QK_K/2];           // quants, low 4 bits
} block_q5_K;

Q4_K plus a fifth-bit plane (qh, 32 bytes), 176 bytes per 256. Same 6-bit scale/min machinery. This is the payload behind Q5_K_S and Q5_K_M, a sensible tier when you have some memory headroom and want inexpensive quality insurance.

Q6_K: 6.5625 bpw

typedef struct {
    uint8_t ql[QK_K/2];      // quants, lower 4 bits
    uint8_t qh[QK_K/4];      // quants, upper 2 bits
    int8_t  scales[QK_K/16]; // scales, quantized with 8 bits
    ggml_half d;             // super-block scale
} block_q6_K;

210 bytes per 256. Codes are 6-bit, split 4+2 across the two planes visible in the struct: each weight's code is reassembled as q = (ql | qh << 4) - 32, where ql is the weight's nibble from the low plane and qh its 2 bits from the high plane, and the - 32 makes the code signed. Sixteen full int8 scales per super-block. Type-0, no mins; at 64 levels per weight they are no longer needed. Q6_K matters beyond being a file type you can choose: it's the type llama-quantize assigns to output.weight in almost every mix, because the final projection back to vocabulary space is the single most quantization-sensitive tensor in a transformer. Measured perplexity for pure Q6_K sits within a rounding error of fp16, which is why it is the standard answer when you want essentially lossless quality in less space than Q8_0.

And that's the K family as it appears in files. Q8_K (the fp32-scaled, bsums-carrying 256-block int8 format from earlier) completes it in memory only.

S, M, L: the mix suffixes

This is the point where filenames and block formats diverge. Q4_K_S and Q4_K_M contain the same block formats; what differs is the recipe: which tensor gets which type. Inside llama-quantize there's a function (llama_tensor_get_type in llama-quant.cpp) containing a long list of hand-tuned rules accumulated over two years of perplexity experiments. The important behaviors, verified against the current source:

• output.weight → Q6_K, almost always, regardless of base type.
• Token embeddings → left at the base type in the mainstream mixes; the ultra-low recipes override them upward (IQ1 and IQ2_XXS/XS files carry a Q2_K embedding, IQ2_S and IQ3_XXS an IQ3_S one). Embeddings are lookup tables, not matmuls against noisy activations; they tolerate a lot, but not the sub-2-bit codebooks.
• attn_v.weight → bumped a tier. For Q4_K_M/Q5_K_M, the value projection goes to Q6_K on a schedule; even Q4_K_S promotes it to Q5_K in the first four layers. The V projection is small (especially with grouped-query attention, where a handful of KV heads serve many query heads), so it contributes few parameters while influencing every attention output, which makes the extra bits cheap and valuable.
• ffn_down.weight → the M mixes promote it to Q6_K on a peculiar schedule: first eighth of layers, last eighth, and every third layer in between (use_more_bits). The schedule was not derived from theory; it is simply the pattern that performed best in perplexity sweeps.

So concretely, for the names you see everywhere: Q5_K_S is "Q5_K nearly pure, output at Q6_K": 5.5 bpw payload, measured whole-file ~5.57 bpw on Llama-3.1-8B. Q5_K_M additionally bumps attn_v and part of ffn_down to Q6_K → ~5.70 bpw. Q3_K_L is Q3_K with the sensitive set at Q5_K. The pattern generalizes: S ≈ pure base type, M ≈ base + promoted sensitive tensors, L ≈ base + promoted further.

This also resolves the puzzle from the intro: a "Q2_K" file measures 3.16 bpw on Llama-3.1-8B, a full half-bit above Q2_K's 2.625, because the recipe quietly upgrades embeddings, attention, and chunks of the FFN. In other words, the type in the filename indicates the lowest precision present in the file, not the average. This recipe machinery, coarse and hand-tuned but effective, is the direct ancestor of what Unsloth and DwarfStar do with much more ambition later in this post.

(You can override all of it: --output-tensor-type, --token-embedding-type, and per-tensor regex rules via --tensor-type "ffn_down=q6_k" are right there in the CLI. --pure disables the mix entirely, which is mostly useful for measuring how much the mix was saving you.)

i-quants: below three bits you need a codebook

Building a 2-bit format the K-quant way runs into a wall, and it is worth being precise about why the wall is fundamental rather than a matter of engineering effort. Four levels per weight, even with perfect per-16 scales and mins, still rounds each weight independently; the quantization error is roughly uniform and there is no metadata refinement left that would reduce it. Below roughly 3 bpw, scalar quantization (any scheme that rounds each weight on its own grid) visibly degrades models. Rate-distortion theory explains why: weights within a block are approximately jointly Gaussian, and quantizing them jointly, as vectors (vector quantization), can shape the error in ways independent rounding cannot. To do better, the question has to change from "which of 4 levels is this weight?" to "which of N patterns are these 8 weights?"

That is what the i-quants do (Kawrakow again, early 2024, taking the lattice-codebook idea explored by the QuIP# paper and re-engineering it for CPU-friendly inference). Three mechanisms replace the plain integer payload:

1. Codebooks: groups of 8 weights are quantized jointly to one of 256–2048 predefined 8-dimensional vectors, stored as an index. The codebook entries are chosen on the E8 lattice, the provably densest sphere packing in 8 dimensions (Viazovska's Fields-Medal result), which means that for a fixed number of code points, the expected distance from an arbitrary point to its nearest code is minimized. A classical sphere-packing result, applied to quantization.
2. Separated signs: the codebook stores magnitudes; signs are stored per-weight in a compressed side channel.
3. Parity encoding: 8 signs should cost 8 bits, but the sign patterns are restricted to those with even parity (128 of the 256 possibilities), so they fit in 7 bits, and the 8th sign is implied. Where the data wants odd parity, the quantizer flips the least-important weight's sign and accepts the error. The format is genuinely this careful about individual bits: the one bit recovered per 8 weights amounts to 0.125 bpw, a meaningful fraction of a 2-bit budget.

IQ2_XXS: 2.0625 bpw, a full decode

IQ2_XXS is the purest expression of this design. Its declaration is minimal, because almost everything lives inside the packed words:

typedef struct {
    ggml_half d;         // super-block scale
    uint16_t qs[QK_K/8]; // 32 uint16: indices, signs and scales, all packed
} block_iq2_xxs;

Per 256-weight block: one fp16 super-scale, then 32 uint16 values. Each 32-weight group owns four of them: 64 bits, which ggml reads as a pair of uint32s:

aux32[0]   bit: 31       24 23       16 15        8 7         0
                ┌───────────┬───────────┬───────────┬───────────┐
                │  index 3  │  index 2  │  index 1  │  index 0  │
                └───────────┴───────────┴───────────┴───────────┘
                 one 8-bit codebook index per group of 8 weights

aux32[1]   bit: 31    28 27      21 20      14 13       7 6        0
                ┌────────┬──────────┬──────────┬──────────┬──────────┐
                │ scale  │ signs 3  │ signs 2  │ signs 1  │ signs 0  │
                └────────┴──────────┴──────────┴──────────┴──────────┘
                 4 bits    one 7-bit sign pattern per group of 8

And the reference dequantization, verbatim from ggml-quants.c:

const float db = d * (0.5f + (aux32[1] >> 28)) * 0.25f;
for (int l = 0; l < 4; ++l) {
    const uint8_t * grid  = (const uint8_t *)(iq2xxs_grid + aux8[l]);
    const uint8_t   signs = ksigns_iq2xs[(aux32[1] >> 7*l) & 127];
    for (int j = 0; j < 8; ++j) {
        y[j] = db * grid[j] * (signs & kmask_iq2xs[j] ? -1.f : 1.f);
    }
    y += 8;
}

These few lines contain the entire format. d is the block's fp16 super-scale and db the effective sub-block scale, built from the 4-bit code in the top of aux32[1] with an implicit $+0.5$ offset and a fixed $\div 4$. aux8 is the byte view of aux32[0], so aux8[l] is the codebook index of the $l$-th group of 8 weights; iq2xxs_grid is the 256-entry codebook, each entry 8 bytes of magnitudes drawn from just three values. ksigns_iq2xs expands each 7-bit pattern to 8 signs via the parity rule, and kmask_iq2xs selects the bit for each position. The bit ledger:

component	bits/weight	share
codebook indices (8 bits / 8 weights)	1.0	48.5%
signs (7 bits / 8 weights)	0.875	42.4%
sub-block scales (4 bits / 32)	0.125	6.1%
super-scale d (16 bits / 256)	0.0625	3.0%

The table rewards a careful look: in a state-of-the-art 2-bit format, almost half of the bits are sign bits, and only one bit per weight goes toward selecting the magnitude pattern. The information about how large each weight is has been compressed into a shared index into a lattice, and the per-weight information that survives is little more than polarity. That this works at all (a 100-billion-parameter model still writes coherent code with one magnitude-bit per weight) is the strongest empirical statement I know about how much redundancy these models carry.

IQ2_XS, IQ2_S, IQ3_XXS, IQ3_S: variations on the design

The rest of the 2- and 3-bit family reuses the same mechanisms with different parameters:

• IQ2_XS (2.3125 bpw): codebook grows to 512 entries (9-bit indices), plus 4-bit sub-block scales per 16 weights. A larger codebook and finer scales, for roughly 0.25 bpw more.
• IQ2_S / IQ2_M (2.5625 / ~2.94 file bpw): a larger grid again, and the sign patterns get an explicit byte-plane instead of the 7-bit parity encoding. IQ2_M is a recipe on top of IQ2_S blocks, not a distinct block format.
• IQ3_XXS (3.0625 bpw): the IQ2_XXS architecture at 3 bits: groups of 4 weights, a 256-entry grid of 4-dimensional magnitude vectors, the same 7-bit-per-8-weights sign machinery.
• IQ3_S (3.4375 bpw): 512-entry grid, full 8-bit sign bytes instead of the parity encoding, same per-32 scale density. Note that it lands on exactly Q3_K's bpw: a deliberate head-to-head where the codebook approach measurably wins on perplexity at identical size, demonstrating that the codebook's added complexity translates into real quality. The IQ3_XS and IQ3_M you see on Hugging Face are recipes over these two block types (like IQ2_M over IQ2_S, and Q2_K_S over Q2_K), not further formats.

For reference, the four declarations:

typedef struct {              // IQ2_XS, 2.3125 bpw
    ggml_half d;
    uint16_t qs[QK_K/8];      // 9-bit grid index + 7 sign bits, per 8 weights
    uint8_t  scales[QK_K/32]; // 4-bit sub-block scales
} block_iq2_xs;

typedef struct {              // IQ2_S, 2.5625 bpw
    ggml_half d;
    uint8_t qs[QK_K/4];       // low index bytes, then explicit sign bytes
    uint8_t qh[QK_K/32];      // 2 extra index bits per group of 8
    uint8_t scales[QK_K/32];  // 4-bit sub-block scales
} block_iq2_s;

typedef struct {              // IQ3_XXS, 3.0625 bpw
    ggml_half d;
    uint8_t qs[3*QK_K/8];     // indices, then packed signs and scales
} block_iq3_xxs;

typedef struct {              // IQ3_S, 3.4375 bpw
    ggml_half d;
    uint8_t qs[QK_K/4];       // 8 index bits per group of 4 weights
    uint8_t qh[QK_K/32];      // 9th index bit
    uint8_t signs[QK_K/8];    // explicit sign bytes
    uint8_t scales[QK_K/64];  // 4-bit sub-block scales
} block_iq3_s;

IQ4_NL and IQ4_XS: non-uniform levels

At 4 bits, the joint-quantization machinery no longer justifies its cost, since 16 levels per weight is enough that scalar quantization works again. But one insight survives: why should the 16 levels be evenly spaced? Weights are bell-curve distributed; a uniform grid wastes resolution in the tails where almost no weights live and starves the dense middle.

IQ4_NL ("non-linear", 4.5 bpw) has exactly Q4_0's layout (fp16 scale, 32 nibbles), but the nibble indexes a fixed lookup table instead of feeding an affine formula:

GGML_TABLE_BEGIN(int8_t, kvalues_iq4nl, 16)
    -127, -104, -83, -65, -49, -35, -22, -10, 1, 13, 25, 38, 53, 69, 89, 113,
GGML_TABLE_END()

Look at the gaps: 23, 21, 18, 16, 14, 13, 12, 11, 12, 12, 13, 15, 16, 20, 24. Dense near zero, sparse at the extremes: in effect a companding curve, the same idea as the NF4 data type from QLoRA, fitted once to the empirical distribution of LLM weight blocks and frozen into every backend's kernels. IQ4_XS (4.25 bpw) wraps the same 16-value table in K-quant-style super-blocks (256-weight blocks, 6-bit sub-scales), shaving off a quarter-bit. IQ4_XS is one of the strongest formats in the entire family: measurably close to Q4_K in quality, about 5% smaller, and fast across backends thanks to the fixed table.

The two declarations:

typedef struct {
    ggml_half d;
    uint8_t qs[QK4_NL/2];       // 4-bit table indices, Q4_0-style layout
} block_iq4_nl;

typedef struct {
    ggml_half d;
    uint16_t scales_h;          // high 2 bits of the eight 6-bit scales
    uint8_t  scales_l[QK_K/64]; // low 4 bits, two scales per byte
    uint8_t  qs[QK_K/2];        // 4-bit table indices
} block_iq4_xs;

The one-bit frontier: IQ1_S, IQ1_M, and the ternary formats

The phrase "1.5-bit quantization" can be confusing, since no hardware stores half bits. The fractional figure is ordinary arithmetic: sub-2-bpw formats spend more than one but much less than two bits per weight, by using an 11-bits-per-8-weights codebook (1.375 bpw of index) and making the remaining metadata nearly free.

IQ1_S: 1.5625 bpw

typedef struct {
    ggml_half d;
    uint8_t  qs[QK_K/8];   // 32 bytes: low 8 bits of grid indices
    uint16_t qh[QK_K/32];  // 8 uint16: high 3 bits × 4 groups, scale, delta sign
} block_iq1_s;

50 bytes per 256 weights. Each group of 8 weights stores an 11-bit index (8 bits in qs, 3 in qh) into iq1s_grid: 2048 entries whose components take only the values $-1, 0, +1$. The format is, at its core, ternary: of the $3^8 = 6561$ possible ternary 8-vectors, the 2048 most useful ones were selected for the codebook, and the quantizer maps each weight-group to the nearest one.

Per 32-weight group, the layout pairs four qs bytes with one qh word:

one 32-weight group = 4 bytes of qs + 1 uint16 of qh:

  qs         ┌─────────┬─────────┬─────────┬─────────┐
             │ index 3 │ index 2 │ index 1 │ index 0 │  low 8 bits of each
             │ low 8   │ low 8   │ low 8   │ low 8   │  group's grid index
             └─────────┴─────────┴─────────┴─────────┘
  qh    bit:  15 14  12 11  9 8   6 5   3 2   0
             ┌──┬──────┬─────┬─────┬─────┬─────┐
             │δ │scale │ i3  │ i2  │ i1  │ i0  │  3 high bits per index,
             │± │  s   │hi 3 │hi 3 │hi 3 │hi 3 │  3-bit scale, delta sign
             └──┴──────┴─────┴─────┴─────┴─────┘

  grid index of group l  =  qs[l] | (i_l << 8)      11 bits → 2048 entries

Two small pieces of metadata, the scale and δ fields visible in qh above, move the format beyond pure ternary quantization. The dequantization is:

$$x = \underbrace{d \cdot (2s+1)}_{\text{3-bit odd scale}} \cdot \left( g_j + \delta \right), \qquad \delta = \pm 0.125$$

where $s$ is a stored 3-bit scale code, $g_j \in {-1, 0, +1}$ is the $j$-th component of the selected grid vector, and $\delta$ is a shared offset whose sign is a stored bit. So: a scale per 32 weights restricted to odd values $1,3,\dots,15$ (one more saved bit), and a single sign bit that shifts the entire 32-weight group by ±0.125 before scaling. The δ term provides asymmetry: instead of levels ${-1, 0, 1}$, the group gets ${-0.875, 0.125, 1.125}$ or ${-1.125, -0.125, 0.875}$, whichever fits better. It is a type-1 min, compressed to one bit per 32 weights. The ledger:

component	bits/weight
grid indices (11 bits / 8 weights)	1.375
scales (3 bits / 32)	0.09375
delta signs (1 bit / 32)	0.03125
super-scale d	0.0625
total	1.5625

IQ1_M (1.75 bpw) refines it: 4 bits of grid-index-plus-shift per 8 weights instead of shared shifts, scales per 16 weights instead of 32.

typedef struct {
    uint8_t qs[QK_K/8];      // grid index, low 8 bits
    uint8_t qh[QK_K/16];     // grid index high 3 bits + shift bit, per 8 weights
    uint8_t scales[QK_K/32]; // 3-bit block scales, 16 nibbles
} block_iq1_m;

Notice what is missing: there is no field for the fp16 super-scale, a detail I find genuinely elegant. The sixteen 3-bit scales occupy the low bits of the 16 nibbles, and the block scale is stored across the spare high bits, reassembled at decode time:

scale.u16 = (sc[0] >> 12) | ((sc[1] >> 8) & 0x00f0)
          | ((sc[2] >> 4) & 0x0f00) | (sc[3] & 0xf000);

At a budget of 1.75 bits per weight, not even the block scale is given dedicated bytes.

It is important to be realistic about what these formats can do. Applied to a dense 8B model, IQ1_S produces something that still generates fluent text but has lost a large part of the model's capability. The formats exist because of a different regime: very large MoE models, where Unsloth's "1.58-bit" DeepSeek-R1 famously ran on workstations. The redundancy arguments in the placement section below are what make 1.5625 bpw viable, and only in that setting.

TQ1_0, TQ2_0, and friends: when the model is born ternary

The ternary formats come from a separate lineage, even though they occupy the same range of bits per weight. BitNet-style models are trained with ternary weights: the float shadow weights (the full-precision copies the optimizer updates) get snapped to ${-1, 0, +1}$ in the forward pass from the start, so the released weights genuinely contain log₂3 ≈ 1.585 bits of information each. No codebook is needed; only packing.

TQ1_0 (1.6875 bpw) packs 5 trits (ternary digits) per byte: because $3^5 = 243 \le 256$, a byte can hold a 5-digit base-3 number, and the decoder peels digits off with multiply-by-3 operations rather than divisions, using positional base-3 notation as a packing scheme. TQ2_0 (2.0625 bpw) spends 2 bits per trit, accepting one wasted state per weight in exchange for much simpler SIMD unpacking; it is a classic size-versus-speed trade, and on most hardware TQ2_0 is the better choice when it fits in memory. The newer Q1_0 (1.125 bpw: 128-weight blocks, one sign bit per weight) serves models trained with binary weights. None of these formats are general-purpose: applied to a normal bf16-trained checkpoint they produce noise, because those weights carry far more than 1.6 bits of information each. They appear here because the "1.58-bit" vocabulary causes frequent confusion with IQ1_S, which is general-purpose (with calibration, on sufficiently large models). The two families sit close together on the bits-per-weight axis but make entirely different assumptions about the model.

Their declarations, for reference:

typedef struct {                            // TQ1_0, 1.6875 bpw
    uint8_t qs[(QK_K - 4 * QK_K / 64) / 5]; // 5 trits per byte (3^5 = 243 < 256)
    uint8_t qh[QK_K/64];                    // remaining weights, 4 trits per byte
    ggml_half d;
} block_tq1_0;

typedef struct {                            // TQ2_0, 2.0625 bpw
    uint8_t qs[QK_K/4];                     // 2 bits per trit
    ggml_half d;
} block_tq2_0;

typedef struct {                            // Q1_0, 1.125 bpw
    ggml_half d;
    uint8_t qs[QK1_0 / 8];                  // one sign bit per weight
} block_q1_0;

The importance matrix

Everything discussed so far minimizes a proxy for the error we actually care about. Round-to-nearest and the K-quant scale search both minimize plain weight-space distortion, $\sum_i (x_i - \hat x_i)^2$, treating every weight as equally worth preserving. But what matters is downstream: the change in the layer's output. Recall from the notation section that $a_i$ is the activation a weight gets multiplied by and $\hat x_i$ the reconstructed weight; a first-order estimate of the expected squared output error $\mathcal{E}$ weights each reconstruction error by the activation energy it meets:

$$\mathcal{E} = \sum_i \mathbb{E}[a_i^2] ,(x_i - \hat x_i)^2$$

A weight that always meets near-zero activations can be altered freely; a weight sitting on a hot input channel must be preserved, and LLMs notoriously have a few channels with very large activations. The importance matrix (imatrix, Kawrakow, January 2024, arriving in the same burst of work as the i-quants it feeds) is exactly those $\mathbb{E}[a_i^2]$ statistics: run the fp16 model over a calibration corpus with llama-imatrix, accumulate mean squared activation per input column of every weight matrix, save to a file. llama-quantize --imatrix then feeds them into the same scale searches as per-weight importance weights. Despite the name, it's not a matrix per tensor: one importance value per column, a diagonal approximation of the Hessian that GPTQ computes in full, and cheap enough to collect in minutes.

How much it matters depends entirely on how few bits you're spending. At Q6_K it's noise. At Q4 it's a small consistent win. Below 3 bpw it separates a degraded model from an unusable one, and the codebook formats (IQ3_XXS, everything IQ2, everything IQ1) flatly require one: llama-quantize refuses to run without it (except for the embedding and output tensors, which the recipes keep at friendlier types anyway). This is a reasonable requirement, because choosing among 256 lattice patterns with no information about which weights matter amounts to guessing.

Since the imatrix is estimated from data, the data becomes a quality parameter, and this is an active debate. Most public quants calibrate on short Wikipedia chunks; Unsloth argues this misrepresents chat/code/tool workloads and curates 300K–1.5M token calibration sets with long-context conversational data. DwarfStar collects its imatrix by running its own prefill over rendered chat prompts, because what DeepSeek's experts see at inference time is chat-formatted text. And when no imatrix is available, ds4's quantizer falls back to a synthetic importance computed from the weights themselves: column energy, importance[col] = Σ_rows w², a self-calibration heuristic its author describes as good enough for the first working 2-bit GGUFs. The general lesson is that below 3 bits you are never choosing just a quantization format; you are choosing a format together with a calibration dataset.

One caveat applies: calibration is estimation, not optimization toward a benchmark. But it does bias the quantizer toward whatever distribution you fed it; an imatrix built from English prose will underweight the channels that fire on, say, CJK tokens. Overfitting worries are mostly theoretical at these token counts; distribution mismatch is not.

Quantization as a placement problem

All of the preceding threads converge here. The S/M/L recipes, the imatrix, and the gap between file bpw and block bpw all point to the same conclusion: which tensor gets which format matters as much as what any single format does. The frontier of GGUF quantization in 2026 is mostly not new block layouts; it is smarter placement. Three examples, in increasing order of specialization.

llama.cpp's built-in mixes are the first generation of placement: a static, hand-tuned decision tree keyed on tensor role and layer index. It encodes general knowledge (that the attention value projection is sensitive, that ffn_down benefits from promotion on a schedule), but nothing about the specific model being quantized.

Unsloth Dynamic 2.0 replaces the fixed rules with measurement. They report selecting the quant type per tensor per model, guided by calibration and KL-divergence against the fp16 model, a sensible metric since perplexity can hide distributional damage that KLD exposes. Findings from their published benchmarks that match the structural picture above: MoE ffn_up/ffn_gate expert tensors tolerate 3-bit; ffn_down is touchier; and in hybrid-attention models some tiny tensors (their example: ssm_out) blow up KLD if quantized at all while saving almost nothing, so those are left alone. The UD-Q4_K_XL files you now see on every major release are these recipes; their headline claim is Q4-class files with near-Q5 quality, and "essentially lossless" dynamic 4/5-bit on recent MoE releases. These are vendor numbers, but the KLD methodology behind them is public and reproducible.

DwarfStar 4 carries the placement idea to its logical end point: it gives up generality entirely and quantizes a single architecture that its author understands in detail. antirez's ds4 runs DeepSeek V4 Flash (284B total parameters, 13B active) on a 96–128GB MacBook. The recipe, from the README and the GGUF filenames themselves (IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8):

tensor family	type	rationale
routed expert up/gate	IQ2_XXS	bulk of the parameters, redundant across experts
routed expert down	Q2_K	more sensitive than up/gate, still 2-bit class
attention projections	Q8_0	fires every token, load-bearing
shared experts	Q8_0	always active, no redundancy to hide in
output head	Q8_0	most sensitive tensor in any transformer
attention compression machinery	F16	small, critical, not worth touching

This is not uniform 2-bit quantization but deliberately asymmetric quantization: 2-bit where nearly all of the parameters live, and 8-to-16-bit wherever errors would propagate globally. Running the arithmetic backwards, the file size itself reveals the split: ~85GB from 284B parameters is ~2.4 bpw blended, and with the spine at 8.5+ bpw that only balances if the routed experts hold roughly 95% of all parameters, which is exactly what fine-grained MoE looks like. The M3 Max generates at ~21–27 t/s, and there's a q2-q4-imatrix variant that additionally keeps the last six layers' experts at 4-bit: late layers feed the output head directly, so their errors have nowhere to wash out.

Why do routed experts tolerate a treatment that would severely damage a dense model? Three effects combine. Each token activates a few experts of many, so any single expert's quantization noise touches a small fraction of tokens. Experts overlap in function, so ensemble-style averaging smooths individual damage. And the per-token compute path in a fine-grained MoE passes through far fewer quantized parameters than the file size suggests: 13B active of 284B total means the "effective model" a token experiences is mostly the Q8_0/F16 spine plus a thin 2-bit slice. Aggressive MoE quantization works because MoE architectures already activate only a small part of the model for any given token; quantization extends the same principle by making the rarely used parts cheap to store. The same logic, under different branding, drives MindStudio's "Dwarf Star" selective-quantization writeups and Unsloth's big-MoE dynamic quants: keep the spine in high precision, compress the experts, and tune the blend to the target hardware's RAM. That last point deserves emphasis: the target hardware has become a design input. One chooses an average bits-per-weight to fit a specific memory budget, and the placement machinery exists to make that budget survivable.

The floating-point cousins: MXFP4 and NVFP4

One more family, arriving from a different direction: not community compression of bf16 checkpoints, but hardware vendors standardizing tiny floats, with models increasingly being trained or QAT-finetuned to ship in them natively.

MXFP4 (from the OCP Microscaling specification): blocks of 32, and every weight is a genuine 4-bit E2M1 float: one sign bit, two exponent bits, one mantissa bit, giving magnitudes ${0, 0.5, 1, 1.5, 2, 3, 4, 6}$. The block scale is a single E8M0 byte: eight exponent bits, no sign, no mantissa; a pure power of two, $2^{e-127}$.

#define QK_MXFP4 32
typedef struct {
    uint8_t e; // E8M0
    uint8_t qs[QK_MXFP4/2];
} block_mxfp4;

Each nibble of qs is a complete miniature float. Written out:

one FP4 (E2M1) code:

  bit:    3  2   1  0
        ┌───┬─────┬───┐      value = (-1)^s · 2^(e-1) · (1 + m/2)   for e > 0
        │ s │ e e │ m │              (-1)^s · (m/2)                 for e = 0
        └───┴─────┴───┘
         sign  exp  mant

Enumerating the sixteen codes reproduces the magnitude set from above: e = 0 gives 0 and 0.5, e = 1 gives 1 and 1.5, e = 2 gives 2 and 3, e = 3 gives 4 and 6, each with its sign bit.

17 bytes per 32 → 4.25 bpw, the cheapest per-block metadata in this entire post: eight bits, because a power-of-two scale needs no mantissa and multiplication by it is an exponent add. Structurally, MXFP4 is close to IQ4_NL's idea (16 fixed non-uniform levels behind a shared scale), but with the level set dictated by float semantics rather than fitted to weight statistics. E2M1's roughly geometric spacing is the opposite allocation to IQ4_NL's center-dense curve, worth remembering when comparing them at equal bpw. What makes it important despite that: OpenAI released gpt-oss with MoE weights natively in MXFP4 (the model was trained for it, so there is no post-training quantization error to argue about), and hardware accelerates the format directly, from Blackwell tensor cores onward. NVFP4 is NVIDIA's finer-grained variant, also now a ggml type: 16-weight micro-blocks, each with an FP8 (E4M3) scale, which puts mantissa bits in the scale and doubles the scale granularity, at 4.5 bpw.

Its declaration:

typedef struct {
    uint8_t d[QK_NVFP4/QK_NVFP4_SUB]; // UE4M3 scales, one per 16-weight sub-block
    uint8_t qs[QK_NVFP4/2];           // packed 4-bit E2M1 values
} block_nvfp4;

The direction of travel seems clear: the industry converging on "4-bit floats with micro-block scales" as a native training format squeezes the post-training-quantization problem from above. What it does not touch is the below-3-bpw regime: the codebooks, the parity encodings, the placement strategies. That territory stays with the community formats for now.

Choosing a quant in practice

This section condenses the article into practical guidance. Numbers below are measured whole-file values for Llama-3.1-8B from the llama.cpp quantize README. Note once more that file bpw runs above the block-format bpw because of the mix rules:

quant	file bpw	size (GiB)	tier
Q8_0	8.50	7.95	reference-grade; use when RAM is free
Q6_K	6.56	6.14	indistinguishable from fp16 in practice
Q5_K_M / Q5_K_S	5.70 / 5.57	5.33 / 5.21	inexpensive insurance above Q4
Q4_K_M	4.89	4.58	the standard default
Q4_K_S / IQ4_NL	4.67 / 4.68	4.36 / 4.38	tight 4-bit
IQ4_XS	4.46	4.17	best size:quality in the 4-bit class
Q3_K_M / IQ3_S	4.00 / 3.66	3.74 / 3.42	noticeable but usable; prefer IQ3 with imatrix
IQ3_XXS	3.25	3.04	the 3-bit floor worth using
Q2_K	3.16	2.95	worse than IQ3_XXS at similar size; mostly legacy
IQ2_XS / IQ2_XXS	2.59 / 2.38	2.42 / 2.23	imatrix mandatory; dense models suffer
IQ1_M / IQ1_S	2.15 / 2.00	2.01 / 1.87	big-MoE territory only

If you would like independent measurements rather than my tier labels, a recent unified evaluation benchmarks the K-quant and legacy formats on Llama-3.1-8B-Instruct across reasoning, knowledge, instruction-following, and truthfulness tasks, alongside perplexity and CPU throughput.

Rules of thumb that hold up in practice:

• Q4_K_M or UD-Q4_K_XL as default. Two years of collective perplexity/KLD measurement sit behind this. If the model's provider ships calibrated dynamic quants (Unsloth-style), prefer those at equal size.
• Q6_K when quality is the point: evals, quantization baselines, or determining whether the model or the quant is at fault. Q8_0 if you prefer to remove quantization quality from consideration entirely.
• In the 3-bit class, i-quants beat K-quants: IQ3_XXS over Q2_K at nearly the same size is one of the clearest wins in the table; Q2_K survives mostly in recipes (ds4's expert-down tensors) rather than as a whole-file choice.
• Below 3 bpw, insist on an imatrix, and check what it was calibrated on if your workload is code or tool-calling rather than prose.
• Below 2.5 bpw, only MoE. A 2-bit dense 8B model is a curiosity; a 2.4 bpw 284B MoE model is genuinely usable day to day on a MacBook. The ratio of total to active parameters is the number that predicts survival.
• Speed follows size during decode: smaller quants generate faster on the same hardware, near-linearly in bytes moved. If tokens/sec is the constraint, dropping a tier is a performance decision, not just a memory one.
• Mind the KV cache too: -ctk q8_0 -ctv q8_0 is generally free quality-wise and roughly halves cache memory against fp16; that is often the difference for long contexts, and it composes with any weight quant.

Appendix: the whole zoo in one table

Block formats, as defined in ggml-common.h on master today. "Block" is bytes per block of the stated weight count; file bpw will run higher per the mix rules.

type	bpw	block	payload	metadata scheme
Q4_0	4.500	18 B / 32	4-bit uint	fp16 scale
Q4_1	5.000	20 B / 32	4-bit uint	fp16 scale + fp16 min
Q5_0	5.500	22 B / 32	4+1-bit	fp16 scale
Q5_1	6.000	24 B / 32	4+1-bit	fp16 scale + fp16 min
Q8_0	8.500	34 B / 32	int8	fp16 scale
Q2_K	2.625	84 B / 256	2-bit uint	4-bit scales+mins /16, fp16 d+dmin
Q3_K	3.4375	110 B / 256	2+1-bit signed	6-bit scales /16, fp16 d
Q4_K	4.500	144 B / 256	4-bit uint	6-bit scales+mins /32, fp16 d+dmin
Q5_K	5.500	176 B / 256	4+1-bit uint	6-bit scales+mins /32, fp16 d+dmin
Q6_K	6.5625	210 B / 256	4+2-bit signed	int8 scales /16, fp16 d
Q8_K	9.125	292 B / 256	int8	fp32 d + bsums (in-memory only)
IQ1_S	1.5625	50 B / 256	11-bit index / 8 weights	2048-entry ternary grid, 3-bit scales, ±0.125 delta
IQ1_M	1.750	56 B / 256	11-bit index / 8 weights	per-8 deltas, fp16 scale hidden in nibble bits
IQ2_XXS	2.0625	66 B / 256	8-bit index / 8 weights	256-entry E8 grid, 7-bit parity signs, 4-bit scales
IQ2_XS	2.3125	74 B / 256	9-bit index / 8 weights	512-entry grid, parity signs, 4-bit scales /16
IQ2_S	2.5625	82 B / 256	8+2-bit index / 8 weights	1024-entry grid, explicit sign bytes, 4-bit scales
IQ3_XXS	3.0625	98 B / 256	8-bit index / 4 weights	256-entry grid, parity signs, 4-bit scales
IQ3_S	3.4375	110 B / 256	8+1-bit index / 4 weights	512-entry grid, sign bytes, 4-bit scales
IQ4_NL	4.500	18 B / 32	4-bit LUT index	16-value non-uniform table, fp16 scale
IQ4_XS	4.250	136 B / 256	4-bit LUT index	same table, 6-bit scales /32, fp16 d
TQ1_0	1.6875	54 B / 256	trits, 5/byte	fp16 scale; ternary-trained models only
TQ2_0	2.0625	66 B / 256	trits, 2 bits each	fp16 scale; ternary-trained models only
Q1_0	1.125	18 B / 128	1-bit sign	fp16 scale; binary-trained models only
MXFP4	4.250	17 B / 32	FP4 (E2M1)	E8M0 power-of-two scale byte
NVFP4	4.500	36 B / 64	FP4 (E2M1)	FP8 (E4M3) scale per 16

File size from first principles: for each tensor, elements / block_weights × block_bytes, summed, plus a few MiB of GGUF header and metadata. For a quick estimate, params × file_bpw / 8: an 8B at Q4_K_M's measured 4.89 bpw → 8.03B × 4.89 / 8 ≈ 4.9 GB ≈ 4.58 GiB. This matches the table.

Finally, the name-decoding reference:

• Leading Q + digit: scalar quant, digit = payload bits. Trailing _0/_1: legacy, type-0 (scale) / type-1 (scale+min).
• _K: super-blocks of 256 with quantized sub-scales.
• _S / _M / _L: not formats but per-tensor recipes; small/medium/large promotion of sensitive tensors.
• IQ: codebook/lattice quants, imatrix-era. _XXS/_XS/_S/_M: size grades within the bit class. _NL: non-linear lookup table.
• TQ / Q1_0: ternary/binary packing for models trained at that precision; not applicable to normally trained checkpoints.
• UD-: Unsloth Dynamic recipe on top of the underlying types. _XL: their promoted-tensor tier.
• MXFP4 / NVFP4: micro-scaled 4-bit floats, OCP / NVIDIA flavors.

New formats will keep appearing, but the ledger stays the same: payload bits, metadata bits, and increasingly, placement. Once you read a quant file as an allocation of a bit budget across a model's anatomy rather than as a single compression dial, any new name that shows up on Hugging Face can be understood the same way: find the struct, count the bytes, and see where the bits go.