The 1.58-bit LLM that still trains in 16 bits

A model weight is just a knob multiplying one signal by another. The neural-network page lets every knob be an ordinary decimal: perhaps 0.1837 or −0.0412. A large language model has billions of them, so merely moving those decimals from memory to the processor becomes expensive. BitNet asks whether the exact decimals are necessary during a forward pass. Its answer is radical: replace each with −1, 0, or +1.

Why “1.58 bits”? A bit has two states. A trit has three, and the information needed to distinguish three equally likely states is log₂(3) ≈ 1.585 bits. That is an information-theoretic limit, not a promise that every program literally allocates 1.585 physical bits. The report's custom GPU kernel, for example, packs four ternary weights into one byte: exactly 2 bits per weight, chosen because unpacking is cheap.

Quantization means replacing a rich set of numbers with a small codebook. BitNet's weight codebook is {−1, 0, +1}, but a layer still needs to remember its overall magnitude. It computes one scale—the mean absolute weight—divides every weight by that scale, rounds, and clips to the three allowed values. Multiplying by the scale later restores the layer's rough amplitude.

def ternary(weights):
    scale = sum(abs(w) for w in weights) / len(weights)
    q = []
    for w in weights:
        value = round(w / scale)
        q.append(min(1, max(-1, value)))
    restored = [value * scale for value in q]
    return scale, q, restored

The site runs the TypeScript version in src/core/quantization/bitnet.ts; Python and C++ above are faithful translations. Take weights [0.2, −0.8, 1.0, −0.1]. Their absmean is 0.525, so the ternary codes are [0, −1, +1, 0] and the restored approximation is [0, −0.525, +0.525, 0]. A test checks those values exactly. Notice the bargain: multiplication by zero deletes a connection; multiplication by ±1 becomes an add or subtract. Notice the loss too: 1.0 and 0.8 now have the same magnitude.

Weights are only half of a linear layer. The incoming values—the activationsproduced for the current token—also need a compact form. BitNet scales each token by its largest absolute activation and rounds to a signed 8-bit integer between −127 and +127. The resulting W1.58A8 layer has ternary weights and 8-bit activations. Its inner loop is an integer dot product; one multiplication after the loop restores the two scales.

def bitlinear(qw, qa, weight_scale, activation_scale, rows, cols):
    output = []
    restore = weight_scale * activation_scale / 127
    for row in range(rows):
        integer_dot = 0
        for col in range(cols):
            integer_dot += qw[row * cols + col] * qa[col]
        output.append(integer_dot * restore)
    return output

This matters because “1.58-bit LLM” can sound as if every number in the model has collapsed to three states. It has not. Embeddings remain higher precision, activations are 8-bit, and the attention machinery still carries caches and normalisation state. On the Transformer page, BitLinear replaces the large projection matrices inside attention and the feed-forward network; it does not replace the entire computation graph.

The report trains BitNet b1.58 2B4T from scratch: roughly 2 billion parameters and 4 trillion pre-training tokens, followed by supervised fine-tuning and direct preference optimisation. Table 2 compares its instruction-tuned checkpoint with Qwen2.5 1.5B in BF16 and two 4-bit post-training formats. These are the authors' numbers, not values produced by our toy.

The most honest reading is not “ternary beats full precision.” It is that a natively trained ternary model can occupy a striking point on the memory–quality frontier. It trails BF16 Qwen on MMLU, leads it on GSM8K and IFEval, and lands 0.71 points lower on the displayed average while using 0.4 GB rather than 2.6 GB of non-embedding weight memory. Different parameter counts, data recipes, and training objectives remain entangled.

Absmean is wonderfully cheap because a whole matrix shares one scale. That also means a few unusually large weights can pull the scale upward. Ordinary weights then fall below the rounding threshold and become zero. The 2025 report does not claim our small stress test; this is a seeded-free, illustrative probe of its quantizer using 24 fixed weights and one fixed activation vector.

In this fixed toy, the no-outlier case starts with 16.7% of weights mapped to zero and 53.2% output NRMSE. At a 32× outlier, the zero share reaches 95.8% and output NRMSE reaches 94.7%. A trained network can adapt its shadow weights around the quantizer, so this is not a prediction of BitNet's benchmark loss. It is the pressure the optimiser must learn around—and why later low-bit work treats outlier channels specially.

That distinction rules out the most tempting shortcut. You cannot generally take an existing BF16 model, round every weight to three values with the function above, and expect the paper's quality. The report contrasts its native training with post-training quantization, and its model consumes four trillion tokens before instruction tuning. The low-bit representation saves persistent inference memory; it does not erase the cost of learning the model in the first place.

Nor does tiny storage automatically mean tiny latency. Commodity GPU libraries are tuned for FP16, INT8, and INT4. The report says W1.58A8 needs a custom CUDA kernel that packs four trits per byte, loads them, unpacks them into on-chip memory, and then computes. Without that path, a generic framework may spend more time converting values than it saves on arithmetic. “1.58-bit” describes the weight alphabet; speed is a property of the entire hardware and software stack.

• Separate representation from runtime. Benchmark the packed checkpoint with the official CPU kernel, then run a naive unpack-and-multiply implementation on the same machine. The gap is the value of systems engineering, not quantization alone.
• Audit end-to-end memory. Increase context length and count embeddings, the key–value cache, activations, and workspace. Weight compression matters less once the cache dominates; the crossover is what an edge-device buyer actually needs.
• Stress outlier channels. Replace one global absmean with per-channel scales, or protect a few outliers at higher precision, and re-run Fig 2. The relevant question is how much metadata buys back how much output fidelity.
• Control the training comparison. Train BF16 and ternary models with the same architecture, tokens, data order, and tuning budget. The report's frontier table is useful for deployment; a controlled ablation is what isolates the quantizer.

Poke at the Transformer and imagine replacing each Q, K, V, output, and feed-forward matrix with the integer loop above. The attention pattern does not change; the alphabet of its largest matrices does. BitNet's surprising result is that three symbols can be enough. Its equally important caveat is that reaching those three symbols still takes high-precision learning and purpose-built execution.