A language model in the kernel

A transformer runs in kernel space, in the same address space as the page tables and the disk driver. There is no allocator underneath it that it did not build, no operating system beneath that, and no maths library. The exponentials and reciprocal square roots are written here. The weights are int8, around 570 MB for the default checkpoint, and they are referenced in place in the memory pool the firmware handed over, with no copy on the heap, because copying them would need the heap to be twice their size.

The theme running through everything below is that a model can be wrong without being broken. Position-encoding convention, query-key normalisation, head width, the normalisation epsilon and the pre-tokenizer pattern each produce a network that loads, runs, stays numerically well-behaved and writes fluent English. Nothing faults, no value goes non-finite, and no error is raised. Every one of them was found by comparing against a host-side reference or by reading output that was supposed to contain a known fact, because there was nothing else to find them with.

Two ways Qwen3 differs from Llama, neither of which fails loudly

Its head width is stated in the file. The obvious calculation, hidden size divided by head count, gives 64, and the correct answer is 128, so the query projection is wider than the residual stream and the attention path is a different shape than the arithmetic suggests. And it normalises each head's query and key before the rotation, which most Llama-derived code does not do at all.

Ignore either and the model loads, runs and generates confident nonsense. Both are carried in the converted file's header per checkpoint, and older files still load because their defaults are exactly the Llama ones.

Rotary embeddings, and the convention that cost months

Rotary position embedding encodes position by rotating pairs of dimensions in the query and key vectors by an angle proportional to the position. Attention then depends on relative distance without any position vector being added anywhere.

Which dimensions form a pair is a convention, and there are two in circulation. The reference implementations most checkpoints are trained through pair dimension i with dimension i plus half the head width. The other convention, which appears in several compact C implementations, pairs adjacent dimensions: 0 with 1, 2 with 3.

This kernel used the second one for a long time. Nothing looked broken, because both are norm-preserving rotations by the same set of angles: no NaN, no drift, no error, no warning. The model stays fluent and attends by a scrambled notion of distance, which is indistinguishable from a small model being small. What it cost was the difference between a prompt completing as "The capital of France." followed by blank lines, and the corrected path giving "The capital of France is Paris. Paris is a city known for..."

The flag for the older convention is true only for genuine checkpoints from the C implementation that uses it.

The pre-tokenizer regex is part of the model

A tokenizer is usually treated as a preprocessing detail. It is not. The pattern that splits text before merging carries as much of the model's training distribution as the merge table does, and the checkpoints in use here disagree about it.

One is the GPT-2 pattern. The other spells out the cl100k one, in which a word may be led by any non-alphanumeric character, so an opening bracket and the word after it are one piece. Digits are emitted one at a time, and punctuation swallows the newlines that follow it.

Using the wrong one moved around 12% of tokens on the training corpus. Again with no error: the model is simply fed sequences it never saw during training, and answers slightly worse in a way nothing reports.

So the pattern travels with the checkpoint, and the host-side converter is always run with verification on. It reimplements the kernel's algorithm and diffs it against the reference library, because a tokenizer that is subtly wrong produces text that still looks like text.

Quantisation, and where the time goes

Generation is bound by memory bandwidth. The useful rule is that each token costs roughly one pass over the weights, which makes the 570 MB checkpoint about 4.4 times slower per token than the 135 MB one, and makes almost every optimisation a question about bytes read.

Around 155 MB of that 570 is the output classifier alone. Constrained decoding only ever needs logits for the tokens a grammar can actually reach, so restricting that final matrix multiply is the largest single win available, and the trainer takes exactly it, dequantising the reachable rows once and never touching the int8 classifier again. On the measured decision layer that is 132 rows out of 49,152.

The int8 KV cache, and the one place it is switched off

Keys and values are cached quantised, which is what makes a long context fit in a laptop's heap at all. It also makes the loss piecewise constant in anything upstream of a cached key or value, which is fatal for training: differencing a first-layer query through the quantised forward pass reported a gradient of -0.305 against an analytic value of order 1e-6.

So the training path keeps the cache in full precision. Not a second forward pass. A switch, because two implementations that are supposed to agree do not stay agreeing. Serving still quantises, which makes training against these gradients a straight-through estimate. That is the usual bargain and it is written down here.

The conversation does not end at the context wall

Within a short distance of the trained length the cache becomes a ring, keeping a few attention sinks at the front and evicting the oldest turns behind them. Measured: a 511-slot ring keeping 468 positions, and then a conversation continuing at position 649 through it.

That is only safe because of a coincidence worth knowing. Unwindowed, a position lives at the slot with its own number. Windowed, it lives at a sink offset plus the remainder of everything after the sinks. Those are the same address for as long as the conversation is shorter than the ring, so switching before the ring would first wrap needs no entry re-seated and the buffers are already larger than the new capacity. The switch takes that path only under those conditions and clears the cache otherwise, because the general case genuinely cannot be re-seated.

The first version of this got it wrong in the loudest possible way and the output still looked fine. It cleared unconditionally, so a feature announcing that it now forgets its oldest turns forgot all of them: position fell from 468 to 72 and the model carried on answering fluently. Only the context report showed it.

Pinning the system turn is the same mechanism. A sink is a slot that never recycles, so setting the sink count to the system turn's token length pins the instructions, the applet list and the operator's remembered facts for as long as the conversation runs, in their original slots and with their original rotation angles. Four sinks buy numerical stability; the whole turn buys memory of what the model is.

The count is taken by encoding the system turn the way generation will, with the same beginning-of-sequence token and the same tokenizer. A count that ran short would pin part of a turn and leave the rest to scroll. It is clamped to a third of the trained length, and the report prints span and pinned separately so a clamp shows up as the two disagreeing. The cost is that the recent window is shorter by exactly the system turn: a fifth of the cache at 512 positions, and noise at 8192.

A turn costs the tokens of that turn

Every chat program re-sends its whole history each turn, because the model sits behind an interface and the cache belongs to somebody else. Here the cache is ours and it stays, so the conversation resumes it instead. The tenth exchange is as cheap as the first. Measured across two turns, position went 135 to 150, growing by the new turn alone, where a rebuild would have re-fed the system turn and landed back where it started.

None of that was new machinery. Resuming, saving and restoring context, and setting the window all existed and had simply never been joined up.

The context is deliberately not in the automatic snapshot. A fact told to the model is a few hundred bytes and is carried automatically; parking the cache writes the whole thing as a blob, and this store is append-only, so two turns of a 512-slot cache wrote 16,375 then 19,625 blocks and took half a 27 MiB region. A cache three orders of magnitude larger has no cadence that makes that affordable. Removing the per-turn park took the same measurement from 16,375 blocks to one.

The hybrid, and a format that is walked from the start

The 2B checkpoint is a hybrid: three layers in four run linear attention and the fourth runs ordinary self-attention. That is what keeps its cache small enough for a laptop, and it is also what breaks the file format. There is no single stride to multiply, because consecutive layers hold different tensors, so grouping the file by tensor stops being possible and the body is written layer-major instead.

The layer schedule travels as an explicit bitmap, so a checkpoint that breaks the expected pattern fails loudly.

Mixture-of-experts variants are refused at load. The smallest published one is 71.9 GB, nothing that size reaches a firmware memory pool on this hardware, and a forward pass that could never be run is a forward pass that could never be contradicted. Refusing is the honest position.

The body carries no names, shapes or lengths, so a writer and a reader disagreeing about a single dimension leaves everything after it as perfectly valid float32 garbage. Both readers therefore walk and never seek, and assert that they land on the last byte. The converter makes the same bargain on input: every tensor must be written or explicitly skipped, and anything else is an error.

The emulator cannot run even the small hybrid, 723 MB against a 516 MB synthetic disk, so, to avoid deferring every bug to hardware, a purpose-built miniature is generated that hits every path the real one does: both layer kinds, packed cache indices, partial rotation, more value heads than key heads, grouped queries, and an untied classifier. It prints what the logits should be, and the kernel is asked for the same ones.