The Wall of Exclamation Marks

A long agent session on my own hardware stopped producing language. Not an error, not a timeout. The model had been thinking for 55 seconds and then emitted this:

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Nothing else. Just exclamation marks until the token budget ran out.

This is the story of finding out why, being confident about the wrong answer for two days, and discovering that the bug was something I had written myself.

Why exclamation marks specifically

The first useful question is not "why did it break" but "why that character".

In this tokenizer, vocab.json maps "!" to token id 0. And the sampler picks the next token with an argmax over the logits. If the logit row is NaN, every comparison against it is false, so the running maximum never updates and the scan returns index 0. Every step. Forever.

So the wall of exclamation marks is not the model being confused. It is a numerical blow-up rendered as text: NaN logits, argmax falls through to index 0, index 0 is "!".

That reframed the problem. I was no longer debugging a language model. I was looking for a source of NaN.

The obvious suspect

The model is Qwen3.8-Flash-Next, served with SGLang across two DGX Spark GB10 boxes. The one exotic thing in the serving config was multi-token prediction: speculative decoding, where a draft head proposes several tokens and the target model verifies them in parallel. I had tuned it in an earlier post and measured a 35 % speedup.

Speculative decoding is also the most intricate part of the stack. So I tested it: same prompts, same server, MTP on versus off.

That looked conclusive. It was even better than conclusive: turning the optimisation off made the model faster in practice, because on long generations the speculation stopped being accepted (accept rate: 0.00 in the server log) and paid the drafting cost for nothing.

I disabled MTP, wrote it up, and moved on.

Being thorough in the wrong direction

Then I tried to get the speed back. If speculation was broken, maybe a gentler setting worked.

About 150 speculative requests, roughly 54 collapses, zero without speculation. I wrote "MTP on this model is a dead end" into the config and meant it.

Every one of those rows is a real measurement. The conclusion drawn from them was wrong.

The statistical trap in the middle

One row deserves its own paragraph, because it nearly cost me a bad deployment.

Rejection sampling first measured 1 collapse in 20, versus 8 in 20 for plain K=1. I ran a Fisher exact test: p = 0.0197. Significant. I wrote that it was "a real fix" and that the corruption had been an acceptance-criterion problem all along.

Then I ran the same configuration at n=40. It came back 15 collapses in 40.

The 5 % was noise. This failure mode is bursty, so small samples swing wildly and flatter whichever change you just made. A significance test does not protect you when the effect you are testing is a lucky draw from a heavy-tailed process. I now have a rule written into the config: never accept a low collapse count below n=40, and always replicate a favourable single measurement before acting on it.

That rule caught a second false positive a day later, when a draft-depth setting looked 3 % faster and did not survive a replicate.

The actual cause

Back in August I had a different problem. Upstream SGLang gated its fast sparse-attention decode path behind a check for SM100 hardware. GB10 reports compute capability sm_121, so the gate refused, and the fallback kernel did not compile on this chip. The model would not serve at all.

So I patched the gate. One line, in a custom Docker image:

if not (is_sm100_supported() or _cap[0] >= 10):
    return None

Widen the condition, take the fast path, ship it. It worked. The model served for weeks.

Twelve days later I pulled a newer upstream image to see whether anything had changed. This is what that same function now looks like:

# This path is numerically validated on SM100 and SM120. Do not widen it
# to every SM12x device: it silently corrupts long-context decode on
# SM121/GB10.
if not (is_sm100_supported() or is_sm120()):
    return None

Upstream had written a comment warning against exactly the change I had made, describing exactly the symptom I had spent two days attributing to speculative decoding.

They had also done the real work: a purpose-built kernel for this chip, qwen38_qsa_sm121_varlen, on a code path that no longer needs patching.

Speculation was the amplifier, not the defect

I served the unpatched upstream image and re-ran everything.

Fisher exact: p = 0.003. On the correct kernel, speculative decoding is clean.

I then swept draft depth again on the fixed image. K=1, K=2 and K=3: zero collapses at every depth, all passing long-context checks to 120k tokens. On the patched image that same sweep produced 40 to 60 % corruption.

That is the cleanest statement of the result. Speculation never was the defect. It amplified an unvalidated kernel by driving more work through it, which is why the corruption scaled with draft volume and looked so convincingly like a speculation bug.

What it costs and what it gives

The final state, on the deployed configuration:

A 30 % speedup, speculation re-enabled, and a custom Docker image retired in favour of upstream code that somebody validated properly.

The part I got wrong, stated plainly

I want to be precise about the mistake, because the interesting failure is not the kernel.

My testing was thorough and my method was sound: controlled arms, adequate samples, significance tests, careful notes. What I never questioned was my own patch. It had been working for weeks, so I filed it under infrastructure rather than under suspects. Everything I tested was downstream of it.

There is also a specific technical lesson. Upstream's warning says the corruption affects long-context decode. Every probe I ran used prompts of about 60 tokens, because that is what reproduced the collapse fastest. Short prompts cannot exercise a long-context bug. When I finally built a probe that walked the context from 2k to 120k tokens with a needle-retrieval check, it took ten minutes to write and would have pointed at the right layer on day one.

Three things I would tell my past self:

The upstream trail

If you are running hybrid linear-attention models with speculative decoding, these are worth reading:

I spent a while convinced my problem was theirs. It was not, but the reading is what eventually pointed me at the kernel gate, so it was not wasted either.

Closing

The model now runs faster than it ever did while corrupting, on less custom code than before, and I understand why. The exclamation marks were the most legible bug report I have had in a while: a single character, repeated, that turned out to encode "your logits are NaN and here is token zero".

The uncomfortable part is that all the discipline in the world pointed at the wrong component, because I never put my own patch on the list.