Ternary 27B on 16GB VRAM, or: the smaller file is not the faster one

The internet is losing its mind over “1-bit” models again, so when I saw Ternary Bonsai 2 27B sitting there as a 5.9GB gguf, the RTX A5000 in my laptop had to earn its keep again. Same 16GB card that carried the Gemma 4 experiment, same question: how much of this thing can I actually use.

Spoiler for the impatient: PQ2_0 packing, 128K context, ~30 tok/s on short prompts, 22 tok/s after a 65K-token prompt.

First thing I learned before even downloading: these files need Prism’s fork of llama.cpp. Custom packing types plus a matching Hadamard activation transform, so your regular llama-server won’t work. .

And the “1-bit” label deserves a squint. This is a ternary model, weights have three values. The two published packings are ~1.75 bits/weight (PTQ1_0, 5.95GB) and ~2.13 bits/weight (PQ2_0, 7.21GB). Two representations of the same ternary weights, not a 1-bit model fighting a 2-bit model.

How much context actually fits

The file is 5.95GB and the card has 16GB, so obviously tons of room, right. Except the KV cache at f16 grows with context and at 131K tokens PTQ alone allocates an 8192 MiB CUDA KV buffer. Plus compute buffers, plus the CUDA context itself.

Results:

(Also, -ngl 99 doesn’t mean the model has 99 layers, its just “offload everything”. Verbose logs said 65/65 layers on GPU. I mention this because I briefly confused myself with it.)

I settled PTQ at 147,456 for headroom and left it there for benchmarking.

The PTQ numbers

Synthetic test prompts: a unique timestamped marker, the word " alpha" repeated to hit a target size, and the instruction “Return exactly: OK”. Temperature 0, one request at a time through the API. Heres what PTQ did as the filled context grew (configured 147K, one slot):

That last row: 581 seconds to prefill. Nine point seven minutes of the GPU chewing before it produced a single token, then 16 tok/s after. So theres the practical distinction between fitting a 131K prompt in VRAM and using a 131K prompt interactively. Both are real achievements, only one of them is fun.

PQ2: bigger file, faster model

Then I loaded PQ2_0. Same flags, 131,072 configured context, ~601 MiB free. The comparison:

The 1.26GB bigger packing is faster across the board on this card. I did not see that coming and neither will you if you only compare file sizes, which is exactly why im writing this down.

Caveats, stated plainly because they matter: these are single-run synthetic measurements, not a benchmark suite. PQ2 was measured with ~1K and ~64K filled prompts; the 128K configured context is capacity, not a measured 128K-filled result. Don’t copy PTQ’s 16 tok/s at 131K into a PQ2 chart, I didn’t test that.

I still picked PQ2 because I prefer speed, and its running at 131,072 tokens with roughly 600 MiB to spare.

The final setup

systemd-run --user --unit=bonsai-ternary --collect \
  ~/.local/src/prism-llama.cpp/build/bin/llama-server \
  -m ~/.local/share/bonsai/Ternary-Bonsai-2-27B-PQ2_0.gguf \
  --host 0.0.0.0 --port 8080 \
  -ngl 99 -c 131072 -fa on --parallel 1

• -ngl 99 offloads all 65 layers to the GPU
• -c 131072 is the context budget for the single slot
• -fa on flash attention, not optional at these lengths
• --parallel 1 one serving slot, concurrency untested
• KV cache stays at default f16, batch defaults (2048/512) untouched

Two things to know. First, if your card also drives a display, start at 65,536 and work up; I had ~600 MiB free with a headless-ish setup and your desktop will want its share. Second, the context is shared between input and output.