Ternary 27B on 16GB VRAM, or: the smaller file is not the faster one
The internet is losing its mind over “1-bit” models again, so when I saw Ternary Bonsai 2 27B sitting there as a 5.9GB gguf, the RTX A5000 in my laptop had to earn its keep again. Same 16GB card that carried the Gemma 4 experiment, same question: how much of this thing can I actually use. Spoiler for the impatient: PQ2_0 packing, 128K context, ~30 tok/s on short prompts, 22 tok/s after a 65K-token prompt. ...