Increased Feasability of Local LLM Infererence On Consumer Hardware

Local LLM inference on consumer hardware has been in a weird place for a while. You can certainly run smaller, quantized models locally, but they are very often less intelligent than a brick. Hardly anyone is using them for any kind of serious, quality work, not because they don't want to, but simply because the models are too dumb. Sure, you can run an LLM locally with frontier-level intelligence, but you're going to end up needing to spend thousands (likely even tens of thousands) of dollars on hardware dedicated solely to LLM inference, and at that point it's really no longer (normal) consumer hardware, which is my point.

This has, very slowly, been changing. PrismML, an AI startup based in California founded by Caltech researchers focused on maximizing intelligence per gigabyte, recently released an LLM called Bonsai 27B, which is based on Alibaba's Qwen3.6 27B, a 27.8 billion parameter multimodal modal. In standard 16-bit form, its weights alone occupy roughly 54 GB of storage and require around 54 GB of RAM/vRAM to load. PrismML reduced the weight footprint to 5.9 GB for its ternary version and 3.9 GB for its 1-bit version, massively reducing the barrier to entry for running (somewhat) capable models on hardware people already own. Bonsai 27B is relatively good at tool-calling and general (but simple) reasoning, though you're not going to use it for serious, quality, medium-to-large scale development work, or really any work of any kind.

A recent advance in local AI is Colibri, which allows you to run a frontier model (Z.ai's GLM 5.2) with 744 billion parameters on consumer hardware, albeit very slowly. It makes use of the fact that GLM 5.2, despite its enormous total size, only actives 40B parameters per token, and only ~11 GB of those (the routed experts) actually change from token to token. Colibri keeps the dense, always-used part of the model resident in RAM (~9.9 GB at int4), and streams the other 19,456 experts off of the disk on demand, using a router-lookahead thread that predicts which experts the next layer will need with ~72% accuracy, in addition to a cache that pins whichever experts your own usage pattersn hit most. This allows a 744B frontier model to run on a machine with a measly 25 GB of RAM. It is, as mentioned, brutally slow on such a machine (just 0.05-0.1 tok/s), but it runs, which is incredible. The tok/s speed can be scaled up by adding GPUs or more RAM.

A similar method of running large LLMs on low-spec consumer hardware is through llama.cpp's memory-mapping (mmap) support. I built a lightweight Rust tool called tinyinference, a llama.cpp wrapper that makes this as straightforward as possible. Rather than loading a model's weights fully into RAM, mmap lets the OS treat the GGUF file on disk as if it were memory, paging pieces in as they're touched and evicting them under normal virtual memory rules. That's why it can run a large model like GPT-OSS-120B on a low-RAM machine at all: the file simply needs to exist on disk.

Tinyinference is a much simpler, more general tool than Colibri. It works with any GGUF model and doesn't require a custom inference engine, but it inherits llama.cpp's default page-fault behavior rather than exploiting MoE-specific structure. So running GLM-5.2 through Colibri will be faster than running it through tinyinference, since Colibri is purpose-built around GLM-5.2's routing, but tinyinference will let you run any GGUF model.

None of these fully solve the original problem: quantization sacrifices intelligence, and disk-streaming (whether MoE-aware or generic mmap) sacrifices speed (to a very significant extent). But, together, they mark a meaningful difference. A year ago, running a frontier-class model on hardware you already owned wasn't really on the table at all, but now, it's slow and impractical rather than impossible, and that gap, based on my observations, tends to close fast once people start building in this space. Bonsai 27B, Colibri, and tinyinference are each optimizing a different axis of the same tradeoff, weight footprint, memory hierarchy, and accessibility, and none of them get you a fast, smart, cheap setup yet. But you only need one of those axes to move a lot further before "local frontier inference on consumer hardware" stops being a caveat-heavy demo and starts being a real option.