Running Oversized GGUF Language Models On Low-RAM Hardware Using System Memory Mapping

I briefly covered the idea of running oversized language models on your own hardware in my previous post, but I think it's deserving of its own. The problem with local LLMs right now is that they're too dumb for regular people to run on their own hardware with any serious work in mind, because most people don't have a 256 GB RAM machine or an H100 enterprise GPU sitting around to offload inference to.

This issue has been improving, and we can certainly expect to see high-efficiency, reasonably intelligent LLMs running on local hardware in the near future, but something most people miss is that you can run giant LLMs with frontier-level intelligence on your own hardware (provided your laptop isn't 15 years old). First, I'll give some context. Traditionally, an LLM must have its weights remain resident in your dGPU VRAM (and/or system RAM), however, llama.cpp has a memory mapping (mmap) toggle which allows your system to, in essence, only load pages containing specific MoE model weights into memory when the inference engine asks for them, and keeps them cached there until eviction becomes necessary to prioritize other processes or fetch new weights. The convenient aspect of mmap is that it's managed entirely by your OS kernel. It links your weights on the disk into the inference engine's address space. In simple terms, this loads and caches new weights into RAM as needed, and evicts unused ones as needed, allowing you to run LLMs that wouldn't ordinarily be able to remain fully resident in RAM.

Now, a few things. I said you can run such models on consumer hardware, I didn't say it'd be a good idea. Thrashing is a problem here, where repeated page faults, disk reads, and page eviction (repeatedly loading and evicting the same model weights constantly) end up using a lot more compute and disk I/O than wanted, because there is not enough RAM in your machine to keep all of the actively used weights cached at the same time. Simply, it keeps fetching, discarding, and re-fetching small parts, one at a time whenever needed, over and over again because there isn't enough space to keep all of them at the same time. This is inevitable for giant LLMs on low-RAM machines, and it typically results in painfully slow inference, often under 5 tokens/second, but many will argue that the ability for a near-frontier-level model to run at all on a measly little laptop is worth something. For example, you could use it for overnight agentic programming tasks that are relatively low priority. It's free inference, after all (if you don't count electricity costs).

Another thing. When I said "only warm MoE model weights into memory when needed" I specified MoE (Mixture of Experts) for a reason. Unlike dense models, which use all of their weights for every token, MoE models activate a subset of experts at a time. This allows inactive experts to remain on the disk while only the pages belonging to the selected experts are pulled into memory. A dense model with 120B parameters run via mmap will be considerably slower (and this is an understatement) on low-RAM machines compared to an MoE model with 120B total parameters (but fewer active parameters).

With a reasonable balance, you can certainly run, at a non-painful speed, LLMs that wouldn't ordinarily be able to run on your hardware. I created tinyinference, a specialized llama.cpp wrapper using Rust, focusing on low-RAM machines, to do everything I mentioned above in a seamless manner, and I was able to run GPT-OSS-120B (MXFP4) on a 24GB RAM MacBook Pro with an M4 Pro chip, using only CPU inference, and achieved a peak of ~4.5 tokens/second. That doesn't seem very meaningful until you realize that, on your MacBook, you can get free o4-mini-level reasoning with no usage limits. I was also able to run much larger, more intelligent models, such as the new DeepSeek V4 Flash (0731), which outperforms GLM 5.2 and Claude Opus 4.6 in benchmarks, but it goes without saying that the speed is much worse (around 2 tokens/second on my MacBook Pro).

mmap and tinyinference are especially useful for machines that just barely don't have enough memory to run an LLM. Because you almost have enough RAM but not quite, most of the actively used weights can remain warmed up/resident in RAM while only some get fetched and evicted. This results in performance very close to if you had the LLM 100% resident in RAM on CPU inference. It's also worth mentioning that you can, via llama.cpp, offload inference to a small dGPU and, then (basically) have the rest spill over into system RAM as cached pages. Of course, if the model is still too big for your RAM even after partial offload to dGPU, the system will evict and fetch weights as needed. I've made this profile a preset in tinyinference, though it's blocked on machines with a unified memory architecture, because the CPU and GPU share the same physical memory pool, so GPU offloading does not provide a separate VRAM tier and will simply crash the system if you don't have enough unified memory (which, if you did, wouldn't be using any of these methods/workarounds).

Oversized LLMs can, in summary, be run on consumer hardware, but it's going to be slow if the model is truly absurdly large compared to your hardware, and if you don't have a realistic balance. GPT-OSS-120B is runnable on machines with as little as 24 GB of RAM and can achieve speeds of 4.5 tok/s on an M4 Pro, and potentially much higher on superior chips. DeepSeek-V4-Flash is runnable on a 14900KS with 64GB of RAM and an NVMe drive at speeds of 8 tok/s or higher (as per tests I conducted). You just need to be careful, because you're not going to run a 2T parameter model on an 8GB RAM Dell laptop from 2019 for any practical purpose. Technically, of course, it will run but the TTFT alone will be a million years. Custom inference engines built from scratch and centered around a specific model architecture (Colibri is a major example), can, in select cases, optimize oversized inference on low-RAM machines, but not by a life-changing amount, at least not yet.

Open weight LLMs are critical pieces of technology that must remain available and easily accessible, and I think it's worth ending this post with a line from my previous post:

A year ago, running a frontier-class model on hardware you already owned wasn't really on the table at all, but now, it's slow and impractical rather than impossible, and that gap, based on my observations, tends to close fast once people start building in this space.