Overview
It is designed to make large, capable LLMs runnable on low-spec, low-RAM machines without a GPU, using CPU inference and file-backed model weights. It will not be fast, in fact, it will often be painfully slow. The point is that it runs at all on low-spec hardware, which is pretty cool.
Requirements
- Rust
llama-serverfrom llama.cpp- A GGUF model available locally or on Hugging Face
llama-server must be on PATH, or you can set its full executable path from tinyinference's Configure screen.
Configuration
Press c to configure the model, llama-server path, host, port, and runtime settings. Press Enter on a value to type an exact value; use s to save.
Switch Model source, then edit the next row to enter either a Hugging Face owner/model repository or a full local .gguf path. A local model's size is detected automatically.
When the model row is selected, press r to cycle through up to eight recently used Hugging Face repositories and local GGUF paths. Recent models are stored with the rest of the configuration when you save.
Settings are saved in the platform configuration directory. Advanced llama.cpp options can be added through server.extra_args.
Statistics and clipboard
The statistics screen shows endpoint state, PID, uptime, process CPU and resident RAM, plus request and token counters and throughput from llama-server. tinyinference enables llama.cpp's local metrics endpoint for managed servers; metrics remain marked unavailable until the server is ready. Clipboard copy uses clip.exe on Windows, pbcopy on macOS, and wl-copy, xclip, or xsel on Linux.
How low-RAM operation works
With mmap, model weights are read-only file-backed pages, so the GGUF file does not need to fit entirely in resident RAM. RAM is still needed for the KV cache, compute buffers, and server state; with little RAM, storage I/O can make inference extremely slow. gpt-oss-120b has 117B total parameters but activates 5.1B per token, so it is a good default choice for tinyinference.
The server binds to 127.0.0.1 by default and has no authentication. Configure authentication and firewalling before exposing it to a network.
