Ollama — run LLMs locally

What Ollama is, architecturally#

Ollama is a Go service wrapping llama.cpp (Gerganov, 2023) as its inference engine, plus a content-addressed model store using an OCI-style manifest and blob layout, plus an HTTP API.

The abstraction it adds over llama.cpp is packaging: a model reference resolves to a manifest listing blobs for weights, template, parameters and licence. Blobs are shared across tags, so derived models created with ollama create cost only the changed layers. This is why a Modelfile that only alters the system prompt adds kilobytes rather than gigabytes.

Weights are stored in GGUF, a single-file format holding tensors plus key-value metadata (architecture, tokenizer, chat template, RoPE parameters). It is memory-mappable, so load time is dominated by page faults rather than parsing.

Quantization, concretely#

Q4_K_M is a k-quant block format. Weights are split into blocks; each block stores 4-bit quantized values plus per-block scale and minimum in higher precision. The _M suffix marks a mixed assignment: attention and feed-forward tensors that are more sensitive get 5 or 6 bits, the rest get 4.

Effective bits per weight land near 4.8 rather than 4.0, because of scale overhead. Memory for weights is therefore:

weight_bytes  ≈  n_params × bits_per_weight / 8

For an 8B model at ~4.8 bits: 8e9 × 4.8 / 8 ≈ 4.8 GB, which matches the observed 4.92 GB for llama3.1:8b.

Perplexity degradation for Q4_K_M on Llama-class models is typically under 1% relative to fp16, and Q8_0 is close to lossless. Below 4 bits the curve steepens sharply, and IQ2-class quants are usable only with importance-matrix calibration. Dettmers and Zettlemoyer (2023) argue 4-bit is near the accuracy-per-bit optimum for a fixed memory budget — a larger model at 4 bits generally beats a smaller model at 8.

The KV cache is the real budget#

Weights are static. The KV cache is what scales with load, and it is what the Developer tab's 6.4 GB versus 1.5 GB observation is measuring:

kv_bytes  =  2 × n_layers × n_kv_heads × head_dim × bytes_per_elem × n_ctx × n_parallel

• 2 — one key tensor and one value tensor per layer.
• n_layers — transformer blocks.
• n_kv_heads — key/value heads. Under grouped-query attention (Ainslie et al., 2023) this is far smaller than the query-head count, which is the single largest reduction available.
• head_dim — dimension per head.
• bytes_per_elem — 2 for f16, 1 for q8_0 cache, 0.5 for q4_0.
• n_ctx — context length.
• n_parallel — concurrent sequences (OLLAMA_NUM_PARALLEL).

For Llama-3.1-8B: 2 × 32 × 8 × 128 × 2 = 131,072 bytes = 128 KiB per token. At 128k context that is 16 GiB of cache for one sequence — larger than the quantized weights by a factor of three.

OLLAMA_KV_CACHE_TYPE=q8_0 halves it, at a small quality cost that is usually invisible. OLLAMA_FLASH_ATTENTION=1 reduces attention working memory, not cache size.

Why CPU inference is slow: it is memory bandwidth, not FLOPs#

Autoregressive decoding at batch size 1 reads every weight from memory to produce one token. Arithmetic intensity is roughly one multiply-accumulate per weight loaded, so the kernel is bandwidth-bound, not compute-bound.

tokens_per_second  ≲  memory_bandwidth / weight_bytes

A laptop with dual-channel DDR4-3200 has about 51 GB/s theoretical, and perhaps 35 GB/s achievable. Against 4.8 GB of weights that ceiling is roughly 7 tokens/second, before any overhead. An RTX 4090 has about 1000 GB/s, and an H100 with HBM3 about 3.35 TB/s. The two-orders-of-magnitude gap in observed speed is almost entirely this ratio.

Prompt processing behaves differently. It is a batched matrix multiply over all prompt tokens at once, so it is compute-bound and far more efficient per token. This asymmetry is why a long prompt costs less wall-clock time than generating an equally long answer.

Speculative decoding (Leviathan et al., 2022; Chen et al., 2023) attacks the bandwidth bound directly: a small draft model proposes k tokens, the target model verifies them in one batched forward pass, and rejection sampling preserves the target distribution exactly. Typical speedups are 2–3× with no quality loss.

Where Ollama is the wrong tool#

Ollama serves requests through llama.cpp's scheduler. It is built for one user, or a handful. It does not implement continuous batching with paged attention, so throughput under concurrency is far below a purpose-built server.

For multi-user serving, vLLM, SGLang or TensorRT-LLM are the correct choices, and the difference at high concurrency is measured in multiples, not percentages. Ollama's design centre is a developer laptop, and it is excellent there.

References#

• Gerganov et al., llama.cpp — github.com/ggml-org/llama.cpp
• Dettmers and Zettlemoyer, The case for 4-bit precision: k-bit inference scaling laws, 2023 — arxiv.org/abs/2212.09720
• Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models, 2023 — arxiv.org/abs/2305.13245
• Leviathan et al., Fast Inference from Transformers via Speculative Decoding, 2022 — arxiv.org/abs/2211.17192
• Frantar et al., GPTQ: Accurate Post-Training Quantization, 2022 — arxiv.org/abs/2210.17323
• Lin et al., AWQ: Activation-aware Weight Quantization, 2023 — arxiv.org/abs/2306.00978

What to learn next#

• vLLM — the same idea built for many users at once, on a GPU.
• Function calling and tools — let a local model run your Python.
• Local LLM assistant — build something with what you now have installed.