Menu
mutinaiPreview: Public preview · figures are illustrative fixture data, attributed to their source · data to May 2025 · read-only
What can I run?
Tool · Inference runtime · ggml.org

llama.cpp

LLM inference in C/C++

License
PermissiveMIT License
Language
C++
Latest release
4 Nov 2026llama.cpp v0.9.0
Measurements
11with this runtime
RepositoryWebsite

Relationships

  • Benchmarkllama-benchImplements
  • ToolOllamaFoundation for

Releases & news

  1. llama.cpp v0.9.0
    This release adds OCP FP8 E4M3 quantization support, a new flash_attn_ext_rows op that makes the unified KV cache efficient for multi-sequence inference, and a generic batched tensor copy API. Four new architectures are supported (Clover, Limite, Metis-1.6 and Mistral Large 4), a
  2. llama.cpp v0.8.0
    This release adds the LiquidAI LFM3 family (text, vision, audio and omni) together with a new experimental streaming-audio API in mtmd, and wires realtime audio in and out of the server through the new /v1/realtime and /tts endpoints. A batch of new architectures landed: DeepSeek
  3. llama.cpp v0.7.0
    llama.cpp v0.7.0 adds the LiliCorr speculative decoding technique and MTP for GLM5-Next, a new GPU cache for MoE experts kept in host memory (--moe-cache-mib, up to ~2.2x decode speedup when experts are offloaded), and a large batch of new models: K2 Horizon (dense and MoVA), Pri
  4. llama.cpp v0.6.0
    llama.cpp v0.6.0 introduces the new llama_batch_ext extended batch API (with llama_process) for mixed token/embedding inputs and MTP/deepstack state embeddings, adds support for the GLM-5.3-Flash (GLM5-Next) 320B hybrid model, the Clef decision model (text and vision) and MTP spe
  5. llama.cpp v0.5.0
    This release focuses on backend performance and correctness, broader model coverage, and more robust server/router operation. It adds HRM-Text (DFM Mimir 1B) support, MiMo-V2.6 and HunyuanOCR conversion support, ggml 0.25.0 backend improvements, multi-address HTTP binding, image
  6. llama.cpp v0.4.1
    llama.cpp 0.4.1 adds Maple 20B-A1B, Tencent Hy 4, and Spark2.5 support, improves JSON schema handling, chat parsing, logging, and server child-process management, and updates ggml to v0.24.0.
  7. llama.cpp b5450
    Automated build release.
  8. Official Qwen3 GGUF quantizations published
    The Qwen team published first-party GGUF files for Qwen3 models.

Measured performance with this runtime

Throughput members and projects recorded, in tokens per secondHow fast the model writes its answer; about 10 tokens per second reads like comfortable typing..

Model · quantizationSystemRuntimeContextMeasurementsSource
Llama 3.1 8B Instruct Q4_K_M (LM Studio Community)CPU-only Ryzen 9 7950X (128 GB DDR5)llama.cppcpu · b46004K
92 tok/s · Prompt processing (512)
12.4 tok/s · Generation (128)
Mutinai illustrative fixtures
Llama 3.3 70B Instruct Q4_K_M (LM Studio Community)Dual RTX 3090 (128 GB DDR5)llama.cppcuda · b46004K
390 tok/s · Prompt processing (512)
17.6 tok/s · Generation (128)
Mutinai illustrative fixtures
Llama 3.1 8B Instruct Q4_K_M (LM Studio Community)MacBook Pro M3 Max 64 GBllama.cppmetal · b46004K
760 tok/s · Prompt processing (512)
54 tok/s · Generation (128)
Mutinai illustrative fixtures
Llama 3.3 70B Instruct Q4_K_M (LM Studio Community)MacBook Pro M3 Max 64 GBllama.cppmetal · b46004K
72 tok/s · Prompt processing (512)
7.4 tok/s · Generation (128)
Mutinai illustrative fixtures
Llama 3.3 70B Instruct Q4_K_M (LM Studio Community)Mac Studio M2 Ultra 192 GBllama.cppmetal · b46004K
135 tok/s · Prompt processing (512)
12.1 tok/s · Generation (128)
Mutinai illustrative fixtures
Qwen2.5 14B Instruct Q4_K_MRTX 4060 Ti 16GB budget build (32 GB DDR5)llama.cppcuda · b46004K
1,250 tok/s · Prompt processing (512)
25.5 tok/s · Generation (128)
Mutinai illustrative fixtures
Llama 3.1 8B Instruct Q4_K_M (LM Studio Community)RTX 4090 workstation (64 GB DDR5)llama.cppcuda · b46004K
12,100 tok/s · Prompt processing (512)
128 tok/s · Generation (128)
Mutinai illustrative fixtures
Qwen3 30B-A3B Q4_K_M (Unsloth AI)RTX 4090 workstation (64 GB DDR5)llama.cppcuda · b53004K
3,400 tok/s · Prompt processing (512)
152 tok/s · Generation (128)
Mutinai illustrative fixtures
Qwen2.5 32B Instruct Q4_K_MRTX 5090 workstation (96 GB DDR5)llama.cppcuda · b46004K
2,900 tok/s · Prompt processing (512)
61 tok/s · Generation (128)
Mutinai illustrative fixtures
Llama 3.1 8B Instruct Q4_K_M (LM Studio Community)RX 7900 XTX desktop (64 GB DDR5)llama.cpprocm · b46004K
3,050 tok/s · Prompt processing (512)
94 tok/s · Generation (128)
Mutinai illustrative fixtures
Qwen3 30B-A3B Q4_K_M (Unsloth AI)Ryzen AI Max+ 395 mini PC (128 GB)llama.cppvulkan · b53004K
420 tok/s · Prompt processing (512)
51 tok/s · Generation (128)
Mutinai illustrative fixtures

Reviews

Contributions are not open yet, so there is nothing here from members.

Sources & history

Sources

  • GitHub (live source, 203 records, 8 Nov 2026)
  • GitHub (fixture) (illustrative fixture, 1 record, 13 Sept 2026)
  • Mutinai illustrative fixtures (illustrative fixture, 1 record, 13 Sept 2026)

External identifiers

Field history

  • homepageUrl = "https://llama.app" — GitHub (current)
  • homepageUrl = "https://llama.app" — GitHub
  • homepageUrl = "https://llama.app" — GitHub
  • homepageUrl = "https://llama.app" — GitHub
  • homepageUrl = "https://llama.app" — GitHub
  • homepageUrl = "https://llama.app" — GitHub
  • homepageUrl = "https://llama.app" — GitHub
  • homepageUrl = "https://llama.app" — GitHub
  • homepageUrl = "https://llama.app" — GitHub
  • homepageUrl = "https://llama.app" — GitHub
  • homepageUrl = "https://llama.app" — GitHub
  • homepageUrl = "https://llama.app" — GitHub