Tools · 14 tracked
The software between you and the model.
A local setup is a runtime that loads the model, plus something you type into. Everything else is optional.
Every tool, by the job it does14 of 14
In roughly the order a setup needs them.
6 tools
Runtimes
Loads a model and answers requests. Every setup needs one.
- A fast inference library for running LLMs locally on modern consumer-class GPUsturboderp · PythonPermissiveexl2NVIDIA GPUs
- PermissiveggufNVIDIA GPUsAMD GPUsApple siliconmost GPUs (Vulkan)CPUs
- PermissivemlxApple silicon
- Get up and running with Kimi, GLM, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.Ollama · GoPermissiveggufNVIDIA GPUsAMD GPUsApple siliconCPUs
- SGLang is a high-performance serving framework for large language models and multimodal models.SGLang Project · PythonPermissivesafetensorsNVIDIA GPUsAMD GPUs
- A high-throughput and memory-efficient inference and serving engine for LLMsvLLM Project · Python · 1 measured resultPermissivesafetensorsNVIDIA GPUsAMD GPUs
2 tools
Chat interfaces
A chat window on top of a runtime.
- Desktop app for finding, downloading and chatting with local models, with a built-in local server. Runs GGUF models through llama.cpp, and MLX models on Macs.LM Studio · TypeScriptAny OpenAI-compatible model APILicense unknown
- Any OpenAI-compatible model APIRestricted
2 tools
Coding assistants
Uses a model inside your editor or terminal.
2 tools
Fine-tuning
Adapts a model to your own data.
1 tool
Evaluation
Measures how good a model really is.
- Any OpenAI-compatible model APIPermissive
1 tool
Gateways
One API in front of many models, local or hosted.
- The fastest, litest AI Gateway. Rust core with Python SDK. Call 100+ LLM APIs in OpenAI (or native) format with cost tracking, guardrails, load balancing, and logging [Bedrock, Azure, OpenAI, Anthropic, OpenAI, VertexAI, vLLM, Nvidia NIM]BerriAI · PythonAny OpenAI-compatible model APIPermissive