Tools · 14 tracked
The software between you and the model.
A local setup is a runtime that loads the model, plus something you type into. Everything else is optional.
Runtimes6 of 14
6 tools
Runtimes
Loads a model and answers requests. Every setup needs one.
- A fast inference library for running LLMs locally on modern consumer-class GPUsturboderp · PythonPermissiveexl2NVIDIA GPUs
- PermissiveggufNVIDIA GPUsAMD GPUsApple siliconmost GPUs (Vulkan)CPUs
- PermissivemlxApple silicon
- Get up and running with Kimi, GLM, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.Ollama · GoPermissiveggufNVIDIA GPUsAMD GPUsApple siliconCPUs
- SGLang is a high-performance serving framework for large language models and multimodal models.SGLang Project · PythonPermissivesafetensorsNVIDIA GPUsAMD GPUs
- A high-throughput and memory-efficient inference and serving engine for LLMsvLLM Project · Python · 1 measured resultPermissivesafetensorsNVIDIA GPUsAMD GPUs