Hardware · GPU · NVIDIA

NVIDIA GeForce RTX 5090

32 GB of GDDR7 memory at 1,792 GB/s — enough for models up to ~47B parameters at 4-bit on one card.

What it runs

Largest models without offloading on the RTX 5090 workstation (96 GB DDR5), 8K context.

Variant
Mixtral 8x7B Instruct v0.1
46.7B · Q4_K_M via llama.cpp
Tight fit
~138 tok/s est.
Variant
Qwen2.5 32B Instruct
32.8B · Q5_K_M via llama.cpp
Runs well
~49 tok/s est.
Variant
Qwen2.5-Coder 32B Instruct
32.8B · Q4_K_M via llama.cpp
Runs well
~57 tok/s est.
Variant
DeepSeek-R1-Distill-Qwen-32B
32.8B · Q4_K_M via llama.cpp
Runs well
~57 tok/s est.
Variant
Qwen3 30B-A3B
30.5B · Q4_K_M via llama.cpp
Runs well
~458 tok/s est.
Variant
Gemma 3 27B IT
27.4B · Q4_K_M via llama.cpp
Runs well
~67 tok/s est.
Variant
Gemma 2 27B IT
27.2B · Q4_K_M via llama.cpp
Runs well
~68 tok/s est.
Variant
Mistral Small 24B Instruct 2501
23.6B · Q6_K via llama.cpp
Runs well
~59 tok/s est.
Variant
Qwen2.5 14B Instruct
14.8B · Q6_K via llama.cpp
Runs well
~92 tok/s est.
Variant
Phi-4
14.7B · Q8_0 via llama.cpp
Runs well
~72 tok/s est.

System RTX 5090 workstation (96 GB DDR5)
Runs models up to ~47B parameters at 4-bit entirely on the GPU. What runs

Throughput on systems containing this device.

Model · quantization	System	Runtime	Context	Measurements	Source
Qwen2.5 32B Instruct Q4_K_M	RTX 5090 workstation (96 GB DDR5)	llama.cppcuda · b4600	4K	2,900 tok/s · Prompt processing (512) 61 tok/s · Generation (128)	Mutinai illustrative fixtures

Contributions are not open yet, so there is nothing here from members.

Contributions are not open yet, so there is nothing here from members.