Hardware · GPU · NVIDIA

NVIDIA GeForce RTX 4090

24 GB of GDDR6X memory at 1,008 GB/s — enough for models up to ~35B parameters at 4-bit on one card.

What it runs

Largest models without offloading on the RTX 4090 workstation (64 GB DDR5), 8K context.

Variant
Qwen2.5 32B Instruct
32.8B · Q4_K_M via llama.cpp
Tight fit
~32 tok/s est.
Variant
Qwen2.5-Coder 32B Instruct
32.8B · Q4_K_M via llama.cpp
Tight fit
~32 tok/s est.
Variant
DeepSeek-R1-Distill-Qwen-32B
32.8B · Q4_K_M via llama.cpp
Tight fit
~32 tok/s est.
Variant
Qwen3 30B-A3B
30.5B · Q4_K_M via llama.cpp
Runs well
~258 tok/s est.
Variant
Gemma 3 27B IT
27.4B · Q4_K_M via llama.cpp
Tight fit
~38 tok/s est.
Variant
Gemma 2 27B IT
27.2B · Q4_K_M via llama.cpp
Runs well
~38 tok/s est.
Variant
Mistral Small 24B Instruct 2501
23.6B · Q6_K via llama.cpp
Runs well
~33 tok/s est.
Variant
Qwen2.5 14B Instruct
14.8B · Q6_K via llama.cpp
Runs well
~52 tok/s est.
Variant
Phi-4
14.7B · Q8_0 via llama.cpp
Runs well
~41 tok/s est.
Variant
Gemma 2 9B IT
9.24B · BF16 via SGLang
Tight fit
~34 tok/s est.

System RTX 4090 workstation (64 GB DDR5)
Runs models up to ~35B parameters at 4-bit entirely on the GPU. What runs

Throughput on systems containing this device.

Model · quantization	System	Runtime	Context	Measurements	Source
Llama 3.1 8B Instruct Q4_K_M (LM Studio Community)	RTX 4090 workstation (64 GB DDR5)	llama.cppcuda · b4600	4K	12,100 tok/s · Prompt processing (512) 128 tok/s · Generation (128)	Mutinai illustrative fixtures
Qwen3 30B-A3B Q4_K_M (Unsloth AI)	RTX 4090 workstation (64 GB DDR5)	llama.cppcuda · b5300	4K	3,400 tok/s · Prompt processing (512) 152 tok/s · Generation (128)	Mutinai illustrative fixtures

Contributions are not open yet, so there is nothing here from members.

Contributions are not open yet, so there is nothing here from members.