Hardware · GPU · NVIDIA

NVIDIA GeForce RTX 3090

24 GB of GDDR6X memory at 936 GB/s — enough for models up to ~35B parameters at 4-bit on one card.

What it runs

Largest models without offloading on the Dual RTX 3090 (128 GB DDR5), 8K context.

Variant
Llama 3.1 70B Instruct
70.6B · Q4_K_M via llama.cpp
Tight fit
~13 tok/s est.
Variant
Llama 3.3 70B Instruct
70.6B · Q4_K_M via llama.cpp
Tight fit
17.6 tok/s measured
Variant
Mixtral 8x7B Instruct v0.1
46.7B · Q4_K_M via llama.cpp
Runs well
~69 tok/s est.
Variant
Qwen2.5 32B Instruct
32.8B · Q5_K_M via llama.cpp
Runs well
~24 tok/s est.
Variant
Qwen2.5-Coder 32B Instruct
32.8B · Q8_0 via llama.cpp
Runs well
~16 tok/s est.
Variant
DeepSeek-R1-Distill-Qwen-32B
32.8B · Q4_K_M via llama.cpp
Runs well
~28 tok/s est.
Variant
Qwen3 30B-A3B
30.5B · Q8_0 via llama.cpp
Runs well
~143 tok/s est.
Variant
Gemma 3 27B IT
27.4B · Q4_K_M via llama.cpp
Runs well
~33 tok/s est.
Variant
Gemma 2 27B IT
27.2B · Q4_K_M via llama.cpp
Runs well
~34 tok/s est.
Variant
Mistral Small 24B Instruct 2501
23.6B · Q6_K via llama.cpp
Runs well
~29 tok/s est.

System Dual RTX 3090 (128 GB DDR5)
Runs models up to ~73B parameters at 4-bit entirely on the GPU. What runs

Throughput on systems containing this device.

Model · quantization	System	Runtime	Context	Measurements	Source
Llama 3.3 70B Instruct Q4_K_M (LM Studio Community)	Dual RTX 3090 (128 GB DDR5)	llama.cppcuda · b4600	4K	390 tok/s · Prompt processing (512) 17.6 tok/s · Generation (128)	Mutinai illustrative fixtures

Contributions are not open yet, so there is nothing here from members.

Contributions are not open yet, so there is nothing here from members.