Hardware · Accelerator · NVIDIA

NVIDIA A100 80GB

80 GB of HBM2e memory at 2,039 GB/s — enough for models up to ~120B parameters at 4-bit on one card.

What it runs

Largest models without offloading on the A100 80GB server (512 GB DDR4), 8K context.

Variant
Llama 3.1 70B Instruct
70.6B · Q4_K_M via llama.cpp
Runs well
~30 tok/s est.
Variant
Llama 3.3 70B Instruct
70.6B · Q4_K_M via llama.cpp
Runs well
~30 tok/s est.
Variant
Mixtral 8x7B Instruct v0.1
46.7B · Q4_K_M via llama.cpp
Runs well
~157 tok/s est.
Variant
Qwen2.5 32B Instruct
32.8B · BF16 via SGLang
Runs well
~20 tok/s est.
Variant
Qwen2.5-Coder 32B Instruct
32.8B · Q8_0 via llama.cpp
Runs well
~38 tok/s est.
Variant
DeepSeek-R1-Distill-Qwen-32B
32.8B · BF16 via SGLang
Runs well
~20 tok/s est.
Variant
Qwen3 30B-A3B
30.5B · Q8_0 via llama.cpp
Runs well
~328 tok/s est.
Variant
Gemma 3 27B IT
27.4B · BF16 via SGLang
Runs well
~24 tok/s est.
Variant
Gemma 2 27B IT
27.2B · BF16 via SGLang
Runs well
~24 tok/s est.
Variant
Mistral Small 24B Instruct 2501
23.6B · BF16 via SGLang
Runs well
~28 tok/s est.

System A100 80GB server (512 GB DDR4)
Runs models up to ~120B parameters at 4-bit entirely on the GPU. What runs

Throughput on systems containing this device.

Model · quantization	System	Runtime	Context	Measurements	Source
Qwen2.5 32B Instruct AWQ 4-bit	A100 80GB server (512 GB DDR4)	vLLMcuda · 0.7.3	8K	48 tok/s · Generation throughput 72 GB · Peak memory 5,200 tok/s · Prompt throughput 95 ms · Time to first token	Mutinai illustrative fixtures

Contributions are not open yet, so there is nothing here from members.

Contributions are not open yet, so there is nothing here from members.

No source records linked.