GPUs and Macs for running AI models locally
Pick your hardware to see every open model that fits, the best quant, and an estimated speed ceiling. Or browse by VRAM size, the Apple Silicon guide, or head-to-head GPU comparisons (RTX 3060 vs 4060, RTX 4090 vs 5090, RX 7900 XTX vs RTX 4090 and more).
Consumer GPUs
| Hardware | Memory | Usable | Bandwidth | Models that fit (8K) |
|---|---|---|---|---|
| RTX 3060 12GB | 12 GB | 11.4 GB | 360 GB/s | 173 |
| RTX 3060 Ti | 8 GB | 7.6 GB | 448 GB/s | 161 |
| RTX 3070 | 8 GB | 7.6 GB | 448 GB/s | 161 |
| RTX 3080 10GB | 10 GB | 9.5 GB | 760 GB/s | 165 |
| RTX 3080 12GB | 12 GB | 11.4 GB | 912 GB/s | 173 |
| RTX 3090 | 24 GB | 22.8 GB | 936 GB/s | 209 |
| RTX 3090 Ti | 24 GB | 22.8 GB | 1008 GB/s | 209 |
| RTX 4060 | 8 GB | 7.6 GB | 272 GB/s | 161 |
| RTX 4060 Ti 8GB | 8 GB | 7.6 GB | 288 GB/s | 161 |
| RTX 4060 Ti 16GB | 16 GB | 15.2 GB | 288 GB/s | 200 |
| RTX 4070 | 12 GB | 11.4 GB | 504 GB/s | 173 |
| RTX 4070 Super | 12 GB | 11.4 GB | 504 GB/s | 173 |
| RTX 4070 Ti | 12 GB | 11.4 GB | 504 GB/s | 173 |
| RTX 4070 Ti Super | 16 GB | 15.2 GB | 672 GB/s | 200 |
| RTX 4080 | 16 GB | 15.2 GB | 717 GB/s | 200 |
| RTX 4080 Super | 16 GB | 15.2 GB | 736 GB/s | 200 |
| RTX 4090 | 24 GB | 22.8 GB | 1008 GB/s | 209 |
| RTX 5060 | 8 GB | 7.6 GB | 448 GB/s | 161 |
| RTX 5060 Ti 16GB | 16 GB | 15.2 GB | 448 GB/s | 200 |
| RTX 5070 | 12 GB | 11.4 GB | 672 GB/s | 173 |
| RTX 5070 Ti | 16 GB | 15.2 GB | 896 GB/s | 200 |
| RTX 5080 | 16 GB | 15.2 GB | 960 GB/s | 200 |
| RTX 5090 | 32 GB | 30.4 GB | 1792 GB/s | 218 |
| RX 7600 | 8 GB | 7.6 GB | 288 GB/s | 161 |
| RX 7700 XT | 12 GB | 11.4 GB | 432 GB/s | 173 |
| RX 7800 XT | 16 GB | 15.2 GB | 624 GB/s | 200 |
| RX 7900 GRE | 16 GB | 15.2 GB | 576 GB/s | 200 |
| RX 7900 XT | 20 GB | 19.0 GB | 800 GB/s | 209 |
| RX 7900 XTX | 24 GB | 22.8 GB | 960 GB/s | 209 |
| RX 9070 | 16 GB | 15.2 GB | 640 GB/s | 200 |
| RX 9070 XT | 16 GB | 15.2 GB | 640 GB/s | 200 |
| Arc B580 | 12 GB | 11.4 GB | 456 GB/s | 173 |
| Arc A770 16GB | 16 GB | 15.2 GB | 560 GB/s | 200 |
Multi-GPU rigs
| Hardware | Memory | Usable | Bandwidth | Models that fit (8K) |
|---|---|---|---|---|
| 2x RTX 3090 | 2×24 GB | 45.6 GB | 936 GB/s | 225 |
| 2x RTX 4090 | 2×24 GB | 45.6 GB | 1008 GB/s | 225 |
| 2x RTX 5090 | 2×32 GB | 60.8 GB | 1792 GB/s | 228 |
Apple Silicon Macs
| Hardware | Memory | Usable | Bandwidth | Models that fit (8K) |
|---|---|---|---|---|
| Mac M1 16GB | 16 GB | 10.7 GB | 68 GB/s | 168 |
| Mac M2 24GB | 24 GB | 16.1 GB | 100 GB/s | 206 |
| Mac M3 24GB | 24 GB | 16.1 GB | 100 GB/s | 206 |
| Mac M4 16GB | 16 GB | 10.7 GB | 120 GB/s | 168 |
| Mac M4 32GB | 32 GB | 21.4 GB | 120 GB/s | 209 |
| Mac M1 Pro 32GB | 32 GB | 21.4 GB | 200 GB/s | 209 |
| Mac M3 Pro 36GB | 36 GB | 24.1 GB | 150 GB/s | 209 |
| Mac M4 Pro 24GB | 24 GB | 16.1 GB | 273 GB/s | 206 |
| Mac M4 Pro 48GB | 48 GB | 32.2 GB | 273 GB/s | 218 |
| Mac M4 Pro 64GB | 64 GB | 48.0 GB | 273 GB/s | 226 |
| Mac M1 Max 64GB | 64 GB | 48.0 GB | 400 GB/s | 226 |
| Mac M2 Max 96GB | 96 GB | 72.0 GB | 400 GB/s | 229 |
| Mac M3 Max 128GB | 128 GB | 96.0 GB | 400 GB/s | 238 |
| Mac M4 Max 64GB | 64 GB | 48.0 GB | 546 GB/s | 226 |
| Mac M4 Max 128GB | 128 GB | 96.0 GB | 546 GB/s | 238 |
| Mac M2 Ultra 192GB | 192 GB | 144 GB | 800 GB/s | 257 |
| Mac M3 Ultra 256GB | 256 GB | 192 GB | 819 GB/s | 261 |
| Mac M3 Ultra 512GB | 512 GB | 384 GB | 819 GB/s | 292 |
Unified-memory PCs
| Hardware | Memory | Usable | Bandwidth | Models that fit (8K) |
|---|---|---|---|---|
| DGX Spark | 128 GB | 109 GB | 273 GB/s | 245 |
| Ryzen AI Max+ 395 128GB | 128 GB | 109 GB | 256 GB/s | 245 |
Workstation GPUs
| Hardware | Memory | Usable | Bandwidth | Models that fit (8K) |
|---|---|---|---|---|
| RTX A6000 | 48 GB | 45.6 GB | 768 GB/s | 225 |
| RTX 6000 Ada | 48 GB | 45.6 GB | 960 GB/s | 225 |
| RTX PRO 6000 Blackwell | 96 GB | 91.2 GB | 1792 GB/s | 238 |