| RTX 5090 | $1,999 | 32 GB | 1792 GB/s | 208 | F16 · ~109 tok/s | F16 · ~648 tok/s | Q5_K_M · ~77 tok/s | Offload Q4_K_M, 64 GB RAM · ~22–44 tok/s |
| RTX 4090 | $1,599 | 24 GB | 1008 GB/s | 190 | F16 · ~62 tok/s | F16 · ~365 tok/s | Q3_K_M only (below 4-bit) | Offload Q4_K_M, 64 GB RAM · ~17–34 tok/s |
| RTX 3090 Ti | $1,999 | 24 GB | 1008 GB/s | 190 | F16 · ~62 tok/s | F16 · ~365 tok/s | Q3_K_M only (below 4-bit) | Offload Q4_K_M, 64 GB RAM · ~17–34 tok/s |
| RX 7900 XTX | $999 | 24 GB | 960 GB/s | 190 | F16 · ~59 tok/s | F16 · ~347 tok/s | Q3_K_M only (below 4-bit) | Offload Q4_K_M, 64 GB RAM · ~17–34 tok/s |
| RTX 3090 | $1,499 | 24 GB | 936 GB/s | 190 | F16 · ~57 tok/s | F16 · ~338 tok/s | Q3_K_M only (below 4-bit) | Offload Q4_K_M, 64 GB RAM · ~17–34 tok/s |
| RX 7900 XT | $899 | 20 GB | 800 GB/s | 175 | F16 · ~49 tok/s | F16 · ~289 tok/s | Offload Q4_K_M, 32 GB RAM · ~6.6–11 tok/s | Offload Q4_K_M, 64 GB RAM · ~16–31 tok/s |
| RTX 5080 | $999 | 16 GB | 960 GB/s | 167 | Q8_0 · ~110 tok/s | Q8_0 · ~395 tok/s | Offload Q4_K_M, 32 GB RAM · ~3.3–6.5 tok/s | Offload Q4_K_M, 64 GB RAM · ~14–29 tok/s |
| RTX 5070 Ti | $749 | 16 GB | 896 GB/s | 167 | Q8_0 · ~103 tok/s | Q8_0 · ~369 tok/s | Offload Q4_K_M, 32 GB RAM · ~3.3–6.4 tok/s | Offload Q4_K_M, 64 GB RAM · ~14–29 tok/s |
| RTX 4080 Super | $999 | 16 GB | 736 GB/s | 167 | Q8_0 · ~85 tok/s | Q8_0 · ~303 tok/s | Offload Q4_K_M, 32 GB RAM · ~3.3–6.2 tok/s | Offload Q4_K_M, 64 GB RAM · ~14–28 tok/s |
| RTX 4080 | $1,199 | 16 GB | 717 GB/s | 167 | Q8_0 · ~82 tok/s | Q8_0 · ~295 tok/s | Offload Q4_K_M, 32 GB RAM · ~3.2–6.2 tok/s | Offload Q4_K_M, 64 GB RAM · ~14–28 tok/s |
| RTX 4070 Ti Super | $799 | 16 GB | 672 GB/s | 167 | Q8_0 · ~77 tok/s | Q8_0 · ~277 tok/s | Offload Q4_K_M, 32 GB RAM · ~3.2–6.1 tok/s | Offload Q4_K_M, 64 GB RAM · ~14–28 tok/s |
| RX 9070 | $549 | 16 GB | 640 GB/s | 167 | Q8_0 · ~73 tok/s | Q8_0 · ~264 tok/s | Offload Q4_K_M, 32 GB RAM · ~3.2–6.1 tok/s | Offload Q4_K_M, 64 GB RAM · ~14–28 tok/s |
| RX 9070 XT | $599 | 16 GB | 640 GB/s | 167 | Q8_0 · ~73 tok/s | Q8_0 · ~264 tok/s | Offload Q4_K_M, 32 GB RAM · ~3.2–6.1 tok/s | Offload Q4_K_M, 64 GB RAM · ~14–28 tok/s |
| RX 7800 XT | $499 | 16 GB | 624 GB/s | 167 | Q8_0 · ~72 tok/s | Q8_0 · ~257 tok/s | Offload Q4_K_M, 32 GB RAM · ~3.2–6.0 tok/s | Offload Q4_K_M, 64 GB RAM · ~14–28 tok/s |
| RX 7900 GRE | $549 | 16 GB | 576 GB/s | 167 | Q8_0 · ~66 tok/s | Q8_0 · ~237 tok/s | Offload Q4_K_M, 32 GB RAM · ~3.2–6.0 tok/s | Offload Q4_K_M, 64 GB RAM · ~14–28 tok/s |
| Arc A770 16GB | $349 | 16 GB | 560 GB/s | 167 | Q8_0 · ~64 tok/s | Q8_0 · ~231 tok/s | Offload Q4_K_M, 32 GB RAM · ~3.2–5.9 tok/s | Offload Q4_K_M, 64 GB RAM · ~14–27 tok/s |
| RTX 5060 Ti 16GB | $429 | 16 GB | 448 GB/s | 167 | Q8_0 · ~51 tok/s | Q8_0 · ~185 tok/s | Offload Q4_K_M, 32 GB RAM · ~3.1–5.6 tok/s | Offload Q4_K_M, 64 GB RAM · ~14–27 tok/s |
| RTX 4060 Ti 16GB | $499 | 16 GB | 288 GB/s | 167 | Q8_0 · ~33 tok/s | Q8_0 · ~119 tok/s | Offload Q4_K_M, 32 GB RAM · ~2.9–5.0 tok/s | Offload Q4_K_M, 64 GB RAM · ~13–25 tok/s |
| RTX 5070 | $549 | 12 GB | 672 GB/s | 150 | Q8_0 · ~77 tok/s | Offload Q4_K_M, 32 GB RAM · ~85–124 tok/s | Offload Q4_K_M, 32 GB RAM · ~2.2–4.4 tok/s | Offload Q4_K_M, 64 GB RAM · ~13–27 tok/s |
| RTX 4070 | $599 | 12 GB | 504 GB/s | 150 | Q8_0 · ~58 tok/s | Offload Q4_K_M, 32 GB RAM · ~75–104 tok/s | Offload Q4_K_M, 32 GB RAM · ~2.2–4.3 tok/s | Offload Q4_K_M, 64 GB RAM · ~13–26 tok/s |
| RTX 4070 Super | $599 | 12 GB | 504 GB/s | 150 | Q8_0 · ~58 tok/s | Offload Q4_K_M, 32 GB RAM · ~75–104 tok/s | Offload Q4_K_M, 32 GB RAM · ~2.2–4.3 tok/s | Offload Q4_K_M, 64 GB RAM · ~13–26 tok/s |
| RTX 4070 Ti | $799 | 12 GB | 504 GB/s | 150 | Q8_0 · ~58 tok/s | Offload Q4_K_M, 32 GB RAM · ~75–104 tok/s | Offload Q4_K_M, 32 GB RAM · ~2.2–4.3 tok/s | Offload Q4_K_M, 64 GB RAM · ~13–26 tok/s |
| Arc B580 | $249 | 12 GB | 456 GB/s | 150 | Q8_0 · ~52 tok/s | Offload Q4_K_M, 32 GB RAM · ~72–98 tok/s | Offload Q4_K_M, 32 GB RAM · ~2.2–4.2 tok/s | Offload Q4_K_M, 64 GB RAM · ~13–25 tok/s |
| RX 7700 XT | $449 | 12 GB | 432 GB/s | 150 | Q8_0 · ~50 tok/s | Offload Q4_K_M, 32 GB RAM · ~70–94 tok/s | Offload Q4_K_M, 32 GB RAM · ~2.2–4.2 tok/s | Offload Q4_K_M, 64 GB RAM · ~13–25 tok/s |
| RTX 3060 12GB | $329 | 12 GB | 360 GB/s | 150 | Q8_0 · ~41 tok/s | Offload Q4_K_M, 32 GB RAM · ~63–83 tok/s | Offload Q4_K_M, 32 GB RAM · ~2.1–4.1 tok/s | Offload Q4_K_M, 64 GB RAM · ~13–25 tok/s |
| RTX 3080 10GB | $699 | 10 GB | 760 GB/s | 146 | Q6_K · ~113 tok/s | Offload Q4_K_M, 32 GB RAM · ~49–85 tok/s | Offload Q4_K_M, 32 GB RAM · ~1.9–3.9 tok/s | Offload Q4_K_M, 64 GB RAM · ~13–26 tok/s |
| RTX 5060 | $299 | 8 GB | 448 GB/s | 135 | Q4_K_M · ~89 tok/s | Offload Q4_K_M, 32 GB RAM · ~29–52 tok/s | Offload Q4_K_M, 32 GB RAM · ~1.6–3.3 tok/s | Offload Q4_K_M, 64 GB RAM · ~12–24 tok/s |
| RTX 3060 Ti | $399 | 8 GB | 448 GB/s | 135 | Q4_K_M · ~89 tok/s | Offload Q4_K_M, 32 GB RAM · ~29–52 tok/s | Offload Q4_K_M, 32 GB RAM · ~1.6–3.3 tok/s | Offload Q4_K_M, 64 GB RAM · ~12–24 tok/s |
| RTX 3070 | $499 | 8 GB | 448 GB/s | 135 | Q4_K_M · ~89 tok/s | Offload Q4_K_M, 32 GB RAM · ~29–52 tok/s | Offload Q4_K_M, 32 GB RAM · ~1.6–3.3 tok/s | Offload Q4_K_M, 64 GB RAM · ~12–24 tok/s |
| RX 7600 | $269 | 8 GB | 288 GB/s | 135 | Q4_K_M · ~57 tok/s | Offload Q4_K_M, 32 GB RAM · ~27–45 tok/s | Offload Q4_K_M, 32 GB RAM · ~1.6–3.2 tok/s | Offload Q4_K_M, 64 GB RAM · ~12–23 tok/s |
| RTX 4060 Ti 8GB | $399 | 8 GB | 288 GB/s | 135 | Q4_K_M · ~57 tok/s | Offload Q4_K_M, 32 GB RAM · ~27–45 tok/s | Offload Q4_K_M, 32 GB RAM · ~1.6–3.2 tok/s | Offload Q4_K_M, 64 GB RAM · ~12–23 tok/s |
| RTX 4060 | $299 | 8 GB | 272 GB/s | 135 | Q4_K_M · ~54 tok/s | Offload Q4_K_M, 32 GB RAM · ~27–44 tok/s | Offload Q4_K_M, 32 GB RAM · ~1.6–3.2 tok/s | Offload Q4_K_M, 64 GB RAM · ~12–23 tok/s |