How to run an LLM locally
Running a large language model on your own computer means no subscription, no data leaving your machine, and no rate limits. You need three things: enough GPU memory for the model, a program to run it, and the model file. This guide covers all three, with numbers from our database of 298 open models.
1. What hardware you need
The single number that matters most is memory the GPU can use: VRAM on a graphics card, or unified memory on a Mac. A model has to fit in it to run at full speed. Speed then depends on memory bandwidth. Here is how much of the 60 most-downloaded open models fits at each size:
| GPU memory | Top-60 models that fit (4-bit) | …at near-lossless 8-bit | Largest popular model that fits |
|---|---|---|---|
| 8 GB | 29 of 60 | 21 | Qwen3 VL 8B Instruct (8.8B) |
| 12 GB | 31 of 60 | 29 | Qwen3.5 9B (9.7B) |
| 16 GB | 32 of 60 | 30 | gpt-oss 20B (21B) |
| 24 GB | 43 of 60 | 32 | GLM 4.7 Flash (31B) |
| 32 GB | 43 of 60 | 37 | Qwen3.6 35B A3B (36B) |
| 48 GB | 43 of 60 | 43 | Qwen3.6 35B A3B (36B) |
8K context, real GGUF file sizes. From the Local LLM Hardware Index (2026-11-17).
On a Mac, about two-thirds to three-quarters of unified memory is available to the GPU by default:
| Mac | Top-60 models that fit | Largest that fits |
|---|---|---|
| Mac M4 16GB | 29 of 60 | Qwen3.5 9B |
| Mac M4 Pro 24GB | 32 of 60 | gpt-oss 20B |
| Mac M4 Pro 48GB | 43 of 60 | Qwen3.6 35B A3B |
| Mac M4 Max 128GB | 45 of 60 | gpt-oss 120B |
No GPU? Small models (1–4B) run on a CPU alone, slowly. A laptop with 16 GB of RAM can manage them; anything bigger really wants a GPU or Apple Silicon.
Local AI hardware: building a PC for local AI
- GPU first. Buy the most VRAM your budget allows: 12 GB is a sensible minimum, 16 GB is comfortable for 12–15B models, 24–32 GB runs most 20–32B models. See the best GPU for AI rankings and head-to-head comparisons.
- System RAM: 32 GB, or 64 GB if you plan to offload. Large mixture-of-experts models such as gpt-oss-120b can run on a 12–16 GB card with the rest in RAM; our offload calculator shows how much RAM each needs.
- Fast RAM helps offload. When part of a model sits in system RAM, its speed sets the pace, so DDR5 beats DDR4.
- Storage: models are 1–70 GB each at 4-bit, and you will collect several. An NVMe SSD makes loading quicker.
- CPU: matters little if the model fits in VRAM.
2. Which tool to use: Ollama vs LM Studio vs llama.cpp
| Ollama | LM Studio | llama.cpp | |
|---|---|---|---|
| What it is | Background service plus a simple command line and app | Desktop app with a model browser and chat window | The engine itself: command-line tools and a server |
| Best for | Getting started fast; powering other apps through its API | Browsing and trying models without a terminal | Full control: offload splits, context and cache settings |
| Models | Ollama library (one command per model) | GGUF from Hugging Face; MLX on Mac | Any GGUF file |
| Runs on | macOS, Windows, Linux | macOS (Apple Silicon), Windows, Linux | Everything, including CPU only |
| Our guide | 70 Ollama models with sizes and VRAM | Which models fit, quants explained | Offload calculator and commands |
All three use the same model format (GGUF) and similar code underneath, so memory needs are about the same whichever you pick. If you're unsure, start with Ollama or LM Studio.
3. Run your first model
- Check what fits. Look up your card on the GPU list, or use the VRAM calculator.
- Pick the largest model that fits fully, at 4-bit (Q4_K_M) to begin with. A bigger model at 4-bit usually beats a smaller one at 8-bit.
- Install and run. With Ollama that is one command, for example
ollama run qwen3:8b(6.7 GB at 4K context, fine on an 8 GB card). In LM Studio, search for the model and pick a file marked for full GPU offload.
Run AI locally or use an API?
Local is private, works offline and costs nothing per token once you own the hardware. Hosted APIs run the largest models and need no hardware. Many people do both. Compare costs on our API price tracker, which lists the same models where they're available.
Questions
Can I run an LLM locally with 8 GB of VRAM?
Yes. 29 of the 60 most popular open models fit in 8 GB at 4-bit, including 7–9B chat models. See every model that fits in 8 GB.
Is a Mac good for local LLMs?
For big models, yes: unified memory lets a 64–128 GB Mac hold models no gaming card can, though more slowly than a high-end GPU. See Mac vs PC for local LLMs.
Numbers on this page are computed automatically from the Hugging Face Hub API, GGUF file sizes and GPU specs, using the formulas on the methodology page. Text drafted with AI assistance; see how this site is made.