Running AI models on your own computer

It means the weights are published — the trained numbers that make the model work. You can download them and run the model on your own machine, offline, for free.

It usually does not mean the training data or training code is public, which is what “open source” implies in ordinary software. “Open weights” is the more accurate term, and the industry uses the two interchangeably. Licences also vary — some allow commercial use, some don't — so check the licence on any model you plan to build on.

A model is billions of numbers. Stored at full precision each takes 2 bytes, so a 9B model is about 19GB — too big for most laptops. Quantization stores each number in fewer bits: 4 bits instead of 16 cuts the file to roughly a quarter.

You lose a little accuracy. In practice the loss at Q4 is small enough that almost everyone uses it, which is why every figure on this site defaults to Q4_K_M.

The names decode simply:

• Q4 — 4 bits per number. Q5, Q6, Q8 are progressively bigger and more accurate.
• _K — a smarter mixed scheme that spends more bits where they matter.
• _M — medium. _S is small, _L large, within the same bit width.

Rule of thumb: at Q4 a model needs roughly half its parameter count in gigabytes. Qwen3.5-9B is 9.7B and needs about 6.8 GB.

You'll see names like 35B-A3B. That means 35 billion parameters total, but only 3 billion active for any given word. These are mixture-of-experts models: they hold many specialist sub-networks and consult only a couple at a time.

The trap is assuming the small number is the one that matters. It governs speed — the model responds like a 3B model. But every parameter still has to be in memory, because any expert might be needed next. So it's as fast as a small model and as memory-hungry as a large one.

For example MiniMax-M2.7 looks approachable but needs 144 GB — server territory.

Three different things people constantly conflate:

• VRAM — memory on a graphics card. Fastest, and a hard ceiling: a 24GB card can't run a 30GB model even if the PC has 128GB of system RAM.
• System RAM — ordinary memory. Slower, but there's usually more of it, and it works fine for smaller models.
• Unified memory — Apple Silicon shares one fast pool between CPU and GPU. It's why a 64GB Mac runs models that would otherwise need a very expensive graphics card.

You don't need a graphics card to start. If you're buying specifically to run models, total memory is the number that decides what you can run; everything else decides how fast.

This is the thing people get wrong most often. Fitting a model in memory and running it at a useful speed are two separate questions, and the second one has almost nothing to do with how much RAM you have.

To write a single word, the computer has to read the entire model out of memory. Then it does it again for the next word. So the limit isn't how fast your processor thinks — it's how fast memory can be read, which is called memory bandwidth. A faster CPU with slow memory barely helps.

The rough sum is simple: divide your memory bandwidth by the size of the model. A 5GB model on a machine reading 70GB per second gives you roughly fourteen words a second. The same model on a graphics card reading 1,000GB per second gives you nearer a hundred and thirty.

Which is why two computers with identical RAM are not remotely equivalent:

A 64GB desktop and a 64GB MacBook Pro will load exactly the same models. The MacBook will answer roughly six times faster. That gap is bandwidth, not RAM, and it is the single biggest thing to understand before spending money.

For a sense of scale: most people read about 7–10 words a second. Below that a model feels like it's lagging behind you. Around 20 and it arrives as fast as you can take it in. Under 3 and it's technically working but not something you'd choose to sit through.

How we work the speeds out. They are calculated, not benchmarked — we don't own these machines and won't pretend otherwise. The sum is bandwidth × 0.65 ÷ the bytes read per token, where the model size is the real published file size, the bandwidth is the figure in the table above, and 0.65 is the share of theoretical peak a runtime like llama.cpp actually reaches once attention, sampling and framework overhead are paid for. It is deliberately conservative; measured results usually come in at or above these numbers. Your own will vary with the runtime, the quantization and what else is running.

One happy exception: mixture-of-experts models only read their active parameters per word, not the whole thing. That's why a 35B MoE with 3B active can outrun a dense 9B while still needing four times the memory — slow to load, quick to answer.

The model's memory of your conversation — its context — is stored separately and grows as the conversation does. That's why every model here shows two figures: one at 4K tokens (a short chat) and one at 32K (a long document).

For some architectures the difference is small. For others it's dramatic: certain models triple their memory use between 4K and 32K. If you plan to feed in long documents, budget for the larger number.

Two tools cover almost everyone, and both handle downloading for you:

• LM Studio — a normal desktop app with a chat window. Easiest start if you don't live in a terminal.
• Ollama — command line, one line to run a model, easy to plug into your own scripts.

Start small — ERNIE-4.5-0.3B-PT needs only 853 MB and runs on almost anything. Confirm it works, then move up.