Running AI models on your own computer

It means the weights are published: the trained numbers that make the model work. You can download them and run the model on your own machine, offline, for free.

It usually does not mean the training data or training code is public, which is what “open source” implies in ordinary software. “Open weights” is the more accurate term, and the industry uses the two interchangeably. Licences also vary. Some allow commercial use and some don't, so check the licence on any model you plan to build on.

A model is billions of numbers. Stored at full precision each takes 2 bytes, so a 9B model is about 18GB, too big for most laptops. Quantization stores each number in fewer bits: 4 bits instead of 16 cuts the file to roughly a quarter.

You lose a little accuracy. In practice the loss at Q4 is small enough that almost everyone uses it, which is why every figure on this site defaults to Q4_K_M.

The names decode simply:

• Q4: 4 bits per number. Q5, Q6, Q8 are progressively bigger and more accurate.
• _K: a smarter mixed scheme that spends more bits where they matter.
• _M: medium. _S is small, _L large, within the same bit width.

Rule of thumb: at Q4 a model needs roughly half its parameter count in gigabytes. Qwen3.8-27B is 28B and needs about 19 GB.

You'll see names like 35B-A3B. That means 35 billion parameters total, but only 3 billion active for any given word. These are mixture-of-experts models: they hold many specialist sub-networks and consult only a couple at a time.

The trap is assuming the small number is the one that matters. It governs speed: the model responds like a 3B model. But every parameter still has to be in memory, because any expert might be needed next. So it's as fast as a small model and as memory-hungry as a large one.

For example Qwen3.8-Flash-Next looks approachable but needs 203 GB, which is server territory.

Three different things people constantly conflate:

• VRAM: memory on a graphics card. Fastest, and a hard ceiling: a 24GB card can't run a 30GB model even if the PC has 128GB of system RAM.
• System RAM: ordinary memory. Slower, but there's usually more of it, and it works fine for smaller models.
• Unified memory: Apple Silicon shares one fast pool between CPU and GPU. It's why a 64GB Mac runs models that would otherwise need a very expensive graphics card.

You don't need a graphics card to start. If you're buying specifically to run models, total memory is the number that decides what you can run; everything else decides how fast.

This is the thing people get wrong most often. Fitting a model in memory and running it at a useful speed are two separate questions, and the second one has almost nothing to do with how much RAM you have.

To write a single word, the computer has to read the entire model out of memory. Then it does it again for the next word. So the limit isn't how fast your processor thinks. It's how fast memory can be read, which is called memory bandwidth. A faster CPU with slow memory barely helps.

So the ceiling on speed is your memory bandwidth divided by the size of the model: the same model reads several times faster from a graphics card than from ordinary RAM. How close a machine gets to that ceiling depends on the software and settings, so we don't quote speeds; we haven't measured them.

Which is why two computers with identical RAM are not equivalent. Published bandwidth:

A 64GB desktop and a 64GB MacBook Pro will load exactly the same models, but the MacBook Pro's memory can be read several times faster, so it generates text faster. That gap is bandwidth, not RAM, and it is worth understanding before spending money.

One happy exception: mixture-of-experts models only read their active parameters per word, not the whole thing. That's why a 35B MoE with 3B active can outrun a dense 9B while still needing four times the memory. It is slow to load but quick to answer.

The model's memory of your conversation (its context) is stored separately and grows as the conversation does. That's why every model here shows two figures: one at 4K tokens (a short chat) and one at 32K (a long document).

For some architectures the difference is small. For others it's dramatic: certain models triple their memory use between 4K and 32K. If you plan to feed in long documents, budget for the larger number.

Two tools cover almost everyone, and both handle downloading for you:

• LM Studio: a normal desktop app with a chat window. Easiest start if you don't live in a terminal.
• Ollama: command line, one line to run a model, easy to plug into your own scripts.

Start small. MiniCPM-V-4.7-1B needs only 1.4 GB and runs on almost anything. Confirm it works, then move up.