K2-Horizon-375B-A23B: hardware requirements
K2-Horizon-375B-A23B needs about 242 GB of memory at Q4_K_M quantization: a 238 GB download plus room for context and a runtime allowance. That is beyond consumer hardware; use it through an API instead.
This is a mixture-of-experts model. Only a fraction of its 379B parameters are used for each token, so it runs faster than its size suggests, but all of it still has to fit in memory. The speed is that of a small model; the memory requirement is that of a large one. More on this →
Will it run on your machine?
Download sizes by quantization
Lower quantization means a smaller file and less memory, at some cost to quality. Q4_K_M is the usual starting point. What is this?
| Quant | Download | RAM @ 4K | RAM @ 32K | Source |
|---|---|---|---|---|
| Q4_K_Mrecommended | 238 GB | 242 GB | 249 GB | bartowski/K2-Horizon-375B-A23B-GGUF |
| Q5_K_M | 290 GB | 294 GB | 301 GB | bartowski/K2-Horizon-375B-A23B-GGUF |
| Q6_K | 334 GB | 339 GB | 346 GB | bartowski/K2-Horizon-375B-A23B-GGUF |
| Q8_0 | 403 GB | 407 GB | 414 GB | bartowski/K2-Horizon-375B-A23B-GGUF |
Download sizes are measured from the published files named in the Source column. The RAM columns add the context cache, calculated from the model's configuration, and a runtime allowance of 10% of the download (at least 0.5GB, at most 3GB).