Taobao's MNN team open-sources MNN-Pocket-0.3B, a 300M-parameter on-device model with a 173MB INT4 package

Alibaba's MNN team at Taobao has open-sourced MNN-Pocket-0.3B, a roughly 300-million-parameter language model whose INT4 deployment package is 173MB. IT Home reported the release on November 6, and Pandaily covered it. The model is built around the MNN inference engine and targets large, all-purpose shopping and service apps such as Taobao, where an on-device model must share compute, memory and battery with everything else on the phone.

Design targets

The team describes MNN-Pocket-0.3B as an "app-native large model" and set four goals:

• Decoding above 200 tokens per second on specified devices.
• Peak memory under 300MB, inside a model budget of about 500MB that leaves room for context growth and integration overhead.
• No more than one ten-thousandth of a reference phone's battery per typical inference, so 100 runs cost under 1%.
• Results matching an 8B-class model on specified business tasks.

Reported device performance

According to the team's own tests:

• iPhone 18 Pro: runs fully offline through the MNN engine at an average decode speed of about 350 tokens per second. The INT4 package includes weights, tokenizer, compute graph and inference configuration.
• iPad Pro (M5, Metal backend): 466 tokens per second decoding and 8,721 tokens per second prefill, with peak memory rising 228MB over baseline.
• Snapdragon 8 Gen 5 phone (CPU backend): 251 tokens per second.
• OnePlus 13: an incremental energy cost of about 0.3% of battery for 100 runs.

Architecture and training

The model was trained from scratch with Chinese and English as focus languages. The team tested dense configurations from about 50M to 1.3B parameters and found around 300M to be the largest size that met both the memory and speed targets. It settled on 28 layers with grouped-query attention. Going to 36 layers did not raise average scores but slowed decoding, the team says.

The vocabulary was trimmed from about 151,000 tokens in the Qwen3 tokenizer to 64,000, which reduces embedding and output-layer costs.

Benchmarks

The team reports an IFEval score of 73.20, a Chinese IFEval score of 74.59, and an average of 50.00 across 11 Chinese and English benchmarks. It says this is ahead of the same-size and larger models it compared, including Qwen3-0.6B. These are the team's figures. The model card says that, like other sub-billion-parameter models, it is not recommended for advanced math or long-horizon code generation.

Released variants

The Hugging Face release includes:

• a base model for continued pretraining
• a supervised fine-tuning checkpoint
• the instruction-tuned model
• a GGUF build
• the INT4 MNN build
• a decision-head model for choosing among candidates

The release also includes a DFlash2 speculative decoding draft model, which the team says can double decoding throughput on certain tasks.

What this means

The release is framed around the constraints of a single large app rather than a general-purpose assistant. The published targets for memory, speed and battery draw per inference are explicit and tied to named devices, so developers can check them against their own hardware. The team's finding that 36 layers added latency without raising average scores, and that about 300M parameters was the ceiling for its memory and speed goals, shows how those limits shaped the architecture. The vocabulary cut from about 151,000 to 64,000 tokens is one concrete way the team reduced footprint. The speed, battery and benchmark figures all come from the team's own testing, and the model card's warning against advanced math and long-horizon code generation defines where the model is not intended to be used.