ByteDance Seed releases Cola2, a 9B continuous-latent language model with 60.10 eight-task average

ByteDance Seed has released Cola2, a 9B-parameter continuous-latent base language model, on Hugging Face. The released checkpoint was trained on 346.03B source tokens and reaches a reported 60.10 average across eight evaluation tasks, according to the accompanying paper.

Architecture

Cola2 combines three components: a dynamic-compression VAE, a block-causal Diffusion Transformer (DiT), and a span-causal token decoder with latent feedback. The model groups text into semantic spans and represents each span with a compact continuous latent. The DiT generates latent blocks, and the decoder realizes those latents as discrete tokens. The paper is titled "Cola2: Co-Adaptive Hierarchical Language Modeling with Continuous Semantic Compression and Balanced Discrete Generation" (arXiv 2610.31305).

Training runs in two stages. Stage I jointly trains the dynamic-compression encoder and the token decoder. Stage II continues adapting the VAE with token-side supervision while an approximately 4B DiT learns the detached latent distribution through Flow Matching. At inference, Cola2 repeatedly proposes a latent block, decodes tokens, and re-encodes the committed text under the fixed span layout so the continuous and discrete histories stay aligned.

Reported benchmarks

The authors describe the checkpoint as resource-constrained, with a latent DiT that is still unconverged. All values are accuracies (%), and each Cola2 task value is the mean of three latent-noise sampled evaluations.

Task	Llama-3-8B	LLaDA-8B-Base	Qwen3-8B-Base	Cola2
Pretraining tokens	15T+	2.3T	~36T	346B
HellaSwag	39.21	69.65	68.13	41.75
StoryCloze	79.49	95.94	97.44	80.42
MMLU	63.19	61.27	72.31	51.32
RACE	75.77	87.40	90.65	69.91
8-task average	66.25	72.11	75.87	60.10

The full comparison also includes Qwen2.5-7B (75.08 average, 18T tokens). Other Cola2 scores: NarrativeQA 32.70, LAMBADA 63.00, ARC-Easy 87.62, TruthfulQA 54.10. The authors say Cola2 exceeds Llama-3-8B on HellaSwag and StoryCloze with more than 43× fewer reported pretraining tokens. They caution that the models use different training recipes, tokenizers and token-counting conventions, so the table contextualizes capability rather than isolating the generation paradigm.

A controlled scaling study compares AR, LLaDA, Cola DLM, uncompressed Cola-MDM and compressed Cola2-MDM at matched model size, data and evaluation settings. Cola-MDM shows the strongest aggregate scaling. Cola2-MDM converges more slowly, continues closing its gap to LLaDA after roughly 150B tokens, and remains unsaturated at about 210B tokens. The optimal decoder classifier-free-guidance weight stays near -0.9 for most of training and gradually moves toward zero.

Release contents and usage

The repository holds eight sharded safetensors weight files (F32 tensors, listed as 9B parameters), tokenizer and generation configs, Transformers remote-code modules, normalized JSONL data for the eight evaluation tasks, and SHA256SUMS. Inference code, command-line tools and the benchmark runner live in the Cola2 GitHub repository. Training code is not included.

Requirements are Python 3.10+, PyTorch 2.2+, Transformers 4.51 to 4.x, and safetensors 0.4+, with CUDA and bfloat16 recommended. Loading requires trust_remote_code=True. Cola2 is a base model, not an instruction-tuned chat assistant.

The default preset uses an 8-step low-noise-biased Euler DiT schedule, DiT CFG scale 10, decoder CFG scale -0.9, greedy token selection and max_new_tokens=32. Because latent generation starts from Gaussian noise, greedy selection does not make output deterministic, and results should be compared with a seeded torch.Generator. The original paper seeds are not published, so small differences from reference results are expected.

The repository code is licensed under Apache License 2.0. Model weights, tokenizer assets and benchmark datasets remain subject to their respective terms. No Inference Provider currently deploys the model. Context window and API pricing are not stated.

What this means

Cola2 is a research checkpoint for testing a different generation paradigm, not a drop-in competitor to the 8B baselines it is compared with. Its 60.10 eight-task average is below Llama-3-8B (66.25) and LLaDA-8B-Base (72.11), and the wins over Llama-3-8B are limited to HellaSwag and StoryCloze. The authors say the latent DiT is unconverged and that the scaling study shows Cola2-MDM still unsaturated at about 210B tokens. The token-efficiency claim is therefore the main point of interest, and the authors themselves caveat it because training recipes and token-counting conventions differ across models. Teams that want to reproduce or extend the work get weights, inference code and evaluation data, but not training code, and the remote-code requirement and base-model-only status limit immediate deployment.