Researchers propose Progressive Residual Warmup (ProRes), a pretraining technique that staggers layer learning by gradually warming residual connections from 0 to 1, with deeper layers taking longer to activate. The method demonstrates faster convergence, stronger generalization, and improved downstream performance across multiple model scales and initialization schemes.
Researchers introduce FlyThinker, a framework that runs reasoning and generation concurrently rather than sequentially, addressing limitations of existing "think-then-generate" approaches in long-form personalized text generation. The method uses a separate reasoning model that generates token-level guidance in parallel with the main generation model, enabling more adaptive reasoning without sacrificing computational efficiency.
A new research paper introduces RePo, a mechanism that replaces fixed positional encoding with learned, context-aware token positioning. Tested on OLMo-2 1B and 7B models, RePo shows consistent improvements on tasks with noisy contexts and longer sequences while maintaining performance on standard benchmarks.
Researchers introduce StructLens, an interpretability framework that analyzes language models by constructing maximum spanning trees from residual streams to uncover inter-layer structural relationships. The approach reveals similarity patterns distinct from conventional cosine similarity and demonstrates practical benefits for layer pruning optimization.
Researchers introduce ByteFlow Net, a tokenizer-free language model architecture that learns to segment raw byte streams into semantically meaningful units through compression-driven segmentation. The method adapts internal representation granularity per input, outperforming both BPE-based Transformers and previous byte-level approaches in experiments.
Researchers have developed BeliefSim, a framework that uses Large Language Models to simulate how different demographic groups respond to misinformation by modeling their underlying beliefs. The approach achieved 92% accuracy in predicting susceptibility across multiple datasets and conditioning strategies.
Researchers propose SureLock, a technique that reduces computational FLOPs in masked diffusion language model decoding by 30-50% on LLaDA-8B by skipping attention and feed-forward computations for tokens that have converged. The method caches key-value pairs for locked positions while continuing to compute for unlocked tokens, reducing per-iteration complexity from O(N²d) to O(MNd).
A new arXiv study systematically characterizes memorization behavior in diffusion language models (DLMs) and finds they exhibit substantially lower memorization-based leakage of personally identifiable information compared to autoregressive language models. The research establishes a theoretical framework showing that sampling resolution directly correlates with exact training data extraction.
A new paper identifies token rounding as the primary bottleneck limiting continuous diffusion language models (DLMs) and proposes CoDAR, a two-stage framework that combines continuous embedding-space diffusion with a contextual autoregressive decoder. Experiments on LM1B and OpenWebText show CoDAR achieves competitive performance with discrete diffusion approaches while offering tunable fluency-diversity trade-offs.
Researchers propose Go-Browse, a method for training web agents through structured exploration that frames data collection as graph search. A 7B parameter language model fine-tuned on 10K trajectories achieves 21.7% success on the WebArena benchmark, outperforming GPT-4o mini by 2.4 percentage points.
Researchers have identified token rounding as a primary bottleneck limiting continuous diffusion language models (DLMs) and propose CoDAR, a two-stage framework that maintains continuous embedding-space diffusion while using an autoregressive Transformer decoder for contextualized token discretization. Experiments on LM1B and OpenWebText show CoDAR achieves competitive performance with discrete diffusion approaches.
Researchers have proposed DiSE, a self-evaluation method designed to assess output quality in diffusion language models (dLLMs) by computing token regeneration probabilities. The technique enables efficient confidence quantification for models that generate text bidirectionally rather than sequentially, addressing a key limitation in quality assessment.
Guide Labs has open-sourced Steerling-8B, an 8 billion parameter language model built with a new architecture specifically designed to make the model's reasoning and actions easily interpretable. The release addresses a persistent challenge in AI development: understanding how large language models arrive at their outputs.
A new research paper introduces methods for predicting when humans will intervene in autonomous web agents by analyzing distinct interaction patterns. The work, which includes a dataset of 400 real-user web navigation trajectories with over 4,200 interleaved human-agent actions, shows that intervention-aware models improved agent usefulness by 26.5% in user studies.