Ai2

Allen Institute for AI (AI2) has released DiScoFormer, a transformer model that estimates both the density and score of any distribution from a sample in a single forward pass without retraining. In 100 dimensions, the model reduces score estimation error by 6.5x and density error by 37x compared to classical kernel density estimation.

Allen Institute for AI researchers conducted token-level analysis comparing their 7B-parameter Olmo 3 transformer and Olmo Hybrid models. The study finds hybrid architectures show a loss gap advantage of 0.04 on content words (nouns, verbs, adjectives) versus 0.02 on function words, while transformers match or exceed hybrids on repeated tokens and closing braces.