LLM News

Researchers introduce ByteFlow Net, a tokenizer-free language model architecture that learns to segment raw byte streams into semantically meaningful units through compression-driven segmentation. The method adapts internal representation granularity per input, outperforming both BPE-based Transformers and previous byte-level approaches in experiments.

Researchers introduce UniLID, a language identification method that leverages tokenizer-based unigram distributions to identify languages and dialects with high sample efficiency. The approach achieves over 70% accuracy on low-resource languages with only five labeled examples per language, substantially outperforming existing systems like fastText, GlotLID, and CLD3 in low-resource settings.