subquadratic attention

Zyphra AI has introduced Zyphra Hierarchical Attention (ZHA), a tree-indexed sparse attention method. It retrofits frozen LLMs with 4B tokens of distillation and matches dense attention on reasoning and 256K retrieval benchmarks, according to MarkTechPost.