model-architecture

Apple detailed its third-generation Foundation Models family: five models including AFM 3 Core Advanced, a 20-billion-parameter on-device model that keeps most parameters in flash storage and loads only 1-4 billion at a time into memory. The models were custom-built with Google and trained on Google's TPUs.

Allen Institute for AI released EMO, a 1B-active, 14B-total-parameter mixture-of-experts model trained on 1 trillion tokens. The model uses 8 active experts per token from a pool of 128 total experts, and can maintain near full-model performance while using just 12.5% of its experts for specific tasks.