trillion parameters

Nvidia is reportedly building Nemotron 4, an open-weight model with at least one trillion parameters — double the size of Nemotron 3 Ultra. The company has tripled its cloud spending on in-house training to $28 billion through 2031, with an earliest possible release this fall.

The Financial Times reports ByteDance is pretraining an AI model with up to 10 trillion parameters, which would make it three times larger than Moonshot's Kimi K3, currently China's largest model. The claim comes from anonymous insiders and has not been confirmed by ByteDance.