Token AI Releases STAM and STAMLite Optimizers for Stable LLM Training
Switch to STAMLite for training LLMs to reduce GPU usage by ~50% and improve stability with dynamic beta1.
Switch your training pipelines to use STAMLite as the default optimizer, update your code to handle dynamic beta1, and benchmark against AdamW to validate performance gains.
Summary
Token AI has released a new optimizer called STAM, standing for Stable Training with Adaptive Momentum, and a lighter variant STAMLite. Both aim to replace traditional optimizers such as Adam and AdamW by dynamically adjusting the beta1 momentum term based on the difference between current gradients and previous momentum. When gradients are noisy, STAM reduces beta1 to curb unstable updates, and when gradients are stable it keeps momentum high, allowing the optimizer to self‑correct during noisy phases. STAMLite keeps the same adaptive beta1 logic but trims the optimizer state to roughly one‑times the parameter size, compared to AdamW's two‑times, cutting GPU memory usage by about 50 %. Benchmarks on a hyperparameter sweep show STAMLite achieving 0.61 accuracy and 0.91 loss, while on a long‑horizon non‑stationary MLP task it matches NAdam with 0.97 accuracy and 0.09 loss. The research paper demonstrates that STAMLite can outperform AdamW and NAdam in both accuracy and resource efficiency across multiple tasks. Token AI claims the optimizer is ready for early adoption, and the author has already begun training a full LLM from scratch using STAMLite. The paper and benchmarks are available on the Token AI website for developers looking to experiment with the new optimizer.
Key changes
- Introduces STAM optimizer with adaptive beta1 based on gradient residual variance
- STAMLite reduces optimizer state to ~1× parameter size, cutting memory usage by ~50 % vs AdamW
- Dynamic beta1 lowers momentum during noisy gradients and maintains it when gradients are stable
- STAMLite self‑corrects momentum to avoid persistent wrong updates seen in SGD
- Benchmarks: Hyperparameter sweep accuracy 0.61, loss 0.91; Long‑horizon MLP accuracy 0.97, loss 0.09
- STAMLite matches NAdam on long‑horizon tasks while using less GPU resources