parameter-golf
Train the smallest LM you can that fits in 16MB. Best model wins!
About this project
OpenAI Model Craft Challenge: Parameter Golf is a challenge to train the best language model that fits in a 16MB artifact and trains in under 10 minutes on 8xH100s, evaluated by compression on the FineWeb validation set (tokenizer-agnostic, bits per byte). This challenge is heavily inspired by the NanoGPT Speedrunning challenge, where participants compete to train a model that reaches 3.28 FineWeb validation loss as quickly as possible. We're excited to see how optimizing for a parameter-constrained setting pushes people toward unique architectures (test-time compute, aggressive parameter tying, depth recurrence, low-rank training, ...), compression schemes (low precision, QAT, bitnets,…
Technologies
Project health
GitHub
Reviews
Built by
Maintain openai/parameter-golf? Claiming verifies admin access through your GitHub account and gives you control of this listing.
