Scaling Laws and Compute Budgets: Chinchilla for LLMs

Previously in this course, we built the foundational components of our Transformer architecture in our Project Milestone: Custom Architecture Setup. Now that we have a working model, the next challenge is determining how large to build it and how much data to feed it before we commit to a massive training run.

In production, compute is your most expensive constraint. Training an LLM is not just about stacking layers; it’s about finding the Pareto-optimal frontier where your model size and data volume minimize loss for a given compute budget.

The Scaling Laws Framework

For years, the industry assumed that adding more parameters was the primary lever for performance. However, the seminal "Chinchilla" study (Hoffmann et al., 2022) fundamentally shifted this by demonstrating that most models—even famous ones—were "undertrained."

The core insight is that for a fixed compute budget, there is an optimal ratio of parameters ($N$) to training tokens ($D$). If you increase compute, you should scale both $N$ and $D$ proportionally.

Estimating Training Compute Requirements

A standard approximation for the compute cost ($C$) in FLOPs required to train a dense Transformer is:

$$C \approx 6ND$$

Where:

• $N$: Number of model parameters.
• $D$: Number of training tokens.
• 6: A constant derived from the forward pass (2 FLOPs per parameter) and backward pass (4 FLOPs per parameter).

If you have a budget of $10^{23}$ FLOPs, you shouldn't just build a massive model and feed it a tiny dataset. You need to solve for the balance that keeps the loss at a minimum.

Applying Chinchilla Scaling Laws

Chinchilla scaling suggests that for compute-optimal training, the model size should scale linearly with the amount of training data. Specifically, for every doubling of compute, you should roughly double the number of parameters and the number of tokens.

The optimal ratio is approximately 20 tokens per parameter.

Designing Model Dimensions for Compute Budgets

When designing your model, you work backward from your budget. Let’s say you have a budget of $10^{22}$ FLOPs.

1. Calculate total compute: $10^{22} \approx 6 \times N \times D$
2. Apply the ratio: $D \approx 20N$
3. Substitute: $10^{22} \approx 6 \times N \times (20N) = 120N^2$
4. Solve for N: $N \approx \sqrt{10^{22} / 120} \approx 288$ million parameters.

If you build a 288M parameter model and train it on $\approx 5.7$ billion tokens, you will achieve lower loss than a 1B parameter model trained on only 1B tokens, despite the latter having more parameters.

Comparison: Scaling Strategies

Worked Example: Calculating Budget for a 7B Model

Let’s say you want to train a 7-billion parameter model. How many tokens do you need to be compute-optimal?

PYTHON
# Constants
FLOP_CONSTANT = 6
TOKENS_PER_PARAM = 20

def estimate_tokens(params):
    CE9178">"""Calculate required tokens for compute-optimality."""
    return params * TOKENS_PER_PARAM

def estimate_compute(params, tokens):
    CE9178">"""Estimate total FLOPs required."""
    return FLOP_CONSTANT * params * tokens

params = 7e9 # 7 Billion
required_tokens = estimate_tokens(params)
compute_needed = estimate_compute(params, required_tokens)

print(f"To train a {params/1e9}B model optimally:")
print(f"Required tokens: {required_tokens/1e12:.2f} Trillion")
print(f"Total compute: {compute_needed:.2e} FLOPs")

Running this, you see that a 7B model requires roughly 140 billion tokens to reach its compute-optimal state. If your dataset is smaller than this, you should either shrink your model or find more data.

Hands-on Exercise

1. Calculate: Based on the code above, if you only have enough data for 50 billion tokens, what is the maximum number of parameters you should use to remain compute-optimal?
2. Adjust: If you decide to increase your model size to 10B parameters, how much additional compute budget do you need to request from your infrastructure team?

Common Pitfalls

• Ignoring Inference Costs: Scaling laws focus on training efficiency. A 100B model is compute-optimal for a massive dataset, but the inference latency might be unusable for your application. Always consider the serving budget alongside training.
• Data Quality vs. Quantity: Chinchilla assumes "high-quality" data. If you have 10T tokens of garbage, the scaling laws break. Quality often trumps quantity in real-world scenarios.
• Over-fitting to Constraints: Don't treat these formulas as absolute laws. They are guidelines. If you have a specific domain (e.g., medical or legal), your scaling behavior may differ from the general-purpose web-crawl data used in the original research.

Recap

We’ve moved from building individual components to understanding the macro-economics of model training. By leveraging the 20:1 token-to-parameter ratio, you can effectively allocate your GPU budget, ensuring you don't waste resources on undertrained models or insufficient data volumes. Always balance your architecture design with the data you have available.

Up next: We will dive into Data Parallelism Strategies, where we learn how to distribute this massive training load across multiple GPUs.