Diminishing Returns: Why Bigger Isn't Always Better

More training data and more computing power ("compute") generally make a model better — more examples to learn patterns from, and more processing capacity to actually learn from them. But this doesn't scale forever in a straight line: each additional chunk of data or compute tends to buy a smaller improvement than the chunk before it, a pattern called diminishing returns.

Diminishing returns don't mean more data stops helping completely — they mean the improvement curve flattens out. Going from a tiny dataset to a medium one can produce a huge jump in quality; going from an already-enormous dataset to an even bigger one tends to produce a smaller, more incremental gain for the extra cost involved.

And more data and compute have real costs, not just theoretical ones: training runs take real time on real hardware, that hardware costs real money to buy and run, and running large amounts of computing hardware uses real electricity — which has a real environmental footprint. None of that is a reason for panic, but it's a genuine trade-off, not a free resource.

What does "diminishing returns" mean when it comes to training data and compute?

The pattern where each additional chunk of training data or compute buys a smaller improvement than the last is called ____.

Training large AI models has real costs in time, money, and electricity use.