Width and Depth Are Choices You Make

A network's width is how many neurons sit in a given layer; its depth is how many layers the network has in total. Both are decided by whoever designs the network, before training ever starts — the network itself never decides to grow a new layer or add extra neurons partway through.

Width and depth are two of the biggest levers in a network's architecture. Two networks trained on the exact same data, using the exact same training process, can end up behaving very differently just because one is wider or deeper than the other — the shape you pick before training genuinely matters.

Training only adjusts the weights inside whatever shape you already built — it cannot add a neuron or a layer that was not there to begin with. If you want a wider or deeper network, that is a decision you make before training starts, not something you leave for the network to figure out along the way.

What does a network's width refer to?

Both a network's width and its depth are ____ made before training begins, not decisions the network makes for itself.

During training, a network can decide to add new layers to itself if needed.