Complexity and time

Now, you have seen some of the most prominent tricks in our repertoire to reduce overfitting through regularization. In essence, regularization is just a manner of controlling the complexity of our network. Complexity control is not just useful to restrict your network from memorizing randomness; it also brings more direct benefits. Inherently, more complex networks are computationally expensive. They will take longer to train, and hence consume more of your resources. While this makes an insignificant difference when dealing with the task at hand, this difference is still quite noticeable. In the following diagram is a time complexity chart. This is a useful way of visualizing training time as a function of network complexity. We can see that an increase in our network's complexity seems to have a quasi-exponential effect on the increase in average time taken per training iteration: