ANNs

Like biological neurons, artificial neurons also do not exist on their own. They exist in a network with other neurons. Basically, the neurons exist by feeding information to each other; the outputs of some neurons are inputs to some other neurons.

In any ANN, the first layer is called the Input Layer. These inputs are real values, such as the factors with weights (w.x) in our previous example. The sum values from the input layer are propagated to each neuron in the next layer. The neurons of that layer do the computation and pass their output to the next layer, and so on:

The layer that receives input from all previous neurons and passes its output to all of the neurons of the next layer is called a Dense layer. As this layer is connected to all of the neurons of the previous and next layer, it is also commonly referred to as a Fully Connected Layer.

The input and computation flow from layer to layer and finally end at the Output Layer, which gives the end estimate of the whole ANN.

The layers in-between the input and the output layers are called the Hidden Layers, as the values of the neurons within these hidden layers are unknown and a complete black box to the practitioner.

As you increase the number of layers, you increase the abstraction of the network, which in turn increases the ability of the network to solve more complex problems. When there are over three hidden layers, then it is referred to as a deepnet.

So, if this was a machine vision task, then the first hidden layer would be looking for edges, the next would look for corners, the next for curves and simple shapes, and so on:

Therefore, the complexity of the problem can determine the number of layers that are required; more layers lead to more abstractions. These layers can be very deep, with 1,000 or more layers, to very shallow, with just about half a dozen layers. Increasing the number of hidden layers does not necessarily give better results as the abstractions may be redundant.

So far, we have seen how artificial neurons can be stacked together to form a neural network. But we have seen that the perceptron neuron takes only binary input and gives only binary output. But in practice, there is a problem in doing things based on the perceptron's idea. This problem is addressed by activation functions.