An Deep Neural Network(DNN) is a learning algorithm or we can say it is a computational learning system that uses a network of functions to understand and translates a data input of one form to desired output.
The concept of the Deep Neural Network(DNN) was inspired by Human Biology and the way neurons of the human brain function together to understand input from human senses.
Neural Networks are class of models that are built with layers.
Each Neuron of each layer is responsible for activation of successor neuron of next layer and it is done by making some calculation.
Following steps, that follows each neurons :
- Multiply all the activation value by their weights w.
- Add them together referred as weighted sum
- Apply the activation function.
Activation functions are used at the end of a hidden unit to introduce non-linear complexities to the model.
In the context of neural networks, the cross-entropy loss L(z, y) is commonly used and is defined as follows:
The learning rate, often noted α or sometimes η, indicates at which pace the weights get updated. This can be fixed or adaptively changed. The current most popular method is called Adam, which is a method that adapts the learning rate.
Backpropagation is a method to update the weights in the neural network by taking into account the actual output and the desired output. The derivative with respect to weight is computed using chain rule and is of the following form:
As a result, the weight is updated as follows:
Diagram
Gradient Descent is an optimization algorithm used to minimize the cost function by making some changes into weight and biases of the network.
Thus, it helps to finding output the local minima / global minima of cost function for each weight and biases.
In a neural network, weights are updated as follows:
- Step 1: Take a batch of training data.
- Step 2: Perform forward propagation to obtain the corresponding loss.
- Step 3: Backpropagte the loss to get the gradients.
- Step 4: Use the gradients to update the weights of the network.
Dropout is a technique meant at preventing overfitting the training data by dropping out units in a neural network. In practice, neurons are either dropped with probability P or kept with probability 1-p.
comment down your views, if any mistakes found the blog will be updated with effect.