
Weight initialization is a crucial step in building deep learning models, as it can significantly impact the model's performance.
The choice of weight initialization strategy can greatly affect the model's ability to learn and converge.
Random weight initialization is a common approach, where weights are initialized with random values between -1 and 1.
This approach can lead to exploding gradients, especially in deep networks.
Xavier initialization, on the other hand, initializes weights with a mean of zero and a variance of 1/n, where n is the number of inputs.
This approach helps to reduce the exploding gradient problem, but can also lead to slow convergence.
For your interest: Multiplicative Weight Update Method
Early Approaches
Early Approaches to weight initialization were a major challenge in deep learning research. Geoffrey Hinton and Salakhutdinov introduced a weight initialization strategy called Greedy Layerwise Unsupervised Pretraining in 2006.
This approach uses an unsupervised learning model, such as an autoencoder, to learn the weights between the input and the first hidden layer. The weights are frozen and used as inputs to learn the weights that flow into the next hidden layer.
The process continues until all the layers in the neural network have been traversed. The weights learned this way are fine-tuned and used as the initial weights to train the neural network.
This approach is greedy because it doesn't optimize the initial weights across all layers in the network but only focuses on the current layer. It's a layerwise approach in an unsupervised setting, and the term pretraining signifies that this process occurs ahead of the actual training process.
Worth a look: MobileNet
Random Initialization
Random initialization is a natural next step after trying to initialize weights to all zeros or any constant k. It involves assigning random values to the weights, which breaks the symmetry problem by making neurons perform differently.
The weights are drawn from a zero-mean Gaussian distribution, but the problem is that it may lead to vanishing or exploding layer outputs in the deep neural network, depending on the variance of the Gaussian distribution.
If the variance of initial weights is set too small, the variance of layer outputs may be extremely small or vanishing due to the accumulation of small variance during forward propagation. On the contrary, if the variance of initial weights is set too large, the variance of layer outputs may be extremely large or exploding due to the accumulation of large variance during forward propagation.
Assigning weights random values can be problematic if they are initialized too high or too low. If weights are initialized with large values, each matrix multiplication will result in a significantly larger value, slowing down the rate of learning. If weights are initialized with small values, each matrix multiplication will result in significantly smaller values, also slowing down the rate of learning.
Here are some key issues with random initialization:
- Large values can cause activations to become saturated, leading to vanishing gradients during backpropagation.
- Small values can cause activations to become vanishingly small, also leading to vanishing gradients during backpropagation.
Zero or Constant
Initializing all weights to a zero or constant value is suboptimal. This is because the weights will evolve symmetrically as the training proceeds, and we will not be able to break this symmetry.
The main advantage of using a neural network is its ability to learn a complex mapping from the input space onto the output, but under zero or constant weight initialization, the neurons in a layer change symmetrically throughout the training process.
The ReLU function, which maps all negative inputs to zero, does not have a zero mean, unlike the sigmoid and tanh activations.
Initializing the weights as zeros leads to vanishing gradient, where all the neurons are zeros during forward propagation and the gradients of the weights are also zeros during back propagation.
Initializing the weights as non-zero constants, like 1s, causes all the neurons in one layer to have the same value and represent only one feature during forward propagation, leading to the symmetry problem.
Using a constant value to initialize a model's weights can result in poor performance, as the outputs of all hidden units in the model would have the same influence on the cost, resulting in identical gradients.
Random
Random initialization can be a bit of a double-edged sword. It breaks the symmetry problem by giving random values to the weights, making neurons perform differently. However, it may lead to vanishing or exploding layer outputs in the deep neural network, depending on the variance of the Gaussian distribution.
Assigning weights random values can result in significantly larger or smaller values during matrix multiplication. This can slow down the rate of learning if the weights are initialized too high or too low.
If the variance of initial weights is set too small, the variance of layer outputs may be extremely small or vanishing. On the other hand, if the variance of initial weights is set too large, the variance of layer outputs may be extremely large or exploding.
Here are some key issues with random initialization:
- If weights are initialized with large values, each matrix multiplication will result in a significantly larger value.
- If weights are initialized with small values, each matrix multiplication will result in significantly smaller values.
The problem of vanishing or exploding gradients can be avoided by finding the optimal scaling factor or variance of the weight distribution. This allows the network to operate in the region between vanishing and saturating activations.
Lecun Uniform
The Lecun Uniform initializer is a popular choice for random initialization in neural networks. It draws samples from a uniform distribution within a specific range, which is determined by the number of input units in the weight tensor.
This range is calculated as [-limit, limit], where limit = sqrt(3 / fan_in), and fan_in is the number of input units in the weight tensor. For example, if the weight tensor has 1000 input units, the limit would be sqrt(3 / 1000).
The Lecun Uniform initializer is designed to help prevent dead neurons in the network by initializing weights with a range that allows for some degree of randomness. This can be beneficial in avoiding the problem of all neurons having the same output, which can lead to poor performance.
By using the Lecun Uniform initializer, you can help ensure that your neural network is initialized in a way that promotes good performance and avoids common pitfalls.
Xavier or Glorot
Xavier or Glorot initialization is a strategy proposed by Xavier Glorot and Yoshua Bengio to initialize weights in deep learning models. This strategy helps prevent the problem of vanishing gradients.
The goal of Xavier initialization is to set the variance of the weights to 1, so that activations neither vanish nor saturate during the forward pass. This is achieved by setting the variance to the average of 1/fan_in and 1/fan_out, where fan_in and fan_out are the number of input and output units in the weight tensor.
Having the variance of the weights equal to 1 helps overcome the problem of vanishing gradients during backpropagation. This is because the variance of a neuron's output is the number of weights that flow out of the neuron into the next layer, and the number of paths through which gradients flow into it.
To implement Xavier initialization in Keras, you can use either the GlorotUniform or GlorotNormal class in the Keras initializers module. If you do not specify a kernel initializer when defining the model, it defaults to GlorotUniform.
Here are the key differences between GlorotUniform and GlorotNormal classes:
The GlorotNormal class draws samples from a truncated normal distribution centered on 0 with stddev = sqrt(2 / (fan_in + fan_out)), where fan_in is the number of input units in the weight tensor and fan_out is the number of output units. This class also takes a seed parameter to make the behavior of the initializer deterministic.
The Glorot uniform initializer, also called Xavier uniform initializer, draws samples from a uniform distribution within [-limit, limit], where limit = sqrt(6 / (fan_in + fan_out)).
He Initialization
He Initialization is a popular technique used to initialize the weights of neural networks. It's named after Xavier Glorot and Yoshua Bengio, but you might also hear it referred to as Kaiming initialization.
The HeNormal class initializes the weight tensors with samples drawn from a truncated normal distribution with zero mean and variance 2 / fan_in, where fan_in is the number of input units in the weight tensor. This is a key concept to understand.
To use HeNormal, you'll need to calculate the stddev, which is sqrt(2 / fan_in). This will give you a good starting point for your neural network.
HeUniform, on the other hand, draws samples from a uniform distribution within [-limit, limit], where limit = sqrt(6 / fan_in). This is a different approach, but one that can be just as effective.
To help you visualize the difference, consider the following table:
By choosing the right initializer, you can set your neural network up for success.
Other Initializers
You can pass built-in initializers to layers using their string identifier, making it easier to implement different initialization techniques.
All built-in initializers are available as part of the keras.initializers module, providing a range of options for weight initialization.
Some built-in initializers are designed for specific use cases, such as the `VarianceScaling` initializer, which is useful for layers with a large number of weights.
The `VarianceScaling` initializer is a popular choice for weight initialization in deep neural networks, as it helps to stabilize the training process.
A unique perspective: Ai Weight Training
Custom Initializers
Custom initializers can be created by implementing a subclass of keras.initializers.Initializer, which allows for configuration via various arguments.
You can implement a custom initializer by creating a class that inherits from Initializer and defining a __call__ method with the signature required by Keras.
To make your custom initializer serializable, you can also implement the get_config and from_config methods.
However, if your class has the same constructor arguments as its keys in the config, you can skip implementing from_config.
Ones
The ones initializer generates tensors initialized to 1, available as part of the keras.initializers module.
You can also use the shortcut function ones to achieve the same result.
Batch norm layers usually have their weights and biases initialized as 1s and 0s, respectively, which reduces the dependency on weight initialization strategies.
Initializing weights as 1s can lead to the symmetry problem, where all the neurons in one layer will have the same value and represent only one feature during forward propagation.
This causes the network to fail to learn different features from the input, making it perform poorly.
Creating Custom Initializers

You can create custom initializers by subclassing keras.initializers.Initializer. This allows you to configure your initializer via various arguments.
To implement a custom initializer, you should implement the __call__ method with the following signature: Optionally, you can also implement the method get_config and the class method from_config to support serialization.
Here's a simple example of a random normal initializer: a random normal initializer. Note that we don't have to implement from_config in this example since the constructor arguments of the class are the same as the keys in the config returned by get_config.
The HeNormal class draws samples from a truncated normal distribution centered on 0 with stddev = sqrt(2 / fan_in) where fan_in is the number of input units in the weight tensor. The LecunNormal class draws samples from a truncated normal distribution centered on 0 with stddev = sqrt(1 / fan_in) where fan_in is the number of input units in the weight tensor.

To create a custom weight initialization function, you can define a function that takes the input sz (the size of the layer weights) and returns an array of weights. For example, the leakyHe function takes the input sz and returns an array of weights given by the He Initializer for convolution layers followed by a leaky ReLU layer.
Here's a summary of the key points to keep in mind when creating custom initializers:
By following these guidelines and using the built-in initializers as a starting point, you can create custom initializers that meet the specific needs of your project.
Featured Images: pexels.com





