
Model compression is a crucial technique in AI that helps reduce the size of machine learning models without sacrificing their accuracy. This technique is particularly useful for deploying models in resource-constrained environments.
By compressing models, developers can save storage space, reduce latency, and improve the overall efficiency of their applications. For instance, a study found that compressing a model by 90% can reduce its storage requirements by 9 times.
Model compression techniques can be broadly classified into two categories: model pruning and knowledge distillation. Model pruning involves removing unnecessary neurons and connections from the model, while knowledge distillation involves transferring knowledge from a larger model to a smaller one.
Discover more: Databricks Automl
What Is Model Compression
Model compression is a technique used to simplify a model without significantly diminishing its accuracy. The goal is to create a smaller and/or faster model that uses less RAM and energy.
A simplified model is one that has fewer and/or smaller parameters, which means it can run on devices with limited computing resources. For example, in space, computing resources face strict size and power constraints, making model compression a necessity.
Model compression can be achieved through various methods, including pruning, quantization, low-rank approximation and sparsity, knowledge distillation, and neural architecture search (NAS). These methods can be used individually or in combination to optimize model performance.
Here are some popular methods for achieving compressed models:
- Pruning
- Quantization
- Low-rank approximation and sparsity
- Knowledge distillation
- Neural Architecture Search (NAS)
Quantization is a technique that reduces the precision of parameters, usually converting from 32-bit float to 8-bit integer representations. This can achieve around 4x model compression, but may cause the model to diverge from its original converged state.
To address this, "quantization aware training" fine-tunes the model after quantization on additional data to regain performance. "Post training quantization" skips this fine-tuning step and instead applies heuristics to alter the quantized weights directly to try preserving model accuracy.
The goal of quantization is to dramatically shrink model size with minimal impact on model predictions. With proper hardware support, processing deep learning kernels using fewer bits can be faster and more memory efficient.
Related reading: Bag-of-words Model
Techniques for Model Compression
Model compression is a technique used to simplify a model without significantly diminishing its accuracy. A simplified model is one that is reduced in size and/or latency from the original.
There are several popular methods for achieving compressed models, including pruning, quantization, low-rank approximation and sparsity, knowledge distillation, and neural architecture search (NAS).
Some of these methods can be used individually, but a complete model compression pipeline might integrate several approaches, each with its own benefits and drawbacks.
Here are some specific techniques for model compression:
- Weight SVD, Spatial SVD, and Channel Pruning are supported by AIMET for model compression.
- Rank-rounding and per-layer fine-tuning are optional techniques that can be used to get better compression results.
These techniques can be used to reduce the size and latency of a model, making it more suitable for deployment in space or other environments with strict size and power constraints.
By using these techniques, developers can create compressed models that are both smaller and faster than their original counterparts, without sacrificing too much accuracy.
Types of Pruning
There are three main types of pruning algorithms: Unstructured Pruning, Structured Pruning, and Iterative Pruning.
Unstructured Pruning is a type of pruning where there are no constraints on which weights can be pruned, resulting in sparse weights. This type of pruning focuses on individual weights rather than larger structures.
Structured Pruning, on the other hand, involves pruning larger structures such as whole neurons, convolution filters, or channels. This type of pruning directly leads to a reduction in memory, which can translate to a speedup in inference time.
Structured Pruning can be further divided into two sub-types: local and global pruning. Local pruning involves pruning specific layers, while global pruning involves pruning across the whole network.
Here's a breakdown of the different types of pruning:
Iterative Pruning involves pruning over several iterations while training or fine-tuning a model, whereas OneShot pruning prunes the model and/or weights in just one pass. It's recommended to use Iterative Pruning as OneShot pruning generally leads to a lot of accuracy degradation.
Worth a look: What Are the Two Main Types of Generative Ai Models
Tensorization
Tensorization is a model compression technique that involves decomposing the weight tensors of a neural network into smaller tensors with lower ranks.
This process is used to reveal underlying patterns and structures within the data, while also reducing its size. Tensorization has many practical use cases in machine learning, such as detecting latent structure in the data for representing temporal or multi-relational data.
The goal of tensorization is to reduce the number of parameters in the network while maintaining its accuracy. Tensorization can be done using different methods, such as singular value decomposition (SVD), tensor train decomposition, or Tucker decomposition.
Tensorization is most useful when a model can be optimized at a mathematical level, i.e. when the model's layers can be further broken down into lower ranking tensors to reduce the number of parameters needed for computation.
Tensor decomposition algorithms aim to factorize a given tensor into a set of smaller tensors, which represent different aspects or modes of the data.
Check this out: Model Drift vs Data Drift
Types of Knowledge Distillation Algorithms
Knowledge distillation is a powerful technique for model compression, and it comes in several flavors. Offline distillation is the most common type, where a pre-trained teacher model guides the student model. This is the easiest to implement.
Offline distillation involves using a pre-trained teacher model to guide the student model. It's a straightforward approach that works well in many cases. The teacher model is used to generate soft targets, which are probability distributions over the classes, to help the student model learn.
There are two other types of knowledge distillation algorithms: online distillation and self distillation. Online distillation involves updating both the teacher and student models simultaneously in a single end-to-end training process. This is useful when a pre-trained teacher model isn't available.
Here are the different types of knowledge distillation algorithms:
Self distillation is a special case of online distillation, where the same model is used as both the teacher and student. This can be useful in certain situations, but it's worth noting that it can also be more computationally expensive.
Model Compression Techniques
Model compression is a technique used to achieve a simplified model that is reduced in size and/or latency from the original, without significantly diminished accuracy. This is desirable for deploying models in space, where computing resources face both strict size and power constraints.
There are several model compression techniques, including Pruning, Quantization, Low-rank approximation, and Knowledge distillation. Pruning involves removing unnecessary neurons or connections in the network, while Quantization reduces the precision of the model's weights and activations. Low-rank approximation reduces the number of parameters in the model by approximating the weights as a product of two smaller matrices.
Some popular pruning techniques include Unstructured Pruning and Structured Pruning. Unstructured Pruning removes individual weights, resulting in sparse weights, while Structured Pruning focuses on pruning larger structures such as whole neurons, convolution filters, channels, etc. Quantization can be categorized into two main approaches: Quantization Aware Training (QAT) and Post-Training Quantization (PTQ). QAT emulates inference-time quantization during training, while PTQ directly applies quantization after training.
Here are some common model compression techniques:
- Pruning
- Quantization
- Low-rank approximation
- Knowledge distillation
Note: This list is not exhaustive, but it covers some of the most popular techniques mentioned in the article section facts.
Low-Rank Approximation
Low-Rank Approximation is a technique used in model compression to simplify large models by reducing the number of parameters and improving computational efficiency. This is achieved by approximating a high-dimensional tensor with a lower-rank tensor, which represents the same information in a more compact form.
In the context of Convolutional Neural Networks (CNNs), Low-Rank Approximation can be used to reduce the number of parameters and improve inference speed. By decomposing a high-dimensional tensor into smaller tensors, Low-Rank Approximation can help to reduce the computational complexity of a model.
Low-Rank Approximation can be achieved through various methods, including Weight/Spatial SVD and Channel Pruning, which are techniques used in AIMET to decompose layers and reduce the number of parameters. These methods can be used to generate more optimal models for running on embedded targets.
Here are some key benefits of using Low-Rank Approximation in model compression:
- Reduced model size and latency
- Improved computational efficiency
- Increased inference speed
Low-Rank Approximation has been extensively researched in the field of machine learning, with various algorithms and techniques being developed to achieve this goal. Some of the most popular algorithms include Canonical Polyadic Decomposition (CP or PARAFAC), Tensor Train Decomposition (TT), and Tucker Decomposition, which are used to factorise a given tensor into a set of smaller tensors.
Per-Layer Fine-Tuning
Per-layer fine-tuning is a technique that can be used to improve compression results. It's invoked by AIMET after compressing every layer that was selected for compression.
The user is responsible for choosing appropriate learning-rates and other training parameters for fine-tuning. This is crucial to get the best results from per-layer fine-tuning.
Per-layer fine-tuning can help in scenarios where a sharp degradation in accuracy is noticed after compression. It's a way to adjust the model to better fit the compressed layers.
The user needs to carefully pick the learning rates and learning-rate-decay parameters to be used during fine-tuning. This requires some trial and error to get it right.
Here are some APIs that can be used to update the fine-tuning status:
- pruner.update_epoch(epoch) - should be invoked in every epoch
- pruner.step() - should be invoked after each minibatch
Note that most algorithms do not require calling these APIs, but it's allowed if needed.
Intriguing read: Fine Tune Embedding Models
Choose an Algorithm
Choosing an algorithm for model compression can be a bit overwhelming, but let's break it down.
NNI provides a range of compression algorithms, including pruning and quantization algorithms.
To choose a compression algorithm, you'll want to consider what type of pruner is best for your model.
Some pruners prune your model once at the beginning, while others prune it iteratively during training.
You can choose a pruner based on your model's needs, and some pruners are more suitable for certain types of models.
By choosing the right algorithm, you can compress your model effectively and improve its performance.
Readers also liked: Difference between Model and Algorithm in Machine Learning
Detailed Usage Guide
To apply model compression on a user model, you can use the example code found in the PyTorch code section. This code can be used with other compression algorithms in the nni.compression package.
The PyTorch code is implemented in the nni.compression.torch module, and you can refer to Pruner and Quantizer for details on the supported algorithms. If you want to use knowledge distillation, you can refer to the KDExample.
A compression algorithm is first instantiated with a config_list passed in, which will be described later. The function call pruner.compress() modifies the user-defined model and inserts masks, which can be adjusted at runtime by the algorithms.
The pruner.compress() function does not include fine-tuning logic, so users need to write their own fine-tuning logic after using this function.
The config_list passed to the compression algorithm has different keys, including common keys supported by all compression algorithms. Some of these common keys are used by the pruner.compress() function.
Here are some common keys supported by all compression algorithms:
To implement dynamic quantization in PyTorch, you can use the torch.quantization.quantize_dynamic function. This function takes in a model and other arguments to produce a quantized model.
Config List Specification
A config list is a Python list object where each element is a dictionary that specifies the configuration for model compression.
To specify the configuration, you can use a list of dictionaries, where each dictionary contains specific keys that define the compression settings.
The `op_types` key specifies the types of operations to be compressed, and you can use 'default' to follow the algorithm's default setting.
The `op_names` key allows you to specify the names of specific operations to be compressed, and if omitted, operations will not be filtered by name.
The `exclude` key is used to exclude certain operations from compression, and its default value is False.
Here's a breakdown of the keys you can use in a config list:
In addition to these keys, you may need to use other keys specific to the algorithm you're using, such as `quant_types` and `quant_bits` for quantization algorithms.
The `quant_types` key specifies the type of quantization to apply, and the `quant_bits` key specifies the number of bits to use for quantization.
For example, you can use the following config list to specify different quantization settings for different operations:
```python
config_list = [
{'op_types': 'weight', 'quant_bits': 8},
{'op_names': 'layer1', 'quant_bits': 16},
{'op_names': 'layer2', 'quant_bits': 32}
]
```
Export Model
You can export a compressed model using the API, which stores the state_dict of the sparse model weights in a file called model.pth. This file can be loaded later using torch.load('model.pth').
The masked weights in the exported model.pth are zero.
You can also export the mask_dict and pruned model in ONNX format by specifying the input shape.
For a really speedy compressed model, check out NNI model speedup for details.
You can export model weights to a file and the generated masks to a file as well.
Exporting an ONNX model is also supported.
Post-Training Static
Post-Training Static Quantization is a technique that can significantly speed up model execution time. This is achieved by knowing the distribution of activations, which can be recorded by feeding batches of data through the network and computing the resulting distributions.
By inserting "observer" modules at different points, we can record the distributions of the different activations. This information is then used to determine how specifically the different activations should be quantized at inference time.
One simple technique is to divide the entire range of activations into 256 levels, but more sophisticated methods are also supported. This additional step allows us to pass quantized values between operations instead of converting these values to floats and back to ints between every operation.
The following features are supported to optimize static quantization:
- Setting up observers
- Running with some data
- Exporting to a new statically quantized model
Here's an example of setting up the observers, running it with some data, and then exporting to a new statically quantized model:
Setting the Scope for the Survey
TensorFlow initially had a significant advantage in deployment due to its well-established ecosystem of tools and Model Optimization Toolkit. However, PyTorch has been rapidly closing the gap with the introduction of torch.compile and built-in Quantization support in Torch 2.0.
PyTorch boasts a vast collection of models, with 148,605 models available on Hugging Face at the time of writing. This is a staggering number compared to the 10,270 TensorFlow models available on the same platform.
PyTorch benefits from a thriving ecosystem of third-party compression tools specifically developed for PyTorch models. This ecosystem provides a wide range of options for compressing and optimizing PyTorch models.
Considering these factors, it's evident that PyTorch's native quantization and accompanying third-party tools have become a focal point in the field.
Model Compression Tools
Model compression tools are designed to reduce the size of neural networks while maintaining their accuracy. This is achieved by pruning, quantization, and knowledge distillation.
Pruning is a technique that removes unnecessary connections in the network, reducing the number of parameters and operations. For example, the article mentions that pruning can reduce the number of parameters by up to 90%.
Quantization reduces the precision of the model's weights and activations, allowing for smaller data storage and faster processing. The article notes that quantization can reduce the model size by up to 32 times.
Knowledge distillation is a technique that transfers knowledge from a large teacher model to a smaller student model. This allows the student model to learn from the teacher model and achieve similar accuracy with a smaller size.
Write Configuration
To write a configuration for model pruning, you need to specify the layers you want to prune and the desired sparsity level.
You can start by identifying the layers you want to prune, such as BatchNorm2ds. The configuration should include the layers to be pruned and their corresponding sparsity levels.
For example, you can prune all BatchNorm2ds to a sparsity of 0.7 while keeping other layers unpruned.
The specification of the configuration can be found in the documentation, but note that different pruners may have their own defined fields, such as start_epoch in AGP pruner.
Export Result
You can export the compressed model using the API, which stores the state_dict of the sparse model weights in model.pth. This file can be loaded later using torch.load('model.pth'), and in the exported model.pth, the masked weights are zero.
Exporting the compressed model allows you to really speed up the model, and for more details on how to do this, refer to NNI model speedup.
The compressed model can also be exported in ONNX format, where you need to specify the input shape. This is useful for further processing or analysis of the model.
After training the pruned model, you can export the model weights to a file and the generated masks to a file as well. Exporting the ONNX model is also supported, making it easy to share or use the compressed model in other applications.
Device Support
Device support is crucial for model compression tools. You can run quantized models on CPU with x86 and ARM backends.
The set of available operators and quantization numerics depend on the backend being used. Both CPU and GPU can run quantization aware training, which occurs in full floating point.
You can specify the backend for quantized models by doing so. This allows for more flexibility in model deployment.
Quantization aware training is typically used in CNN models when post-training static or dynamic quantization doesn’t yield sufficient accuracy. This can occur with models that are highly optimized for small size, such as Mobilenet.
Here's a brief summary of supported backends for quantized models:
Integration in torchvision
Integration in torchvision is a significant step forward in model compression. PyTorch has enabled quantization for popular models in torchvision.
Googlenet, Inception, Resnet, ResNeXt, Mobilenet, and Shufflenet are some of the models that have benefited from this integration. These models are widely used in computer vision tasks.
The changes have been upstreamed to torchvision in three forms. This means that developers can easily access and use these models with quantization enabled.
Here are the models that have been integrated with quantization:
- Googlenet
- Inception
- Resnet
- ResNeXt
- Mobilenet
- Shufflenet
Frequently Asked Questions
What is meaning model compression?
Model compression is a technique used in deep learning to reduce the size and energy consumption of AI models, making them suitable for use on mobile devices. By compressing models, developers can enable faster and more efficient on-device inference, improving overall user experience.
What is a model in data compression?
A model in data compression refers to a neural network, which is a complex mathematical structure used for machine learning tasks. By compressing these models, developers can make them more deployable on devices with limited resources.
What are the model compression techniques for LLM?
Model compression techniques for LLMs include distillation, quantization, and pruning, which can be combined to create more efficient and effective models. By judiciously applying these techniques, you can compress large language models while maintaining their ability to generalize well.
Featured Images: pexels.com





