Introduction<\/mdspan><\/h2>\nWhether you\u2019re preparing for interviews or building Machine Learning systems at your job, model compression has become a must-have skill. In the era of LLMs, where models are getting larger and larger, the challenges around compressing these models to make them more efficient, smaller, and usable on lightweight machines have never been more relevant.<\/p>\n
In this article, I will go through four fundamental compression techniques that every ML practitioner should understand and master. I explore pruning, quantization, low-rank factorization, and Knowledge Distillation<\/a>, each offering unique advantages. I will also add some minimal PyTorch code samples for each of these methods.<\/p>\nI hope you enjoy the article!<\/p>\n
\n
\n\n- Feel free to connect on\u00a0LinkedIn<\/a>!<\/li>\n
- Follow me on\u00a0GitHub<\/a>\u00a0and visit my website<\/a> for more content.<\/li>\n<\/ul>\n
\nModel pruning<\/h2>\n
Pruning is probably the most intuitive compression technique. The idea is very simple: remove some of the weights of the network, either randomly or remove the \u201cless important\u201d ones<\/strong>. Of course, when we talk about \u201cremoving\u201d weights in the context of neural networks, it means setting the weights to zero<\/strong>.<\/p>\n![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/05\/blog_img1-1024x557.png?resize=1024%2C557&ssl=1\")
Model pruning (Image by the author and ChatGPT | Inspiration: [3])<\/figcaption><\/figure>\nStructured vs unstructured pruning<\/h3>\n
Let\u2019s start with a simple heuristic<\/strong>: removing weights smaller than a threshold.<\/p>\n[ w\u2019_{ij} = begin{cases} w_{ij} & text{if } |w_{ij}| ge theta_0 \\
0 & text{if } |w_{ij}| < theta_0
end{cases} ]<\/p>\n
Of course, this is not ideal because we would need to find a way to find the right threshold<\/strong> for our problem! A more practical approach is to remove a specified proportion of weights with the smallest magnitudes<\/strong> (norm) within one layer. There are 2 common ways of implementing pruning in one layer:<\/p>\n\n- \nStructured pruning<\/strong>: remove entire components of the network (e.g. a random row from the weight tensor, or a random channel in a convulational layer)<\/li>\n
- \nUnstructured pruning<\/strong>: remove individual weights regardless of their positions and of the structure of the tensor<\/li>\n<\/ul>\n
We can also use global pruning<\/strong> with either of the two above methods. This will remove the chosen proportion of weights across multiple layers, and potentially have different removal rates depending on the number of parameters in each layer.<\/p>\nPyTorch makes this pretty straightforward (by the way, you can find all code snippets in my GitHub repo<\/a>).<\/p>\nimport torch.nn.utils.prune as prune\n\n# 1. Random unstructured pruning (20% of weights at random)\nprune.random_unstructured(model.layer, name=\"weight\", amount=0.2) \n\n# 2. L1\u2011norm unstructured pruning (20% of smallest weights)\nprune.l1_unstructured(model.layer, name=\"weight\", amount=0.2)\n\n# 3. Global unstructured pruning (40% of all weights by L1 norm across layers)\nprune.global_unstructured(\n [(model.layer1, \"weight\"), (model.layer2, \"weight\")],\n pruning_method=prune.L1Unstructured,\n amount=0.4\n) \n\n# 4. Structured pruning (remove 30% of rows with lowest L2 norm)\nprune.ln_structured(model.layer, name=\"weight\", amount=0.3, n=2, dim=0)<\/code><\/pre>\nNote: if you have taken statistics classes, you probably learned regularization-induced methods that also implicitly prune some weights during training, by using L0 or L1 norm regularization. Pruning differs from that because it is applied as a post-Model Compression<\/a><\/strong> technique<\/em><\/p>\nWhy does pruning work? The Lottery Ticket Hypothesis<\/h3>\n![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/05\/lottery-ticket-1-1024x683.png?resize=1024%2C683&ssl=1\")
Image generated by ChatGPT<\/figcaption><\/figure>\nI would like to conclude that section with a quick mention of the Lottery Ticket Hypothesis<\/strong>, which is both an application of pruning and an interesting explanation of how removing weights can often improve a model. I recommend reading the associated paper ([7]) for more details.<\/p>\nAuthors use the following procedure:<\/p>\n
\n- Train the full model to convergence<\/li>\n
- Prune the smallest-magnitude weights (say 10%)<\/li>\n
- Reset the remaining weights to their original initialization values<\/li>\n
- Retrain this pruned network<\/li>\n
- Repeat the process multiple times <\/li>\n<\/ol>\n
After doing this 30 times, you end up with only 0.930<\/sup> ~ 4% of the original parameters. And surprisingly, this network can do as well as the original one<\/strong>.<\/p>\nThis suggests that there is important parameter redundancy. In other words, there exists a sub-network<\/strong> (\u201ca lottery ticket\u201d) that actually does most of the work! <\/p>\n\nPruning is one way to unveil this sub-network.<\/p>\n
\n\n<\/blockquote>\n<\/blockquote>\n\nI recommend this very good video that covers the topic!<\/summary>\n\n
Whether you\u2019re preparing for interviews or building Machine Learning systems at your job, model compression has become a must-have skill. In the era of LLMs, where models are getting larger and larger, the challenges around compressing these models to make them more efficient, smaller, and usable on lightweight machines have never been more relevant.<\/p>\n
In this article, I will go through four fundamental compression techniques that every ML practitioner should understand and master. I explore pruning, quantization, low-rank factorization, and Knowledge Distillation<\/a>, each offering unique advantages. I will also add some minimal PyTorch code samples for each of these methods.<\/p>\n I hope you enjoy the article!<\/p>\n \n Pruning is probably the most intuitive compression technique. The idea is very simple: remove some of the weights of the network, either randomly or remove the \u201cless important\u201d ones<\/strong>. Of course, when we talk about \u201cremoving\u201d weights in the context of neural networks, it means setting the weights to zero<\/strong>.<\/p>\n Let\u2019s start with a simple heuristic<\/strong>: removing weights smaller than a threshold.<\/p>\n [ w\u2019_{ij} = begin{cases} w_{ij} & text{if } |w_{ij}| ge theta_0 \\ Of course, this is not ideal because we would need to find a way to find the right threshold<\/strong> for our problem! A more practical approach is to remove a specified proportion of weights with the smallest magnitudes<\/strong> (norm) within one layer. There are 2 common ways of implementing pruning in one layer:<\/p>\n We can also use global pruning<\/strong> with either of the two above methods. This will remove the chosen proportion of weights across multiple layers, and potentially have different removal rates depending on the number of parameters in each layer.<\/p>\n PyTorch makes this pretty straightforward (by the way, you can find all code snippets in my GitHub repo<\/a>).<\/p>\n Note: if you have taken statistics classes, you probably learned regularization-induced methods that also implicitly prune some weights during training, by using L0 or L1 norm regularization. Pruning differs from that because it is applied as a post-Model Compression<\/a><\/strong> technique<\/em><\/p>\n I would like to conclude that section with a quick mention of the Lottery Ticket Hypothesis<\/strong>, which is both an application of pruning and an interesting explanation of how removing weights can often improve a model. I recommend reading the associated paper ([7]) for more details.<\/p>\n Authors use the following procedure:<\/p>\n After doing this 30 times, you end up with only 0.930<\/sup> ~ 4% of the original parameters. And surprisingly, this network can do as well as the original one<\/strong>.<\/p>\n This suggests that there is important parameter redundancy. In other words, there exists a sub-network<\/strong> (\u201ca lottery ticket\u201d) that actually does most of the work! <\/p>\n Pruning is one way to unveil this sub-network.<\/p>\n \n<\/blockquote>\n<\/blockquote>\n
\n\n
\nModel pruning<\/h2>\n
Structured vs unstructured pruning<\/h3>\n
0 & text{if } |w_{ij}| < theta_0
end{cases} ]<\/p>\n\n
import torch.nn.utils.prune as prune\n\n# 1. Random unstructured pruning (20% of weights at random)\nprune.random_unstructured(model.layer, name=\"weight\", amount=0.2) \n\n# 2. L1\u2011norm unstructured pruning (20% of smallest weights)\nprune.l1_unstructured(model.layer, name=\"weight\", amount=0.2)\n\n# 3. Global unstructured pruning (40% of all weights by L1 norm across layers)\nprune.global_unstructured(\n [(model.layer1, \"weight\"), (model.layer2, \"weight\")],\n pruning_method=prune.L1Unstructured,\n amount=0.4\n) \n\n# 4. Structured pruning (remove 30% of rows with lowest L2 norm)\nprune.ln_structured(model.layer, name=\"weight\", amount=0.3, n=2, dim=0)<\/code><\/pre>\nWhy does pruning work? The Lottery Ticket Hypothesis<\/h3>\n
\n
\n
\n
I recommend this very good video that covers the topic!<\/summary>\n