WTF is Language Model Quantization?!?

Image by Author | Ideogram

Quantization is a revolutionary technique capable of turning AI and machine learning models more lightweight execution-wise. It reduces the memory requirements to perform inference tasks like predictions, text generation, and so on. This technique has proven especially useful in the space of language models. This article explains why and how.

Quantization in Machine Learning

 
Despite having gained significance in very recent times due to its ability to improve the performance and efficiency of language models, quantization is far from a novel technique, being rooted in numerical analysis and signal processing practices that have been in use for quite a few decades now. In simple terms, quantization converts a digital signal to a format that occupies less space, thereby losing some accuracy. The aim here is to turn the signal smaller so it can be processed more efficiently, so that the advantages of this more efficient processing outweigh the accuracy loss incurred.

This principle was later translated into the AI and machine learning landscape, to make models faster to execute and perform inference, and require less computational cost. Decision trees, random forest ensembles, support vector machines, and linear models like regressors and perceptrons, are typical examples of machine learning models that have historically benefited from quantization. Examples of specific quantization strategies in these models include:

• Quantizing feature values or thresholds in tree-based models and ensembles to fixed-point or integer representations.
• Discretizing continuous features, particularly in probabilistic models like Naïve-Bayes.
• Quantizing neuron inputs and weights to 8-bit integers in neural networks.

Of course, quantization acquires real value in the face of machine learning models that are particularly larger at complex. This is why quantization in complex deep learning architectures turns out to be even more effective than in their smaller, classical machine learning counterparts. And unsurprisingly, in massive models like language models, which have millions to billions of parameters, quantization can make them shine.

To better understand how quantization affects accuracy of a machine learning model, let's have a look at these two "before vs. after" examples.

Model quantization

By looking at the very subtle differences among weights before and after quantization, while more general aspects like their distribution and general properties remain, you can think of quantization as a similar process to turning a raw, high-resolution photo into a pixel art version with slightly fewer colors.
After being quantized, the weights are like pixel art composed by fewer shades, but the overall picture is still fully recognizable.

Quantization in Language Models

 
Quantization has turned into a key strategy to adapt language models, especially large ones (LLMs), to environments with limited computing resources like mobile devices, real-time applications, or local applications. Unlike their application in traditional machine learning models and other neural network-based models, where quantization primarily focuses on reducing the model size and enabling faster inference, in the context of language models, quantization also aims to facilitate model fine-tuning processes without significantly impacting the performance of the resulting fine-tuned model.

Quantizing the billions of parameters a language model has from, say, 32 bits to 8 bits, drastically decreases memory usage and computational load, which is crucial to fluently execute these models in devices with limited capabilities. And best of all: as we saw earlier, the loss of numerical precision would not be that significant.

Among the most popular quantization approaches for training and fine-tuning language models, we have:

• QLoRA: it combines Low Rank Adaptation (LoRA) with quantization, yielding as a result a more efficient model fine-tuning in terms of memory.
• LoftQ: it also integrates LoRA and quantization, but subtly differs from QLoRA in when and how quantization is applied during the model fine-tuning process. While QLoRA quantizes weights before fine-tuning, LoftQ behaves more dynamically, learning better quantized representations in parallel to the training process.
• L4Q: this approach introduces a layer-based design for optimized memory, combining quantization and LoRA (yes, again!) with primary focus on reducing training cost.

These hybrid techniques have been applied in popular language model families like LLaMA, Mistral, and Qwen. Quantized models in these families have demonstrated how it is possible to obtain a competitive performance even with numerical precisions as low as 4-bit.

Wrapping Up

 
In summary, language model quantization not only facilitates their implementation and use for inference in machines and devices with resource constraints. It also facilitates efficient fine-tuning processes, paving the way for a much wider adoption of these advanced forms of AI systems in a variety of settings.

Here is an example article that illustrates, from a practical standpoint point, how to locally set up and use a personal assistant demo application powered by a lightweight quantized language model.
 
 

Iván Palomares Carrascosa is a leader, writer, speaker, and adviser in AI, machine learning, deep learning & LLMs. He trains and guides others in harnessing AI in the real world.