GGUF, AWQ, GPTQ, EXL2 and FP8: LLM Quantization Formats Compared

LLM quantization is what makes it possible to run large language models on ordinary hardware: it stores a model’s weights at lower precision so they take far less memory. This guide compares the five formats most people meet when running models locally, GGUF, AWQ, GPTQ, EXL2 (now EXL3) and FP8, and explains which suits which hardware.

Update, 3 November 2026: tools and figures revised

This guide has been revised. AutoAWQ and AutoGPTQ, the libraries used in the original examples, are now archived: AutoAWQ says it is deprecated and has been adopted by the vLLM project’s LLM Compressor, and AutoGPTQ recommends GPTQModel instead [8][7]. ExLlamaV2 is archived and development continues on ExLlamaV3 and its EXL3 format [4]. llama.cpp now lives at ggml-org, builds with CMake and runs models with llama cli [1]. The unsourced speed and perplexity tables in the first version have been replaced with llama.cpp’s published Llama 3.1 8B measurements, the description of k-quants and AWQ has been corrected, and FP8 is no longer described as needing data-centre GPUs [1][10].

Why LLM quantization matters: the memory problem

The biggest barrier to running modern models at home is memory, not compute. A 70-billion-parameter model stored at 16-bit precision needs about 140 GB just for its weights, two bytes per parameter. A 24 GB consumer graphics card cannot hold a fraction of that.

LLM quantization reduces the precision of those weights from 16-bit floating point to 8-bit, 4-bit or fewer bits. llama.cpp’s own documentation shows the effect: Llama 3.1 70B shrinks from 280.9 GB in its original 32-bit files to 43.1 GB at the Q4_K_M setting, and Llama 3.1 8B from 32.1 GB to 4.9 GB [1]. Some accuracy is lost along the way, and the formats below differ mainly in how they decide where that loss falls.

TSN’s guide to self-hosting LLMs covers the wider setup; this piece concentrates on the file formats.

How LLM quantization works, in brief

A neural network is mostly large matrices of numbers, its weights. Training usually keeps them at 32-bit or 16-bit floating point. Quantization maps those values onto a much smaller set: an 8-bit integer has 256 possible values and a 4-bit integer only 16.

The simplest scheme spreads those values evenly between a tensor’s minimum and maximum. That is rarely ideal, because weight distributions are uneven and contain outliers. Better methods quantize small groups of weights separately, each with its own scale, and treat the most influential weights or channels with extra care.

Most local formats use post-training LLM quantization: the finished model is converted without retraining. Some methods run a small calibration dataset through the model first to see which weights matter most for real inputs. AWQ and GPTQ both do this [2][3].

The five LLM quantization formats compared

GGUF (llama.cpp)

GGUF is the file format of llama.cpp, the C/C++ inference engine, and of the many tools built on it [1]. A single GGUF file holds the weights plus metadata such as the tokenizer and model architecture, and it runs on CPUs, GPUs or a mix of the two, which is why it is the usual choice for laptops, Macs and machines with modest graphics cards.

GGUF offers many quantization types. The “k-quants” (Q2_K to Q6_K) split weights into blocks and sub-blocks with their own scales, and the S/M/L variants keep some sensitive tensors at higher precision; the “i-quants” (IQ1 to IQ4) push quality further at very low bit rates [1]. On llama.cpp’s Llama 3.1 8B figures, Q4_K_M averages 4.89 bits per weight and Q8_0 8.50 [1].

Best for: CPU or mixed CPU/GPU inference, Apple Silicon, edge devices and anyone who wants one file that runs almost everywhere.

# Build llama.cpp
cmake -B build
cmake --build build --config Release

# Convert a Hugging Face model, then quantize to Q4_K_M
python convert_hf_to_gguf.py /path/to/model --outfile model-f16.gguf --outtype f16
./build/bin/llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M

AWQ (Activation-aware Weight Quantization)

AWQ comes from Lin and colleagues’ 2023 paper. Its observation is that weights are not equally important: protecting only about 1% of “salient” weights, identified by looking at activations rather than the weights themselves, greatly reduces quantization error [2]. Rather than keeping those weights at higher precision, which is awkward for hardware, AWQ scales the important channels before quantizing everything to the same low bit width, usually 4-bit [2].

Best for: GPU serving, especially with vLLM, where AWQ checkpoints are widely available. New AWQ conversions are now done with LLM Compressor rather than the archived AutoAWQ [8][9].

GPTQ

GPTQ, from Frantar and colleagues’ 2022 paper, quantizes a model one layer at a time and uses approximate second-order information to adjust the remaining weights so that each layer’s output changes as little as possible [3]. The paper reports quantizing 175-billion-parameter GPT models to 3 or 4 bits per weight in about four GPU hours with negligible accuracy loss [3]. A common setting is 4-bit with a group size of 128.

Best for: broad tool support; GPTQ models load in Hugging Face Transformers, vLLM and many other frameworks [9][11]. For new quantization work, AutoGPTQ points users to GPTQModel [7].

from transformers import AutoModelForCausalLM, AutoTokenizer, GPTQConfig

model_id = "your-org/your-model"
tokenizer = AutoTokenizer.from_pretrained(model_id)
config = GPTQConfig(bits=4, group_size=128, dataset="c4", tokenizer=tokenizer)
model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=config, device_map="auto")
model.save_pretrained("your-model-gptq")

EXL2 and EXL3 (ExLlama)

EXL2 was the format of ExLlamaV2, an engine built for fast inference on consumer GPUs. Its distinctive feature was mixing bit widths within one model, measuring each layer’s sensitivity and spending more bits where they matter, so users could target an average such as 4.0 or 4.5 bits per weight [4]. ExLlamaV2 is now archived; its successor ExLlamaV3 uses EXL3, a format based on the QTIP research method, and also supports 2- to 8-bit cache quantization [4][5].

Best for: single-user inference on NVIDIA consumer GPUs when you want to fit the largest possible model into your VRAM.

# ExLlamaV3: convert a model to EXL3 at a target bitrate
python convert.py -i /path/to/model -o /path/to/output -w /path/to/workdir -b 4.0

FP8 (8-bit floating point)

FP8 keeps a floating-point layout with fewer bits. There are two variants: E4M3 (4 exponent bits, 3 mantissa bits, values up to ±448) and E5M2 (5 exponent bits, 2 mantissa bits, up to ±57,344), trading precision against range [10]. vLLM’s documentation says FP8 computation is supported on NVIDIA GPUs with compute capability 8.9 or higher, which means Ada Lovelace cards such as the RTX 40 series as well as Hopper and Blackwell, and that older Turing-generation GPUs can still run FP8 models as weight-only [10].

Best for: GPU serving where 8-bit quality is wanted with less memory than 16-bit. At 8 bits it saves less memory than the 4-bit formats.

Comparing quality and speed with published figures

Comparisons between formats are only meaningful when measured on the same model, hardware and settings, and most published tables are not. llama.cpp’s documentation gives one consistent set of measurements for Llama 3.1 8B [1]:

Text generation generally gets faster as files shrink, because generating each token is limited mostly by how fast weights can be read from memory. Quality goes the other way. llama.cpp measures it with perplexity and KL divergence against the original model [1], and ExLlamaV3 publishes its own charts comparing formats across bits per weight [5]. As a rule of thumb, 8-bit is close to the original, around 4 to 5 bits per weight is the usual compromise, and below about 3 bits quality drops noticeably. Code and maths tend to suffer before casual chat does, so test on your own tasks.

Which format to choose for your hardware

• CPU only, a Mac, or a GPU too small for the whole model: GGUF. It is the format built for partial GPU offload, with some layers on the GPU and the rest on the CPU. Tools such as Hugging Face’s Llama Windows app and Ollama use llama.cpp-style models.
• NVIDIA GPU, single user, maximum model size: EXL3 with ExLlamaV3, or GGUF.
• Serving many users with vLLM: AWQ, GPTQ or FP8. Check vLLM’s quantization hardware table for your exact GPU, because support for AMD and other accelerators varies by method [9].
• Recent NVIDIA GPU and 8-bit quality wanted: FP8 [10].

Remember that the model’s weights are not the only thing in memory. The KV cache for the context window grows with context length and can take several more gigabytes, so leave headroom. For very small machines, see TSN’s guide to self-hosting small LLMs from a Raspberry Pi to a MacBook Pro.

Common problems and fixes

Out of memory during inference. Reduce the context length (for example --ctx-size 4096 in llama.cpp), choose a smaller quantization, or offload fewer layers to the GPU with -ngl [1].

Slower than expected. Check that your build actually uses your GPU (llama.cpp’s build guide covers CUDA, Metal, Vulkan and other back-ends), and that the whole model fits in VRAM; spilling to system memory is a common cause of slow speeds.

Noticeably worse answers. Move up a step, for example from Q4_K_M to Q5_K_M or Q6_K. For GPTQ, calibration data close to your own use can help, and enabling act-order (desc_act=True) often improves accuracy at some cost to speed [7].

Model will not load. Update your tools, since support for new architectures arrives constantly, and check that the format matches the engine: GGUF for llama.cpp, EXL3 for ExLlamaV3, and so on.

LLM quantization trades precision for access, and the tooling keeps moving

LLM quantization is the reason a 70B-class model can run on a well-equipped desktop rather than a rack of data-centre GPUs, and the reason an 8B model fits comfortably on a laptop. That widens who can run AI privately, offline and without sending data to a cloud provider, which is a real benefit for researchers, small companies and anyone handling sensitive material. The cost is a measurable drop in quality, and it is not spread evenly: tasks that depend on exact detail, such as code and arithmetic, show it first.

The other lesson from the past year is how quickly the tooling changes. Within months, two of the best-known quantization libraries were archived, ExLlamaV2 gave way to ExLlamaV3, and llama.cpp changed its build system and command names. The underlying methods, AWQ’s activation-aware scaling, GPTQ’s error-correcting rounding, GGUF’s block-wise k-quants, have lasted much better than the libraries that first packaged them. So pick a format for your hardware, prefer projects that are actively maintained, and check quality on your own prompts rather than trusting a single benchmark table. Hardware is shifting too: machines such as AMD’s Ryzen AI Halo developer platform with up to 192GB of memory make larger models practical at higher precision.

Related on TSN:

• Self-Hosting LLMs in 2026: The Complete Setup Guide
• Self-Hosting Small LLMs: From Raspberry Pi to MacBook Pro
• Prompt Engineering for Self-Hosted LLMs
• Running Vision LLMs Locally
• AI Glossary for Developers: 60 Key Terms Defined and Sourced

Sources

1. ggml-org, llama.cpp repository, README, build guide and tools/quantize README (Llama 3.1 sizes and quantization measurements), read 3 November 2026. github.com/ggml-org/llama.cpp
2. Lin et al., “AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration”, arXiv:2306.00978, 2023. arxiv.org/abs/2306.00978
3. Frantar et al., “GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers”, arXiv:2210.17323, 2022. arxiv.org/abs/2210.17323
4. turboderp-org, ExLlamaV2 repository (archived; development continues on ExLlamaV3). github.com/turboderp-org/exllamav2
5. turboderp-org, ExLlamaV3 repository and EXL3 quantization notes, read 3 November 2026. github.com/turboderp-org/exllamav3
6. Tseng et al., “QTIP: Quantization with Trellises and Incoherence Processing”, arXiv:2406.11235, 2024. arxiv.org/abs/2406.11235
7. AutoGPTQ repository (archived; recommends GPTQModel). github.com/AutoGPTQ/AutoGPTQ
8. AutoAWQ repository (archived; deprecated and adopted by vLLM’s LLM Compressor). github.com/casper-hansen/AutoAWQ
9. vLLM documentation, Quantization. docs.vllm.ai/en/latest/features/quantization
10. vLLM documentation, FP8 W8A8 (E4M3/E5M2 and GPU support), read 3 November 2026. docs.vllm.ai/en/latest/features/quantization/llm_compressor/fp8
11. Hugging Face, Transformers quantization documentation. huggingface.co/docs/transformers/main_classes/quantization
12. Dettmers et al., “LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale”, arXiv:2208.07339, 2022. arxiv.org/abs/2208.07339
13. Dettmers et al., “QLoRA: Efficient Finetuning of Quantized LLMs”, arXiv:2305.14314, 2023. arxiv.org/abs/2305.14314
14. Xiao et al., “SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models”, arXiv:2211.10438, 2022. arxiv.org/abs/2211.10438