🔍 Read the full analysis: A Closer Look At Transformers’ Llama.cpp Quant Support on ThorstenMeyerAI.com
Get the latest gadgets delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Hugging Face has added support for loading GGUF-quantized checkpoints through the Transformers library’s from_pretrained API. The early rollout targets Apple Silicon and Qwen3.5, and depends on compatible PyTorch and kernel versions; the feature is currently on the main branch.
Hugging Face has added native support for GGUF-quantized models in its Transformers library, as detailed in the original analysis, letting users load supported checkpoints through the from_pretrained API on their own machines. The initial rollout focuses on Apple Silicon Macs and the Qwen3.5 architecture, bringing a widely used local-inference model format into a library commonly used for PyTorch-based model development.
Users can select a GGUF checkpoint hosted on the Hugging Face Hub and pass its file through the gguf_file argument to from_pretrained. Hugging Face says text generation then works through the Transformers interface without additional configuration. The capability is on the library’s main branch, so users need the latest development version; the company has not announced when it will reach a stable release.
The implementation reuses llama.cpp’s ggml kernels to run quantized weights. On Apple Silicon, when the weights remain packed on Metal, Transformers can load compatible ggml/Metal layer kernels and use ggml-org/ggml-attn for attention. If that attention kernel cannot be fetched, the software falls back to standard SDPA and issues a warning. Users can also select SDPA directly. Without a compatible quantization kernel, the loader dequantizes the model, which requires more memory.
Hugging Face lists specific requirements: an Apple Silicon Mac, a PyTorch version supported by the available kernel builds, and compatible recent versions of Transformers and the kernels library. The published kernel builds generally support the two latest PyTorch releases. GGUF checkpoints can also be served through transformers serve, which provides an OpenAI-compatible API on localhost; clients such as Jan or Pi can connect using a custom provider pointed at that endpoint.
GGUF Comes to Transformers Workflows
The change gives developers already using Transformers a way to load GGUF checkpoints without switching to a separate llama.cpp-based application or building an entirely different loading path. GGUF files are widely used for local inference and are available from publishers including Unsloth, LM Studio Community, bartowski and ggml-org. Bringing the format into Transformers may make it easier to use those existing checkpoints in Python projects and local services.
Quantization reduces the memory needed to store model weights, which can make local use practical on laptops and other machines with limited memory. Hugging Face’s example for Unsloth’s Qwen3.5-4B lists a BF16 file at 8.42 GB and a Q4_K_M file at 2.74 GB. That is a file-size comparison for this model and these formats; it does not establish a general memory requirement or predict performance on every machine. Actual results depend on hardware, software versions and the workload.
Hugging Face says its performance reference is llama.cpp and describes a benchmark comparison covering three GGUF checkpoints: a small dense model, a larger dense model and a mixture-of-experts model. The supplied source material does not include the full results, so it does not support a precise performance comparison. The implementation’s reuse of ggml kernels is intended to keep performance close to llama.cpp, but users should treat that as the company’s stated goal until they can assess results on their own hardware.
Quantized Checkpoints and Their Tradeoffs
GGUF is a file format for model weights and related information, including tokenizer data and, optionally, a chat template. It supports multiple quantization levels that trade weight precision for a smaller file and a lower memory footprint. A format such as Q4_K_M uses mostly 4-bit weights while retaining higher precision for some tensors. That label alone does not determine a model’s quality or speed across tasks and devices.
For its Qwen3.5-4B example, Hugging Face lists file sizes of 8.42 GB for BF16, 3.53 GB for Q6_K, 3.14 GB for Q5_K_M and 2.74 GB for Q4_K_M. The company recommends starting with Q4_K_M and trying Q5_K_M or Q6_K when more memory is available. It also says the quality cost of more aggressive quantization depends on the model and task, and advises users to evaluate checkpoints on their own workloads.
The feature follows wider interest in running models locally. Hugging Face co-founder Julien Chaumond recently described Qwen3.6 27B running in the Pi coding agent through llama.cpp on a MacBook Pro. That demonstration is separate from the new Transformers feature and does not establish how every model or task will perform with it.
“We’re adding support for running GGUF models efficiently in transformers, so you can use checkpoints sized for your laptop’s memory through the familiar transformers APIs.”
— Hugging Face announcement
Hardware and Model Coverage Remain Limited
The current support described by Hugging Face targets Apple Silicon and Qwen3.5. The source material gives no schedule for CUDA, Linux or Windows support, and does not say which additional model families will be added or when. It also does not provide the full benchmark figures or enough detail to compare speed across particular hardware configurations.
The feature remains on the main branch, and no stable release date has been announced. Users also need compatible PyTorch and kernel versions to use the quantized kernels; when those are unavailable, dequantization uses more memory. The extent of any quality change from quantization will vary by model and task, according to Hugging Face.
Stable Release and Broader Support
The next concrete milestone is a stable Transformers release that includes GGUF loading; Hugging Face has not given a date. Until then, users interested in the feature can check the library’s main branch and the published requirements for compatible PyTorch and kernel versions.
Broader architecture and device support remain open development areas. The announcement does not commit to particular additions or timelines. Users can follow updates in Hugging Face’s GGUF documentation and kernels library for changes to supported formats and compatibility.
Key Questions
How do users load a GGUF checkpoint?
Choose a checkpoint from the Hugging Face Hub and pass its file through the gguf_file argument to from_pretrained, using a compatible Transformers version.
Which computers and models are supported initially?
The described rollout targets Apple Silicon Macs and the Qwen3.5 architecture. The announcement does not give a schedule for other devices or model families.
Does loading GGUF always use quantized kernels?
No. Compatible kernel and PyTorch versions are required. If a compatible quantization kernel is unavailable, the loader dequantizes the model, which uses more memory.
When will the feature reach a stable release?
Hugging Face has not announced a date. The feature is currently available on the Transformers main branch.
Primary source: Hugging Face · via ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
