Skip to content
Huawei Computing Systems LabSwitzerland
GitHubLike our research?Star a repository to help others discover our work.

Explore CSL

Explore our research and teams

Interactive blogMACaQModel Agnostic Calibrated QuantizationInteractive blogThe AI AlmanacThe AI Almanac: finding the next paper worth readingPaperBeyond Equivalence: Task-Level Verification Unlocks Architectural Optimizations by LLM AgentsAI for Chip Design NeurIPS Workshop 2026PaperKnowing the Questions before the Exam: Enabling Selective Memory in Linear Attention ModelsSLM-Agents: NeurIPS Workshop 2026PaperPretext: Defeating Malicious Skill Detection Frameworks for AI AgentsAIWILD@NeurIPS 2026PaperRAG-Pull: Turning Retrieval into a Code-Injection Channel via Invisible Unicode PerturbationsAIWILD@NeurIPS 2026PaperRTLScout: Joint Agentic Code and Synthesis Optimization for Efficient Digital CircuitsAI for Chip Design NeurIPS Workshop 2026PaperElasticity in Parallel Sparse Triangular SolveICPP 2026PaperApproaching I/O-optimality for Approximate AttentionNeurIPS 2026PaperDiff3R: Feed-forward 3D Gaussian Splatting with Uncertainty Aware Differentiable OptimizationNeurIPS 2026PaperFast and Stable Triangular Inversion for Delta-Rule Linear TransformersNeurIPS 2026PaperGenRecon: Bridging Generative Priors for Multi-View 3D Scene ReconstructionNeurIPS 2026

From the authors of KVarN

AI Team - Quantization

MACaQ

Model Agnostic Calibrated Quantization

One quantizer.Text, Image, Audio, Video, 3D.

Compatible with
  • llama.cpp
  • vLLM
  • Diffusers
Code on GitHub
Scroll to explore
Back to research

AI Team · Quantization

from the authors ofKVarN

MACaQ: Fast, Model Agnostic Calibrated Quantization for Low‑Precision Weights and Activations

Lorenz K. MullerPhilippe BichChiara BorettiHyun-Min ChangJiawei ZhuangLukas Cavigelli
Huawei Computing Systems Lab (CSL), Switzerland

MACaQ /məˈkɑːk/: Model Agnostic Calibrated Quantization. So easy to use, even a macaque could get the hang of it.

Abstract

MACaQ calibrates 4-bit weights and activations one linear layer at a time. Channel balancing, small Hadamard transforms, and weight refinement improve low-precision representations while keeping every change inside the layer. The same recipe applies across architectures and modalities: text, images, video, audio, and 3D reconstruction.

Runs on: vLLM llama.cpp
01 / PERFORMANCE

MACaQ in numbers.

Measured speed, fidelity, and complete model size.

Runtime
SIDE BY SIDE

Inference comparison

2 methods
Performance comparison
MethodPrecisionvs. BF16

Select a method to explore the comparison.

INFERENCE

Decode speedup

Higheris better
MACaQ2.74×

relative to the original FP16 model

FP16 = 1×
Qwen3.8-27B · llama.cppDecode speedup · FP16 = 1×

02 / EXAMPLES

MACaQ across modalities.

Hugging FaceWan2.2-I2V-A14B
Reference BF16
MACaQ W4A8

Prompta group of jellyfish swimming in an aquarium

03 / INTEGRATIONS

MACaQ, ready to run.

Terminal

vLLM

Model serving

# Supply representative UTF-8 calibration text
.venv-vllm/bin/python vllm/cli.py quantize \
  --model Qwen/Qwen3.8-27B \
  --calibration data/calibration.txt \
  --output models/qwen38-27b-macaq --gpu 0
MACaQ runtimeQwen3.8-27B
Terminal

llama.cpp

Local generation

# First download the original HF checkpoint locally
.venv-llamacpp/bin/python llamacpp/llama.cpp/convert_hf_to_gguf.py \
  models/Qwen3.8-27B --outtype f16 --outfile models/original-f16.gguf

.venv-llamacpp/bin/python llamacpp/macaq.py quantize \
  --model models/original-f16.gguf --hf models/Qwen3.8-27B \
  --calibration data/calibration.txt --format IQ4_XS --had-block 64 \
  --device cuda --gpu 0 --gpu-layers 99 \
  --output models/qwen38-27b-macaq.gguf --work-dir work/qwen38-27b
MACaQ runtimeQwen3.8-27B

Run from the MACaQ repository root. Use representative calibration text and keep evaluation text separate. Use Linux, Python 3.12 and a separate environment for each backend. Set MACAQ_TOOLKIT to your installed CUDA toolkit. Choose fresh output and work paths.

04 / METHOD

MACaQ, inside the layer.

One local recipe. Across architectures and modalities.

THE IDEA BEHIND MODEL AGNOSTIC

Different models.
The same building block.

Text, images, video, audio and 3D models share a building block: the linear layer.

MACaQ keeps its transforms and quantization decisions inside that layer. Nothing needs to be absorbed into a neighboring normalization, projection or residual branch.

Change the representation.
Keep the transformation local.
MACaQ: one fused kernel inside each linear layerThe input passes through diagonal balancing D, Hadamard rotation H32, quantization Q4 and MXFP4 matrix multiplication. All four operations belong to the same fused kernel. Weights are prepared offline and feed the multiplication from above. The layer's input and output stay in the original basis. MACaQ · ONE LINEAR LAYER OFFLINE · ONCE Prepared weights ONE FUSED KERNEL Balance, rotate, quantize and multiply xy D H₃₂ Q₄ MXFP4 multiply BalanceRotateQuantize4-bit matrix multiply
MACaQ · ONE LINEAR LAYER OFFLINE Prepared weights ONE FUSED KERNEL xy D H₃₂ Q₄ MXFP4multiply BalanceRotateQuantize
↳

The layer’s input and output stay in the original basis.
The same local recipe can be reused across the model.

Local by design.
Light at runtime.

Calibration and refinement happen offline. At inference, input scaling and a small Hadamard transform share the fused MXFP4 path.

2.3%
Measured transform overhead

vs. MXFP4 without scaling and rotation

Qwen3-32B · vLLM · batch 1 · context 512 · Paper, Figure 3
MXFP4 · 4-bit weights + 4-bit activations · 32 values per block

BEYOND THE DEMO

Want to know more?
Read the paper.

Explore the layer-local design, Flip-Refinement and the experiments behind MACaQ. The full method, analysis and results are in the paper.

Read the paper

Open source. Yours to explore.

MACaQ is open source under the Apache 2.0 license.

Explore GitHub
First page of MACaQ: Fast, Model Agnostic Calibrated Quantization for Low-Precision Weights and Activations MACaQ · 16 pagesOpen paper

FLUX.2 [KLEIN] 4B · IMAGE COMPARISON

Quantization details.

ReferenceBF16
RTNMXFP4 · W4A4
MACaQMXFP4 · W4A4

All three versions use FLUX.2 [klein] 4B, with the same prompt, random seed and generation settings for each scene.

Same layers. Same 4-bit format.

RTN and MACaQ quantize the same 100 attention and MLP linear layers in the diffusion transformer. Weights and dynamically quantized activations use MXFP4 E2M1, with shared E8M0 scales in blocks of 32. The text encoder, VAE, embeddings, modulation, output head and normalization retain their original precision.

How the weights are prepared

RTN rounds directly to MXFP4 with scale-exponent offset 0, without balancing, rotation or refinement.

MACaQ uses channel balancing (α = 0.75), a 32-point Hadamard transform, scale selection with exponent offsets (+1, 0, −1), ridge = 1.0 and 512 bit-flip refinement iterations. Both methods use the same native W4A4 matrix-multiplication kernel.

Generation settings

1344 × 768 pixels, six denoising steps and guidance scale 1.0. The images are served as lossless WebP, preserving every generated pixel. The four selected scenes use seeds 2030543 (Light and stone), 50232 (Evening in Provence), 2028512 (The quiet lake) and 2026074 (Sapphire wings).

WAN2.2 · VIDEO COMPARISON

Quantization details.

Comparison setup

Wan2.2-I2V-A14B: a BF16 reference and MACaQ W4A8, generated from the same input image, prompt and seed. These are image-to-video outputs.

Precision and layer coverage

Weights use MXFP4 E2M1 with E8M0 scales in blocks of 32; activations use per-token FP8 E4M3. All 800 linear layers inside the two diffusion-transformer experts are quantized. The text encoder, VAE, embeddings, output projection and normalization retain their original precision.

Calibration and inference

MACaQ uses four VBench-I2V calibration clips, separate from the evaluation images, with balance α = 0.75, block-Hadamard size 32, scale-exponent offsets {+1, 0, −1} and 512 refinement iterations. The W4A8 path converts the packed MXFP4 weights exactly to FP8 for the matrix multiplication.

Generation settings

832 × 480 pixels, 61 frames at 24 fps, 40 denoising steps and guidance scale 3.5. Each scene lasts 2.54 seconds. The Home demo plays three scenes in sequence; the Examples show each scene separately. Both versions retain the original generated frames and playback speed.

QWEN3-TTS · AUDIO EXPERIMENT

Quantization details.

ReferenceBF16
RTNMXFP4 · W4A4
MACaQMXFP4 · W4A4

All three versions use Qwen3-TTS-12Hz-1.7B-CustomVoice, with the same text, stock voice, voice direction and random seed.

Same layers. Same 4-bit format.

RTN and MACaQ quantize the same 231 attention and MLP linear layers in the talker and code predictor: about 1.49 billion weights. Weights and activations use OCP MXFP4, with 32 values per block and shared E8M0 scales. Embeddings, text projections, normalization, output heads and the speech codec remain in BF16.

How the weights are prepared

RTN rounds directly to the nearest MXFP4 code, without balancing, rotation, scale screening or refinement.

MACaQ calibrates on eight separate prompts. It uses channel balancing (α = 0.75), a 32-point Hadamard transform, scale selection with exponent offsets (+1, 0, −1), ridge = 1.0, and up to 512 bit-flip refinement iterations. Both methods run through the same native W4A4 matrix-multiplication kernel.

Generation and audio settings

Temperature 0.9, top-k 50, top-p 1.0, repetition penalty 1.05, and a maximum of 256 audio codes. The code predictor also uses temperature 0.9, top-k 50 and top-p 1.0. No selected sample reaches the generation limit.

Outputs are mono 24 kHz, 16-bit PCM WAV. No denoising, loudness matching, artificial degradation or playback-speed changes are applied. Quantization can change timing and intonation even with the same seed. These comparisons assess the audio outputs; generation times on the shared GPU are not a speed benchmark.

MAPANYTHING APACHE · 3D RECONSTRUCTION

Quantization details.

ReferenceBF16 inference
MACaQMXFP4 · W4A4

Both versions use MapAnything Apache (1.23 billion parameters), with the same input images, preprocessing and mesh filtering. Every scene shown here was reconstructed separately with the original and quantized models.

What is quantized?

MACaQ quantizes 160 attention and MLP linear layers in the DINOv2 image encoder and multiview transformer: 1.13 billion weights, or 92.18% of the model’s parameters. Weights and activations use OCP MXFP4, with 32 values per block and shared E8M0 scales, executed by the native W4A4 kernel.

Patch and geometric input embeddings, normalization, layer scales, and DPT, pose and scale prediction heads retain the original FP32 storage under BF16 autocast. The reference also uses the upstream BF16 autocast inference policy.

Calibration and refinement

Calibration uses eight separate scenes and 14 images, with 2,048 activation samples per layer at the standard 518 resolution setting. None of the four displayed scenes is used for calibration.

The recipe is channel balancing (α = 0.75), a 32-point Hadamard transform, scale selection with exponent offsets (+1, 0, −1), ridge = 1.0, and up to 512 bit-flip refinement iterations.

Reconstruction and verification

These are image-only reconstructions, without supplied depth, camera poses or calibration. Both variants use identical masking and mesh export settings. Home shows the same recorded camera movement around the two studio meshes; Examples lets you rotate and zoom the real reconstructions.

All four matrix shapes passed native-kernel checks. Reloading the saved checkpoint reproduced the same geometry, depth, camera and mask outputs for all four scenes. These visual comparisons are not a reconstruction-accuracy or speed benchmark.
THE LOCAL CHANGE OF BASIS

A matching pair of transforms.

D is a positive diagonal scale. B is an orthogonal block-Hadamard transform, with 32 channels per block.

Inputs, onlineX′ = XDB
Weights, offlineW′ = WD⁻¹B
X′W′ᵀ = XDBBᵀD⁻¹Wᵀ = XWᵀ

The paired transforms preserve the linear operation before quantization. The output requires no extra inverse rotation.

At inference, transformed inputs and prepared weights are quantized to MXFP4. Calibration guides the block-scale search and Flip-Refinement to reduce the layer’s reconstruction error.

MACaQ paper · Method, Equations 3–7. The supplied experiments cover language models and image diffusion.
INTEGRATION PREVIEW

Check the linked resource for availability and updates.