Hardware-Aware Additive Quantization for LLM Inference on AMD Zen 5

Summary

Many quantization methods prioritize reconstruction quality and adapt the resulting representation to available hardware. This proposal instead begins with a specific hardware primitive and derives the quantization format and kernel around it.

Permaq is my proposed additive quantization format built around AVX-512 VBMI and its VPERMI2B instruction, which treats two 64-byte registers as a 128-entry byte table addressed by 7-bit indices. Permaq uses that 128-entry boundary directly. In its initial configuration, each group of eight weights is represented by two 7-bit indices, each selecting an 8-dimensional codeword from one of two learned 128-entry codebooks. Adding the two selected codewords approximates the original weight group. This requires fourteen index bits per eight weights, or 1.75 bits per weight before codebooks and other model overhead.

For each activation group x, the kernel computes the dot product of x with every codeword in both codebooks, producing two 128-entry projected tables whose values are quantized to a signed byte representation suitable for register lookup. If the projected tables contributing to an output accumulation can use a common scale, encoded weights can then be evaluated by unpacking their indices, retrieving the corresponding projected values, and accumulating them in integer form. The proposal tests whether codebooks, packing, data layout, numerical representation, and kernel execution designed together around this SIMD primitive can produce a better quality-throughput tradeoff than an additive format designed primarily for reconstruction accuracy.

Design

Low-batch CPU inference is often constrained by memory bandwidth, so reducing weight traffic can justify additional/more difficult computation during decoding. AQLM and QuIP# have demonstrated the viability of additive and structured quantization at very low bit rates. Permaq focuses on how such a representation maps onto the processor by fixing each codebook at 128 entries, the largest byte table directly addressable by the intended VPERMI2B lookup. A 256-entry design using the same register-resident lookup strategy would require additional permutation and selection instructions. Permaq therefore gives up some representational capacity in exchange for a simpler lookup path, with its value depending on whether that execution advantage exceeds the associated quality and decoding costs. The choice of target is quite important here, because Zen 5 carries two 512-bit byte-granular shuffle units and sustains even the most expensive of those shuffles at roughly two per cycle, which is the throughput the entire proposed design depends on.

The basic computation can be expressed as pseudocode:

derive shared_scale from calibrated activation statistics

for each activation group g:
    for k in 0..127:
        T1[g][k] = quantize_int8(dot(C1[k], x[g]), shared_scale)
        T2[g][k] = quantize_int8(dot(C2[k], x[g]), shared_scale)

for each output row r:
    acc = 0                                    # int32
    for each activation group g:
        i1, i2 = unpack_7bit_codes(encoded[r][g])
        acc += sign_extend(T1[g][i1]) + sign_extend(T2[g][i2])
    output[r] = row_scale[r] * shared_scale * acc

Each projected table contains 128 byte values, so a single table occupies two 512-bit registers and both tables together occupy four. The lookup uses VPERMI2B rather than VPERMT2B because VPERMI2B overwrites the register holding the indices and leaves both table registers intact, which is exactly what reuse of a table across many output rows requires. Table construction occurs once per activation group, so the layout must reuse each table across many output rows. That reuse requirement also constrains threading, because partitioning output rows across threads would force every thread to rebuild the same tables while weight traffic stays fixed, making table construction grow with core count. Partitioning instead along the input dimension, with each thread building tables only for its own activation groups and contributing partial sums to a final reduction, keeps total table construction constant. Packing two indices into fourteen bits reduces the nominal index payload relative to two byte-aligned codes but requires unpacking before lookup, and retrieved signed byte values must be extended to a wider integer type for accumulation. Summing signed bytes is not a single cheap operation, so accumulation needs explicit widening or a multiply-add against a vector of ones, with periodic promotion to thirty-two bit accumulators to avoid overflow across long input dimensions. Tiling, index packing, table reuse, and vectorization therefore need to be optimized and measured together.

Quality and Numerical Constraints

Restricting each codebook to 128 entries reduces its representational capacity. AQLM suggests that larger effective state spaces can improve approximation quality at a fixed index budget, so Permaq may incur greater reconstruction error. Training the codebooks under the final codebook size, group size, numerical range, and execution constraints may recover enough of that loss to preserve a useful quality-throughput tradeoff. The use of two 7-bit codes over eight weights is only the initial configuration, and other group sizes or codebook counts can retain the same 128-entry lookup structure if they provide a better balance of reconstruction quality, index payload, table construction, and throughput.

Projected-table scaling introduces a second constraint. Different activation groups can produce substantially different dynamic ranges, and rescaling individual groups inside the accumulation loop would add cost to the critical path. Randomized Hadamard preprocessing, as used in QuIP#, may suppress outliers enough for the projected tables contributing to an output accumulation to use a common compact integer scale. Wider integer entries, limited outlier handling, grouped accumulation with a small number of scales, or a training penalty on projected-value range provide fallback approaches if a single byte-valued scale cannot preserve sufficient accuracy.

Prior Research

AQLM and QuIP# establish the viability of extreme additive and structured quantization. T-MAC explores lookup-based low-bit CPU inference, CodeGEMM and EVA precompute activation-codeword products for indexed retrieval, and Quicker ADC demonstrates SIMD register lookup for sub-byte codes on AVX-512. GPTVQ applies the same hardware-first reasoning on a different instruction set, capping index width at five bits because wider indices would double the number of ARM TBL instructions required. The precomputation of activation-codeword products is therefore not novel here. Permaq combines these lines of work in a specific Zen 5 design whose contribution is the constraint that those precomputed tables stay register-resident within a single instruction's addressing limit, with 128-entry additive codebooks, 7-bit indices, packing, projected numerical representation, and execution kernel all derived from the VPERMI2B lookup structure.

Evaluation

The primary comparison should probably be a byte-aligned additive baseline at a similar effective bit rate on the same processor. Measurements must account for total model memory rather than packed indices alone, including codebooks, scales, metadata, and components stored at other precisions. Packing two 7-bit codes reduces the nominal index payload by 12.5 percent relative to two byte-aligned codes, but the actual memory-traffic reduction will depend on packing, alignment, cache behavior, and vectorized loading. Attention, KV-cache traffic, normalization, sampling, and other model operations remain unchanged. Any throughput advantage must therefore exceed the combined costs of index unpacking, projected-table construction, lookup, sign extension, and accumulation.

Table construction introduces fixed work for every new activation vector, making large matrix-vector products more favorable than narrow projections or routing layers. The initial design is aimed at low-batch decoding, where projected tables can be reused across many output rows and reduced weight traffic matters most. Batched inference is likely to provide a weaker case because each activation vector requires its own projected tables.

Evaluation would measure VPERMI2B lookup cost, 7-bit index handling, projected-table dynamic range, and quantization error with and without Hadamard preprocessing. Kernel-level benchmarks would compare unpacking, lookup, sign extension, and accumulation against byte-aligned storage using the same 128-entry codebooks. A second baseline using larger conventional codebooks would isolate the cost of restricting the representation to the VPERMI2B lookup boundary.

End-to-end evaluation would ideally compare model quality, token-generation throughput, and total memory footprint across processor and thread configurations. Each result would identify the exact processor and thread configuration, since AVX-512 VBMI implementations differ in vector width and shuffle throughput, and reduced-width Zen 5 variants do not carry the same SIMD execution resources as the full core. Expectations should stay modest, because GPTVQ's ARM lookup kernel delivered large footprint reductions but only about two to thirteen percent lower decode latency than a 4-bit integer baseline, which is a reasonable prior for how much of a memory-traffic advantage survives into wall-clock throughput.

Potential Impact

CPU systems can often be provisioned with more memory at a lower cost per gigabyte than GPU systems, and if Permaq improves low-bit CPU inference enough, it could make better use of that existing cost advantage for private and offline deployment. A faster CPU kernel could make larger models practical on hardware that individuals and organizations already own.

Beyond the specific Zen 5 kernel, Permaq provides a test of hardware-first quantization design and may clarify when representational constraints imposed by an instruction set are offset by simpler and faster execution.

Additional Comments

This proposal is intentionally brief and does not attempt to specify every low-level implementation detail. It also does not assume that low-bit CPU inference will become the dominant approach to local LLM deployment. Permaq is simply a proposed focused research and engineering experiment that could inform future quantization methods, CPU inference kernels, and hardware-aware AI software. If you have any thoughts or suggestions you'd like to share, feel free to drop me an email at [email protected].