CIA (Constructive Integer Attention)

CIA replaces standard floating-point attention with integer-only arithmetic to slash memory overhead and latency in LLM inference.

Constructive Integer Attention (CIA) eliminates the precision bottleneck in Transformer models by mapping high-dynamic-range attention scores to 8-bit or 4-bit integers. By utilizing constructive quantization techniques, CIA maintains model accuracy (within 0.1% of FP16 baselines) while enabling hardware-level acceleration on commodity GPUs and edge devices. This approach targets the memory wall, reducing KV cache requirements by up to 50% and accelerating throughput for long-context sequences.

https://arxiv.org/abs/2312.09252