Constructive Integer Attention

A hardware-optimized attention mechanism that replaces floating-point operations with integer-only arithmetic to slash latency and power consumption.

Constructive Integer Attention (CIA) eliminates the need for high-precision floating-point units by performing the entire attention calculation using fixed-point integers. By leveraging bit-shifting and integer-based scaling, the system maintains 99% of FP16 accuracy while achieving a 4x speedup on commodity hardware (specifically ARM and RISC-V processors). This approach is critical for edge deployment where memory bandwidth and thermal limits are tight: it allows 7B-parameter models to run locally on mobile chipsets without the thermal throttling typical of standard Transformer architectures.

https://arxiv.org/abs/2312.04547