Inference engine

The core AI component: it applies rules to a knowledge base (forward/backward chaining) or executes trained machine learning models (e.g., Llama 3) to generate predictions.

An Inference Engine is the operational core of an intelligent system, tasked with translating knowledge into actionable output. Historically, it was the 'brain' of expert systems, using logic to apply rules to a knowledge base and deduce new facts, crucial for early medical diagnostic tools. Today, the term primarily refers to specialized software and hardware (like NVIDIA TensorRT or Intel OpenVINO) optimized for high-speed model execution. Its mandate is clear: minimize latency and maximize throughput for real-time applications, such as autonomous vehicle object detection or high-volume LLM serving. This performance is achieved through rigorous optimizations, including model quantization and layer fusion.

https://en.wikipedia.org/wiki/Inference_engine