entropix

Entropix is a novel, open-source sampling algorithm that uses entropy and varentropy to enhance Large Language Model (LLM) reasoning and drastically reduce hallucinations.

This technology is an advanced LLM sampling method designed to improve reasoning without retraining or fine-tuning the base model. Entropix leverages two key metrics—entropy and varentropy—to gauge a model's uncertainty at the token level. By dynamically adjusting the sampling strategy (like temperature, top-k, and top-p) based on these uncertainty signals, it encourages deeper, context-aware reasoning. This approach allows smaller models (e.g., 1B parameter count) to exhibit complex problem-solving capabilities, simulating Chain-of-Thought (CoT) or o1-like reasoning with only inference-time compute. The project is open-source, with a primary repository on GitHub for both large-scale and local research applications.

https://github.com/xjdr-alt/entropix