Orca

Orca is a Microsoft Research method to train small language models (SLMs) to mimic the complex reasoning of larger models like GPT-4, achieving comparable performance with greater efficiency.

Orca is a cutting-edge research project from Microsoft focused on building highly capable, smaller Large Language Models (LLMs). The core technique, demonstrated in Orca 2 (7B and 13B parameters), uses a ‘teacher-student’ approach: a larger model (like GPT-4) generates tailored, high-quality synthetic data, including complex explanation traces and diverse reasoning strategies (e.g., step-by-step processing). This unique fine-tuning process enables the smaller model, often based on Llama 2, to learn strategic problem-solving and attain performance levels similar to or better than models five to ten times its size on complex zero-shot reasoning benchmarks. The goal is clear: develop smaller, more efficient AI models that retain high-level reasoning capabilities for scalable generative AI applications.

https://www.microsoft.com/en-us/research/project/orca/