Terrier

Terrier is an open-source search engine framework designed for rapid development of large-scale information retrieval applications.

Developed at the University of Glasgow, Terrier (Terabyte Retriever) handles high-volume indexing and retrieval for datasets like the 25-billion-page ClueWeb12. It provides researchers and engineers with a modular Java library to implement state-of-the-art ranking models (BM25, PL2, DFR) and query expansion techniques. The framework supports diverse data formats (HTML, PDF, WARC) and integrates directly with Hadoop for distributed processing. Its lean architecture makes it a standard choice for TREC evaluations and enterprise-grade search experimentation.

https://terrier.org/