BM25

BM25 (Best Matching 25) is the core keyword-based ranking function used in information retrieval to score a document's relevance to a query, effectively replacing traditional TF-IDF.

BM25 is a robust, decades-old ranking function, developed for the Okapi information retrieval system in the 1980s and 1990s, that remains foundational for modern search engines (e.g., Elasticsearch, Lucene). It calculates a relevance score by combining three factors: Term Frequency (TF), Inverse Document Frequency (IDF), and Document Length Normalization. Critically, it improves upon TF-IDF by incorporating a saturation mechanism for TF, meaning additional term occurrences in a single document contribute diminishing returns to the score. Furthermore, it normalizes scores by document length using the parameter 'b' to prevent overly long documents from unfairly dominating rankings, ensuring accurate, focused results for keyword-based queries.

https://en.wikipedia.org/wiki/Okapi_BM25