Faster BM25 Queries

This feature is available in versions 0.26.0 and above.

Top K queries ordered by BM25 score can be sped up by setting pnorms=true when creating your index, at the cost of a larger index and longer builds.

Enable the Optimization

Set pnorms=true on the tokenizer cast for each text field used in BM25 scoring. The option defaults to false. This example enables it for description:

CREATE INDEX search_idx ON mock_items
USING paradedb (id, (description::pdb.unicode_words('pnorms=true')), category)
WITH (key_field = 'id');

Queries automatically use the setting from the index:

SELECT id, description, pdb.score(id)
FROM mock_items
WHERE description ||| 'shoes'
ORDER BY pdb.score(id) DESC
LIMIT 5;

This option is only supported for text fields. Existing indexes must be rebuilt with pnorms=true to enable it.

Performance Tradeoff

BM25 scoring uses the length of each text field to help determine relevance. Setting pnorms=true stores an extra, denser copy of this information where the scoring process can read it more efficiently. Depending on the data and query, this can make queries several times faster without changing scores or relevance ranking. For each field with pnorms enabled, the extra storage is roughly one byte per distinct term in each document, plus metadata. The extra data also increases the work required to build and merge index segments. Enable pnorms on fields where faster BM25 top K queries justify those costs.

Indexing

Full-Text / Lexical Search

Vector / Semantic Search

Hybrid Search

Aggregations

JOINs

Sharding

Cheatsheets

Faster BM25 Queries

Enable the Optimization

Performance Tradeoff

​Enable the Optimization

​Performance Tradeoff

Enable the Optimization

Performance Tradeoff