Regex Patterns

The regex_pattern tokenizer tokenizes text using a regular expression. The regular expression can be specified with the pattern parameter. For instance, the following tokenizer creates tokens only for words starting with the letter h:

CREATE INDEX search_idx ON mock_items
USING paradedb (id, (description::pdb.regex_pattern('(?i)\bh\w*')))
WITH (key_field='id');

The regex tokenizer uses the Rust regex crate, which supports all regex constructs with the following exceptions:

Look-around assertions such as (?=...)
Backreferences such as \1

Unlike regex queries, which match against the index’s term dictionary, the tokenizer runs over the source text directly, so constructs like word boundaries (\b) and lazy quantifiers (+?) are supported, as the examples on this page show. To get a feel for this tokenizer, run the following command and replace the text with your own:

SELECT 'Hello world!'::pdb.regex_pattern('(?i)\bh\w*')::text[];

Expected Response

  text
---------
 {hello}
(1 row)

Indexing

Full-Text / Lexical Search

Vector / Semantic Search

Hybrid Search

Aggregations

JOINs

Cheatsheets