Unicode

The unicode_words tokenizer splits text according to word boundaries defined by the Unicode Standard Annex #29 rules. All characters are lowercased by default. This tokenizer is the default text tokenizer. If no tokenizer is specified for a text field, the unicode_words tokenizer will be used.

-- The following two configurations are equivalent
CREATE INDEX search_idx ON mock_items
USING paradedb (id, description);

CREATE INDEX search_idx ON mock_items
USING paradedb (id, (description::pdb.unicode_words));

To get a feel for this tokenizer, run the following command and replace the text with your own:

SELECT 'Tokenize me!'::pdb.unicode_words::text[];

Expected Response

     text
---------------
 {tokenize,me}
(1 row)

Remove Emojis

By default, emojis in the source text are preserved. To remove emojis, set remove_emojis to true.

SELECT 'Tokenize me! 😊'::pdb.unicode_words('remove_emojis=true')::text[];

Expected Response

     text
---------------
 {tokenize,me}
(1 row)

Indexing

Full-Text / Lexical Search

Vector / Semantic Search

Hybrid Search

Aggregations

JOINs

Sharding

Cheatsheets

Remove Emojis

​Remove Emojis

Remove Emojis