Literal

The literal tokenizer is not ideal for text search queries like match or phrase. If you need to do text search over a field that is literal tokenized, consider using multiple tokenizers.

Because the literal tokenizer preserves the source text exactly, token filters cannot be configured for this tokenizer.

The literal tokenizer applies no tokenization to the text, preserving it as-is. It is the default for uuid fields (since exact UUID matching is a common use case), and is useful for doing exact string matching over text fields. Text fields used as a sort field in a Top K query, or as part of an aggregate, must use either this tokenizer or literal_normalized.

CREATE INDEX search_idx ON mock_items
USING paradedb (id, (description::pdb.literal))
WITH (key_field='id');

To get a feel for this tokenizer, run the following command and replace the text with your own:

SELECT 'Tokenize me!'::pdb.literal::text[];

Expected Response

       text
------------------
 {"Tokenize me!"}
(1 row)

Indexing

Full-Text / Lexical Search

Vector / Semantic Search

Hybrid Search

Aggregations

JOINs

Cheatsheets