> ## Documentation Index
> Fetch the complete documentation index at: https://www.paradedb.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Unicode

> The default text tokenizer in ParadeDB

The `unicode_words` tokenizer splits text according to word boundaries defined by the [Unicode Standard Annex #29](https://www.unicode.org/reports/tr29/)
rules. All characters are [lowercased](https://www.paradedb.com/docs/reference/token-filters/lowercase.md) by default.

This tokenizer is the default text tokenizer. If no tokenizer is specified for a text field, the `unicode_words` tokenizer will be used.

```sql theme={null}
-- The following two configurations are equivalent
CREATE INDEX search_idx ON mock_items
USING paradedb (id, description);
```

Or specify the tokenizer explicitly:

```sql theme={null}
CREATE INDEX search_idx ON mock_items
USING paradedb (id, (description::pdb.unicode_words));
```

To get a feel for this tokenizer, run the following command and replace the text with your own:

```sql theme={null}
SELECT 'Tokenize me!'::pdb.unicode_words::text[];
```

```ini Expected Response theme={null}
     text
---------------
 {tokenize,me}
(1 row)
```

## Remove Emojis

By default, emojis in the source text are preserved. To remove emojis, set `remove_emojis` to `true`.

```sql theme={null}
SELECT 'Tokenize me! 😊'::pdb.unicode_words('remove_emojis=true')::text[];
```

```ini Expected Response theme={null}
     text
---------------
 {tokenize,me}
(1 row)
```


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.