> ## Documentation Index
> Fetch the complete documentation index at: https://www.paradedb.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Lindera

> Uses prebuilt dictionaries to tokenize Chinese, Japanese, and Korean text

The Lindera tokenizer is a more advanced CJK tokenizer that uses prebuilt Chinese, Japanese, or Korean dictionaries to break text into meaningful tokens (words or phrases) rather than on individual characters.
Chinese Lindera uses the CC-CEDICT dictionary, Korean Lindera uses the KoDic dictionary, and Japanese Lindera uses the IPADIC dictionary.

By default, non-CJK text is lowercased, and punctuation is not ignored.
As of version 0.22.4, whitespace is removed by default. On earlier versions it is preserved.

<CodeGroup>
  ```sql Chinese Lindera theme={null}
  CREATE INDEX search_idx ON mock_items
  USING paradedb (id, (description::pdb.lindera(chinese)))
  WITH (key_field='id');
  ```

  ```sql Korean Lindera theme={null}
  CREATE INDEX search_idx ON mock_items
  USING paradedb (id, (description::pdb.lindera(korean)))
  WITH (key_field='id');
  ```

  ```sql Japanese Lindera theme={null}
  CREATE INDEX search_idx ON mock_items
  USING paradedb (id, (description::pdb.lindera(japanese)))
  WITH (key_field='id');
  ```
</CodeGroup>

To get a feel for this tokenizer, run the following command and replace the text with your own:

```sql theme={null}
SELECT 'Hello world! 你好!'::pdb.lindera(chinese)::text[];
```

```ini Expected Response theme={null}
              text
------------------------
 {hello,world,!,你好,!}
(1 row)
```

## Keep Whitespace

By default, whitespace is not tokenized. To include it, set `keep_whitespace` to `true`.

```sql theme={null}
SELECT 'Hello world! 你好!'::pdb.lindera(chinese, 'keep_whitespace=true')::text[];
```

```ini Expected Response theme={null}
              text
--------------------------------
 {hello," ",world,!," ",你好,!}
(1 row)
```

## NFKC Normalization

Set `nfkc` to `true` to apply [Unicode NFKC normalization](https://unicode.org/reports/tr15/) to the text before it is segmented. This collapses compatibility characters such as full-width Latin letters and digits to their canonical half-width forms, so that, for example, `ＡＢＣ` and `ABC` produce the same tokens.

```sql theme={null}
SELECT 'ＡＢＣ１２３'::pdb.lindera(japanese, 'nfkc=true', 'lowercase=false')::text[];
```

```ini Expected Response theme={null}
   text
----------
 {ABC,123}
(1 row)
```

## Reading Form

Set `reading_form` to `true` to replace each token with its dictionary reading form after segmentation. For Japanese (IPADIC), the surface form is replaced with its katakana reading; for Korean (KoDic), Hanja are replaced with their Hangul reading. This is supported for Japanese and Korean only.

```sql theme={null}
SELECT '日本語'::pdb.lindera(japanese, 'reading_form=true', 'lowercase=false')::text[];
```

```ini Expected Response theme={null}
    text
------------
 {ニホンゴ}
(1 row)
```