> ## Documentation Index
> Fetch the complete documentation index at: https://www.paradedb.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Architecture
> A deep dive into how ParadeDB is built on Postgres
ParadeDB introduces modern query execution paths and data structures, optimized for high-ingest search and analytics workloads, to Postgres.
## Custom Index
In Postgres, indexes provide alternative data structures for accessing the data in a table (which Postgres calls a "heap table") more efficiently.
ParadeDB introduces a custom index called the *ParadeDB index*.
When a table row is inserted or updated, the ParadeDB index is immediately notified. These changes are recorded as part of the current transaction, ensuring that index updates are real-time.
## Data Model
The ParadeDB index is laid out as an [LSM tree](#lsm-tree), where each segment in the tree consists of both an inverted index and columnar index.
The inverted and columnar indexes optimize for fast reads, while the LSM tree optimizes for high-frequency writes.
### Inverted Index
An inverted index is a structure that maps each term (i.e., tokenized word) to a list of documents that contain that term (called a "postings list") along with metadata like term frequency and document frequency. This structure allows ParadeDB to efficiently retrieve all documents matching a particular search term or phrase without scanning the entire table.
### Columnar Index
Alongside the inverted index, ParadeDB also maintains a structure that stores fields in a column-oriented format. Columnar formats are standard
for analytical (i.e. OLAP) databases because they store values contiguously and enable efficient scans over large datasets compared to Postgres'
row-oriented layout. All text fields which use the [literal](/docs/documentation/tokenizers/available-tokenizers/literal) or [literal normalized](/docs/documentation/tokenizers/available-tokenizers/literal-normalized) tokenizer, or are non-text,
are stored in the columnstore.
In Tantivy these structures are referred to as [fast fields](https://docs.rs/tantivy/latest/tantivy/fastfield/index.html), but they are largely transparent in ParadeDB.
### LSM Tree
To support real-time updates, the ParadeDB index uses a [Log-Structured Merge (LSM) tree](https://en.wikipedia.org/wiki/Log-structured_merge-tree).
An LSM tree is a write-optimized data structure commonly used in systems like RocksDB and Cassandra. The core idea behind an LSM tree is to turn random writes into sequential ones. Incoming writes are appended to a mutable segment, which buffers rows across statements. Once the segment reaches the configured [`mutable_segment_rows`](/docs/documentation/performance-tuning/writes#increase-mutable-segment-size) threshold—1,000 rows by default—it is frozen and becomes eligible for conversion into an immutable segment.
These segment files are organized by size into layers or levels. Newer data is written to the topmost layer. Over time, data is gradually pushed down into lower levels through a process called merging or compaction, where data from smaller segments is merged, deduplicated, and rewritten into larger segments.
ParadeDB reuses the current mutable segment across `INSERT`, `UPDATE`, and `COPY` statements until it reaches that threshold. Each resulting immutable segment has its own inverted index and columnar index, which means that the ParadeDB index
is actually a collection of many inverted/columnar indexes, each of which allows for very dense intersection queries to rapidly filter matches.
## Query Execution
### Custom Operators
ParadeDB introduces several new text search operators to Postgres. For example, `|||` is used for [match disjunction](/docs/documentation/full-text/match) queries, whereas `###`
is for [phrase](/docs/documentation/full-text/phrase) queries.
```sql theme={null}
SELECT * FROM mock_items
WHERE description ||| 'running shoes';
```
ParadeDB’s custom query execution paths are only triggered when at least one of ParadeDB's operators is present in the query. Otherwise, it is executed entirely by native Postgres.
### Custom Scan
When a supported query uses a ParadeDB operator and a matching ParadeDB index is present, ParadeDB can execute it using a [custom scan](https://www.postgresql.org/docs/current/custom-scan.html).
Custom scans are execution nodes set aside by Postgres that allow extensions to run custom logic during a query. They are more powerful and versatile than typical Postgres index scans because they
allow the extension to "take over" large parts of the query, including aggregates, `WHERE`, and even [`GROUP BY` clauses](/docs/welcome/roadmap#analytics).
From a performance perspective, custom scans significantly speed up queries by pushing down filters, aggregates, and other operations directly into the index, rather than applying them afterward in separate phases.
To understand what kind of scan is used, run `EXPLAIN`:
```sql theme={null}
-- Native Postgres scan, no ParadeDB operator
EXPLAIN SELECT * FROM mock_items
WHERE description = 'running shoes' AND rating <= 5;
-- Custom scan, ParadeDB operator used
EXPLAIN SELECT * FROM mock_items
WHERE description ||| 'running shoes' AND rating <= 5;
```
As a rule of thumb: if `EXPLAIN` shows a custom scan (or, in rare cases, a ParadeDB index scan), then that part of query is going through ParadeDB. Otherwise, the query passes through standard Postgres.
### Parallelization
For queries that need to read large amounts of data, such as [Top K](/docs/documentation/sorting/topk) or aggregate queries, ParadeDB custom scans can use additional Postgres workers to execute the query
in parallel. To see if a query was parallelized, run `EXPLAIN ANALYZE`:
```sql theme={null}
-- Top K queries may be parallelized
EXPLAIN ANALYZE SELECT * FROM mock_items
WHERE description ||| 'running shoes'
ORDER BY rating LIMIT 5;
```
Parallelization also depends on the [number of available
workers](/docs/documentation/performance-tuning/reads).
Postgres supports [parallel scans, joins, and two-stage aggregation](https://www.postgresql.org/docs/current/parallel-plans.html). ParadeDB builds on those parallel-worker primitives while pushing supported filters, Top K, joins, and aggregates into the index. For joins and aggregates, ParadeDB can distribute the pushed-down plan across workers using an MPP execution strategy.
## Design Philosophy
* **Keep it Boring**. Use robust extension points in Postgres vs. hacking around the internals. Adopt battle-tested tools, like industry standard file formats and query engine libraries, instead of cutting-edge but less-proven alternatives.
* **Behave Exactly Like Postgres**. This extends from user-facing aspects, like the SQL query syntax and ORM compatibility, all the way down to low-level integrations with Postgres' storage system and query planner.
* **Works Out of the Box**. Users should be able to get satisfying search results and performance with minimal tuning or configuration.
## Dependencies
The three main dependencies of `pg_search` are:
* [`pgrx`](https://github.com/pgcentralfoundation/pgrx/tree/develop) — the library for writing Postgres extensions in Rust
* [Tantivy](https://github.com/quickwit-oss/tantivy) — a Rust-based full-text search library inspired by [Lucene](https://github.com/apache/lucene)
* [Apache DataFusion](https://github.com/apache/datafusion) — an extensible query execution framework for OLAP processing