> ## Documentation Index
> Fetch the complete documentation index at: https://www.paradedb.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Architecture
> A deep dive into how ParadeDB is built on Postgres
export const ArchitectureDiagram = () => {
const IndexMark = ({x, y}) => ;
const Arrow = ({x, y, direction = "down"}) => ;
const Segment = ({number, compact = false}) =>
{compact ?
Segment
{number}
:
Segment {number}
}
;
const DataModelSvg = ({compact = false}) => ;
const DataModelDiagram = () => ;
return ;
};
Search belongs with your application data.
ParadeDB brings full-text search, vector search, and columnar execution into
Postgres. It does this with modern query execution paths and data structures
optimized for high-ingest search and analytics workloads.
## Custom Index
In Postgres, indexes provide alternative data structures for accessing the data in a table (which Postgres calls a "heap table") more efficiently.
ParadeDB introduces a custom index called the *ParadeDB index*.
When a table row is inserted or updated, the ParadeDB index is immediately notified. These changes are recorded as part of the current transaction, ensuring that index updates are real-time.
For a focused explanation of the index itself, see [How the ParadeDB Index
Works](/docs/concepts/how-the-paradedb-index-works).
## Data Model
The ParadeDB index is laid out as an [LSM tree](#lsm-tree), where each segment contains the structures needed by the indexed fields: an inverted index for tokenized text, columnar storage for scalar fields, and vector index structures for vector fields.
The inverted, columnar, and vector structures optimize for fast reads, while the LSM tree optimizes for high-frequency writes.
### Inverted Index
An inverted index is a structure that maps each term (i.e., tokenized word) to a list of documents that contain that term (called a "postings list") along with metadata like term frequency and document frequency. This structure allows ParadeDB to efficiently retrieve all documents matching a particular search term or phrase without scanning the entire table.
### Columnar Index
Alongside the inverted index, ParadeDB also maintains a structure that stores fields in a column-oriented format. Columnar formats are standard
for analytical (i.e. OLAP) databases because they store values contiguously and enable efficient scans over large datasets compared to Postgres'
row-oriented layout. Non-text, non-JSON fields are stored in columnar storage by default. Text and JSON fields can opt into columnar storage through tokenizer configuration, and the [literal](/docs/reference/tokenizers/available-tokenizers/literal) and [literal normalized](/docs/reference/tokenizers/available-tokenizers/literal-normalized) tokenizers are columnar by default.
### Vector Index
When a `vector` column is indexed with a vector operator class, ParadeDB stores it in ParadeDB-built vector index structures alongside text and columnar fields. This lets vector retrieval compose with full-text predicates, filters, ranking, and aggregations inside the same ParadeDB index. See [Indexing Vectors](/docs/reference/indexing/indexing-vectors) for setup details.
### LSM Tree
To support real-time updates, the ParadeDB index uses a [Log-Structured Merge (LSM) tree](https://en.wikipedia.org/wiki/Log-structured_merge-tree).
An LSM tree is a write-optimized data structure commonly used in systems like RocksDB and Cassandra. The core idea behind an LSM tree is to turn random writes into sequential ones. Incoming writes are appended to a mutable segment, which buffers rows across statements. Once the segment reaches the configured [`mutable_segment_rows`](/docs/operate/performance-tuning/writes#increase-mutable-segment-size) threshold—1,000 rows by default—it is frozen and becomes eligible for conversion into an immutable segment.
These segment files are organized by size into layers or levels. Newer data is written to the topmost layer. Over time, data is gradually pushed down into lower levels through a process called merging or compaction, where data from smaller segments is merged, deduplicated, and rewritten into larger segments.
ParadeDB reuses the current mutable segment across `INSERT`, `UPDATE`, and `COPY` statements until it reaches that threshold. Each resulting immutable segment has its own inverted, columnar, and vector index structures, which means that the ParadeDB index
is actually a collection of many segment-local indexes. These structures allow dense intersection queries to rapidly filter matches and route Top K work to the most promising segments.
## Query Execution
### Custom Operators
ParadeDB introduces several new text search operators to Postgres. For example, `|||` is used for [match disjunction](/docs/reference/full-text/match) queries, whereas `###`
is for [phrase](/docs/reference/full-text/phrase) queries.
```sql theme={null}
SELECT * FROM mock_items
WHERE description ||| 'running shoes';
```
ParadeDB’s custom query execution paths are only triggered when at least one of ParadeDB's operators is present in the query. Otherwise, it is executed entirely by native Postgres.
### Custom Scan
When a supported query uses a ParadeDB operator and a matching ParadeDB index is present, ParadeDB can execute it using a [custom scan](https://www.postgresql.org/docs/current/custom-scan.html).
Custom scans are execution nodes set aside by Postgres that allow extensions to run custom logic during a query. They are more powerful and versatile than typical Postgres index scans because they
allow the extension to "take over" large parts of the query, including aggregates, `WHERE`, and even [`GROUP BY` clauses](/docs/project/roadmap#deeper-analytics-improvements).
From a performance perspective, custom scans significantly speed up queries by pushing down filters, aggregates, and other operations directly into the index, rather than applying them afterward in separate phases.
To understand what kind of scan is used, run `EXPLAIN`:
```sql theme={null}
-- Native Postgres scan, no ParadeDB operator
EXPLAIN SELECT * FROM mock_items
WHERE description = 'running shoes' AND rating <= 5;
-- Custom scan, ParadeDB operator used
EXPLAIN SELECT * FROM mock_items
WHERE description ||| 'running shoes' AND rating <= 5;
```
As a rule of thumb: if `EXPLAIN` shows a custom scan (or, in rare cases, a ParadeDB index scan), then that part of query is going through ParadeDB. Otherwise, the query passes through standard Postgres.
### Parallelization
For queries that need to read large amounts of data, such as [Top K](/docs/reference/full-text/top-k) or aggregate queries, ParadeDB custom scans can use additional Postgres workers to execute the query
in parallel. To see if a query was parallelized, run `EXPLAIN ANALYZE`:
```sql theme={null}
-- Top K queries may be parallelized
EXPLAIN ANALYZE SELECT * FROM mock_items
WHERE description ||| 'running shoes'
ORDER BY rating LIMIT 5;
```
Parallelization also depends on the [number of available
workers](/docs/operate/performance-tuning/reads).
Postgres supports [parallel scans, joins, and two-stage aggregation](https://www.postgresql.org/docs/current/parallel-plans.html). ParadeDB builds on those parallel-worker primitives while pushing supported filters, Top K, joins, and aggregates into the index. For joins and aggregates, ParadeDB can distribute the pushed-down plan across workers using an MPP execution strategy.
## Design Philosophy
* **Keep it Boring**. Use robust extension points in Postgres vs. hacking around the internals. Adopt battle-tested tools, like industry standard file formats and query engine libraries, instead of cutting-edge but less-proven alternatives.
* **Behave Exactly Like Postgres**. This extends from user-facing aspects, like the SQL query syntax and ORM compatibility, all the way down to low-level integrations with Postgres' storage system and query planner.
* **Work Out of the Box**. Users should be able to get satisfying search results and performance with minimal tuning or configuration.
## Dependencies
The three main dependencies of `pg_search` are:
* [pgrx](https://github.com/pgcentralfoundation/pgrx/tree/develop) — the library for writing Postgres extensions in Rust
* [Tantivy](https://github.com/quickwit-oss/tantivy) — a Rust-based full-text search library inspired by [Lucene](https://github.com/apache/lucene)
* [Apache DataFusion](https://github.com/apache/datafusion) — an extensible query execution framework for OLAP processing