Perplexity Replaces DynamoDB With CobbleDB for AI Retrieval

Key takeaways

• 4 ms to 5.60 ms (an 82% decrease) and cut 99th-percentile tail latency from 123 ms to 24.2 ms.
• Decoupled Architecture: Perplexity separated durable storage on YTsaurus (Pillar) and batch queueing in S3 (Lorry) from the low-latency serving tier (CobbleDB) powered by RocksDB and local NVMe SSDs.
• Workload Mismatch: DynamoDB’s usage-based billing and black-box partitioning proved uneconomical for AI retrieval patterns requiring 100 to 120 keys with 50 KB payloads per query at 200,000 requests per second.
• Operational Trade-offs: Migrating off AWS managed databases shifts hardware lifecycle management, backup verification, and partition rebalancing to internal SRE teams under an eventual consistency model.
• AI-Assisted Development: Two systems engineers built the 40,000 lines of Rust in two months, using autonomous swarms of AI coding agents for testing, build monitoring, and runbooks.

Perplexity has transitioned its core search serving tier away from Amazon DynamoDB to CobbleDB, an internally developed distributed key-value store written in Rust. The migration directly addresses severe latency bottlenecks and rising cloud expenses driven by large language model retrieval patterns. By decoupling durable document storage from hot-tier retrieval, Perplexity reduced median batch-read latency from 31.4 milliseconds to 5.60 milliseconds—an 82 percent drop—while trimming 99th-percentile tail latency from 123 milliseconds to 24.2 milliseconds and lowering cloud storage expenses by at least 20 percent. Synthetic benchmarks show CobbleDB sustains up to 500,000 requests per second across 100-kilobyte payloads, demonstrating how specialized database architectures can overcome the latency and cost limitations of general-purpose managed cloud databases.

The AI Retrieval Bottleneck That Broke DynamoDB

Artificial intelligence answer engines impose operational patterns that diverge sharply from traditional web search engines. Conventional search platforms index documents to return concise metadata snippets, title tags, and short page descriptions. In contrast, generative AI platforms like Perplexity must extract and feed substantial text context and mathematical embeddings into downstream language models for real-time inference and synthesis.

Each query submitted to Perplexity generates between 100 and 120 target page keys. The retrieval service divides these keys into parallel batches of 10 to 20 keys each. Unlike traditional systems where individual records contain small metadata fragments, retrieval for generative models requires extracting full chunked passages alongside dense vector representations. This requirement results in average record payloads of approximately 50 kilobytes.

At production traffic scales exceeding 200,000 requests per second, Amazon DynamoDB became economically and architecturally unsustainable. DynamoDB uses a usage-based pricing model that meters every byte transferred and every read capacity unit consumed. Processing tens of thousands of concurrent 50-kilobyte batch requests generated massive data-transfer bills. While traditional database architectures often integrate lexical retrieval directly into relational storage—such as Google Cloud adding native BM25 ranking to AlloyDB and Cloud SQL—generative retrieval workloads demand sustained high-throughput bulk reads of multi-kilobyte records that strained DynamoDB’s metering structure.

Beyond financial constraints, DynamoDB operates as a managed black box that conceals internal partition placement, memory caching policies, and replica routing from infrastructure teams. Perplexity engineers could not mitigate tail-latency spikes caused by uncached reads, cross-zone networking hops across AWS datacenters, or lagging internal storage replicas. Furthermore, offline reprocessing pipelines triggered by updated text-chunking algorithms or newer embedding models pushed heavy write volumes directly into DynamoDB. These write workloads created severe noisy-neighbor contention against live user requests, degrading end-user query response times.

Decoupling Storage: Pillar, Lorry, and CobbleDB

To eliminate resource contention and isolate retrieval traffic, Perplexity split its data infrastructure into three specialized, decoupled subsystems: Pillar for durable state management, Lorry for batch aggregation and staging, and CobbleDB for low-latency search serving.

Pillar serves as the system’s durable ground-truth foundation. Running on YTsaurus over high-capacity mechanical hard disk drives (HDDs), Pillar manages versioned table families for web page metadata, extracted text passages, and dense vector representations. Mechanical hard drives provide cost-effective capacity for petabyte-scale web indexes where bulk sequential write throughput matters more than sub-millisecond random access. Atomic YTsaurus transactions guarantee that web crawl updates, state mutations, and export queues commit together reliably, preventing data corruption or state divergence across versions.

Lorry functions as a stateless queue consumer that bridges durable archival storage with the hot serving layer. Lorry ingests mutation streams and export queues from Pillar, organizes them into partition-aligned batch files, and uploads the compressed payloads to Amazon S3. Once batch files are safely committed to object storage, Lorry transmits lightweight metadata notices to the CobbleDB cluster indicating that fresh partition snapshots are ready for ingestion.

CobbleDB operates exclusively as the high-throughput, low-latency serving tier. Worker nodes within the CobbleDB cluster pull and ingest the partition batch files from Amazon S3 asynchronously and independently. Because CobbleDB nodes never handle crawl writes, database mutations, or transaction commits directly, the hot serving tier remains completely isolated from the noisy-neighbor interference that previously affected DynamoDB during index refreshes.

Inside CobbleDB: RocksDB MultiGet, AZ Routing, and Speculative Hedging

Written in Rust, CobbleDB is a distributed key-value store designed specifically for batched key lookups. The database partitions its global keyspace using consistent hashing, maintaining three replicas for every partition distributed across independent compute nodes to provide resilience against hardware failures.

At the node level, the core CobbleDB daemon embeds RocksDB as its underlying storage engine. Each node pairs RocksDB with high-performance local NVMe solid-state drives (SSDs) and memory-mapped page caching. Frequently accessed records remain pinned in system memory, while colder records are fetched from local NVMe storage with microsecond read latencies, bypassing network hops entirely.

In front of the storage nodes, a stateless query router coordinates read execution for incoming requests from Perplexity’s inference pipeline. The router hashes incoming page identifiers to locate responsible partition replicas. To minimize network overhead across AWS infrastructure, the query router enforces availability-zone (AZ) affinity, preferentially directing queries to node replicas situated within the same datacenter availability zone as the calling inference service.

To address the long-tail latency spikes common in distributed systems, the query router incorporates speculative request hedging. If a primary replica exhibits elevated response times or experiences temporary garbage collection pauses, the router issues a concurrent, speculative read request to an alternate replica on a different node. The router accepts whichever response returns first and cancels the slower execution, effectively trimming tail-latency outliers.

Get the Weekly Brief

Curated analysis for tech leaders. Every Thursday.

Subscribe

Within individual storage nodes, CobbleDB eliminates sequential key lookup overhead. Instead of issuing repetitive single-key reads, the engine executes batched retrieval requests using RocksDB’s native MultiGet interface:

pub struct BatchedPageRequest {
    pub keys: Vec<PageKey>,
    pub zone_affinity: AvailabilityZone,
}

impl StorageEngine {
    pub fn multi_get_pages(&self, keys: &[PageKey]) -> Result<Vec<Option<PageRecord>>, Error> {
        let rocksdb_keys: Vec<&[u8]> = keys.iter().map(|k| k.as_bytes()).collect();
        self.rocksdb.batched_multi_get(&rocksdb_keys)
    }
}

By leveraging RocksDB’s batched multi-get operations, CobbleDB reads contiguous disk blocks and shared memory buffers in a single pass, eliminating redundant context switches and internal locking contention.

Measured Latency Reductions and Cloud Cost Impact

Production benchmarks published by Perplexity demonstrate that replacing DynamoDB with CobbleDB yielded significant latency reductions across all measurement percentiles. In live traffic serving production search queries, CobbleDB achieved the following performance metrics:

• Median (p50) Latency: Dropped from 31.4 milliseconds to 5.60 milliseconds, representing an 82 percent reduction in batch retrieval time.
• 90th-Percentile (p90) Latency: Decreased from 56.7 milliseconds down to 9.77 milliseconds.
• 99th-Percentile (p99) Tail Latency: Fell from 123.0 milliseconds to 24.2 milliseconds, an 80 percent reduction that stabilizes multi-turn model synthesis.

In synthetic stress testing conducted with record payloads up to 100 kilobytes, CobbleDB demonstrated sustained query throughput up to 500,000 requests per second before encountering performance degradation. This throughput headroom provides substantial operational buffer beyond Perplexity’s baseline traffic of 200,000 requests per second.

From an infrastructure economics standpoint, decoupling long-term data durability from hot retrieval enabled Perplexity to lower overall storage expenses by at least 20 percent. Storing immutable web indices on mechanical hard drives in Pillar and intermediate snapshots in Amazon S3 proved substantially cheaper than retaining multi-terabyte active indices within DynamoDB’s metered storage tier.

Operational Trade-offs, Eventual Consistency, and AI-Assisted Development

While CobbleDB resolves severe latency and cost bottlenecks, migrating away from a fully managed cloud service introduces notable operational trade-offs and complexity for platform engineering teams.

Under Amazon DynamoDB, AWS manages server provisioning, disk failure recovery, security patches, automated backups, and partition rebalancing behind service-level guarantees. By adopting a self-hosted database, Perplexity transfers node lifecycle management, backup verification, and partition rebalancing directly onto its internal site reliability engineering (SRE) staff. Platform teams must actively manage hardware failures, monitor disk wear on NVMe drives, and provision compute nodes to meet shifting query loads.

CobbleDB also alters data consistency semantics. To maintain maximum read performance and avoid the latency overhead of distributed consensus protocols like Raft or Paxos, CobbleDB discards distributed transactions. Because search serving can tolerate minor replication lag, replicas pull and ingest batch files from S3 asynchronously at independent intervals. Consequently, the serving tier operates under an eventual consistency model where different replicas may briefly serve different versions of an indexed page.

Despite these operational demands, Perplexity developed CobbleDB with a remarkably small human engineering footprint. Chief Executive Officer Aravind Srinivas noted that the database—comprising roughly 40,000 lines of Rust code—was implemented in just two months by two systems engineers. The engineering team augmented their efforts using an autonomous swarm of AI coding agents that generated unit and integration tests, monitored continuous integration builds, and authored operational runbooks. Perplexity has also confirmed plans to release CobbleDB as open-source software, providing the broader developer community with access to its specialized serving architecture.

Bottom line

Perplexity’s migration from Amazon DynamoDB to CobbleDB highlights a growing challenge across generative AI infrastructure: standard managed cloud databases built for general-purpose web transactions are poorly aligned with the massive batch-read payloads and tight latency constraints of real-time language model retrieval. By decoupling durable storage from low-latency serving and tailoring its engine around RocksDB and local NVMe hardware, Perplexity achieved an 82 percent reduction in median read latency and cut cloud storage costs by more than 20 percent.

For engineering leaders and cloud architects, CobbleDB demonstrates that specialized, in-house data stores can deliver decisive performance and cost advantages at high traffic scales. However, these gains require accepting eventual consistency and dedicating internal site reliability resources to node lifecycle management, partition rebalancing, and cluster maintenance.

Sources

• Home Made CobbleDB Replaces DynamoDB at Perplexity to Cut Query Latency 5x and Reduce Cloud Storage – InfoQ
• Perplexity’s AI agents helped build a database. They weren’t allowed to run it. – The New Stack
• CobbleDB Achieves Faster AI Search with Cost-Effective Storage Solutions | Welcome.AI
• Perplexity CobbleDB: When Leaving DynamoDB Makes Sense
• Perplexity Replaces DynamoDB With Homegrown CobbleDB, Cutting Query Latency Fivefold | LavX News
• Perplexity CobbleDB DynamoDB replacement cuts batch-read latency 82 percent | Justo Global
• Perplexity CobbleDB Replaces DynamoDB, With a Claimed $100 Million Annual Saving