Spotify's RAP Shows Data Lakes Can Handle Online Queries Too

The Data Lake's Long-Standing Problem

For years, the data lake has been the workhorse of analytics and machine learning. It holds everything—clickstreams, user profiles, transaction logs—in cheap object storage. But it has a dirty secret: if you need to fetch a single record quickly, it's painfully slow. Distributed query engines like Trino and BigQuery are built for scanning vast swaths of data, not for answering a question like "What's this user's current subscription tier?" That kind of point query is what operational databases do best.

Spotify knows this tension well. The company stores petabytes of online data in Bigtable, a NoSQL database built for low-latency access. Meanwhile, its data lake on Google Cloud Storage holds exabytes of data for analytics and AI. Keeping those two worlds in sync means copying data at massive scale, and that's getting expensive.

Enter RAP: An Index Layer for Parquet

In a recent technical post, Spotify engineers introduced Random Access Parquet, or RAP. The idea is simple: add an external index layer on top of Apache Parquet files so that point queries can skip the full-file scan. Instead of reading thousands of files, the query engine first looks up the key—say, a user ID—in the index, which points directly to the right file and the row position within it. Then it does a targeted range read from object storage.

The beauty is that the underlying Parquet files remain untouched. New data written to Apache Iceberg tables gets indexed in append-only segments, so the same dataset can serve analytics, machine learning pipelines, and latency-sensitive online applications without duplicating storage. It's a clever way to get the best of both worlds.

Why Point Queries Are So Slow in Data Lakes

To understand why RAP matters, you have to look at the mechanics of a typical point query on a data lake. Cloud object storage like Google Cloud Storage now offers millisecond-level latency for individual reads. But that's just the raw I/O. The real overhead comes from query planning, metadata traversal, and file discovery. A query engine might need to list thousands of files, read metadata to figure out which ones contain the relevant data, and then open those files—all before it can even start reading rows.

Spotify's engineers point out that this overhead can dominate the total query time. In their tests, a simple point query could end up scanning dozens or hundreds of files just to find one row. RAP sidesteps that by precomputing the mapping from key to file and row position, so the query engine knows exactly where to look.

Storage Layout Tweaks That Cut Latency Further

RAP isn't just about the index. Spotify also implemented several storage layout optimizations to squeeze out even more performance. For starters, they sort data by the query key—like user ID—so that records for the same user end up in the same file, reducing the number of files that need to be touched.

They also group related records together and interleave value columns so that a single continuous read can fetch multiple attributes at once. In some cases, they use covering indexes, which allow certain queries to be answered entirely from the index without ever reading the Parquet file. The result: some point queries now require just one range read of a few kilobytes.

Secondary Indexes Without Rewriting Files

One of the trickier challenges is supporting multiple query dimensions. A user might search by buyer ID, seller ID, or some other attribute. Rewriting the Parquet files to sort by each dimension would be impractical. Instead, Spotify built secondary indexes that live at the service layer. These indexes map keys to file and row positions, just like the primary index, but they don't require any changes to the data files themselves.

This means you can add a new access path without touching your data pipeline. The same Parquet dataset continues to serve analytics scans, while the service layer handles interactive point queries. For range queries, Spotify uses sorted indexes; for exact lookups, hash-based indexes work well. And techniques like Z-ordering or Hilbert curves can improve data locality for these secondary dimensions.

A Growing Trend: Data Lakes for Operational Workloads

Spotify's RAP is part of a broader industry push to make data lakes work for more than just batch analytics. Google Cloud recently described a lakehouse architecture based on Apache Iceberg that aims to support AI applications with operational data access. The idea is to reduce data duplication by letting the lake serve both analytical and operational queries.

RAP differentiates itself by adding a dedicated external index layer optimized for point queries, while staying fully compatible with existing Parquet files and Iceberg tables. That's a pragmatic approach—it doesn't require rewriting your storage format or migrating to a new system.

Community Reactions and the Bigger Picture

The data engineering community has taken notice. Andrew Lamb sees RAP as an example of extending open data formats to support interactive workloads. Vikas Singh, in a LinkedIn discussion, noted that as cloud object storage gets faster, the bottleneck for point queries shifts to query planning and metadata access—exactly what RAP's precomputed indexes address.

For companies running massive data lakes, the implications are significant. If you can serve online queries directly from the lake, you might be able to retire some of those operational databases, or at least shrink the amount of data you need to copy. That's a cost saving, but it's also a simplification of your data architecture.

What This Means for Industry Analysts

From an industry analysis perspective, RAP is a signal that data lakes are evolving into multi-purpose platforms. The traditional boundary between analytical systems and operational systems is blurring. Open table formats like Iceberg and Hudi are already making lakes more database-like, and now we're seeing index layers that bring point-query performance into the mix.

The challenge is that these solutions are still early. RAP is specific to Spotify's stack, though the underlying concepts are generalizable. We're likely to see more open-source projects and commercial offerings that add indexing to data lakes in the coming years. For vendors, this is both an opportunity and a threat: if lakes can handle operational workloads, why would you need a separate database for that?

That's a question that keeps database vendors up at night. For now, the pragmatic answer is that operational databases still have advantages in consistency, transactionality, and mature tooling. But the gap is narrowing. As cloud storage gets faster and index layers get smarter, the data lake may become the single source of truth for everything—analytics, AI, and online services.

The Bottom Line

Spotify's RAP is a clever engineering solution to a real problem. By adding an index layer on top of Parquet, they've made it possible to run low-latency point queries on data lake data without copying it elsewhere. It's a win for efficiency and a step toward a more unified data architecture.

For industry analysts, the takeaway is clear: keep an eye on data lake indexing technologies. They have the potential to reshape how companies store and serve data, and they're coming faster than you might think.