Random Access Parquet: Serving Point Queries Straight from the Data Lake

Random Access Parquet (RAP) is a way to look up a single key directly in the Parquet files you already have in your data lake. An external index records exactly which file, and where in it, each key lives — so a lookup becomes one small ranged read from cloud storage instead of a query engine scan. The index stays small (terabytes of data need only gigabytes of index), and with some care in how the files are written, a lookup can shrink to a few kilobytes read, or skip the file entirely.

This unlocks data that never earns a place in an online store: history, long-tail entities, everything too big or too rarely asked for to justify keeping resident in a key-value store. Instead of a binary choice between "loaded online" and "stuck in the lake", RAP offers a smooth dial between the two — trading a little latency for a lot of cost, key by key. This talk covers how the index works, how to prepare files so reads stay small, and how to place your workload on that dial.