Feature Serving Beyond RAM

Feature Serving Beyond RAM Online feature stores have a quiet cost problem: the long tail. Per-user and per-query features get read maybe once a day, but they sit in Redis at RAM prices, just in case.

We hit this wall in Metarank, an open-source search reranker. Ranking never does polite point lookups: one request wants 100 features for 500 candidates, a hash-per-row layout turns that into thousands of scattered reads, and Python parsing pushes the total past 100 ms. Then the AWS bill arrives, and your better ranking costs more than the revenue it brings.

This talk is the story of Murrdb, an open-source columnar cache for batch feature reads, from tiny counters to fat embedding vectors; a serving layer that slots under whatever feature store you already run. Hot features live in RAM, warm ones spill to NVMe, cold ones retire to S3, so memory prices apply only to data that earns them. It speaks Arrow on the wire, so even in Python you can basically YOLO-cast a response into a pd.DataFrame, and workers stay fresh by polling the offline store for new Parquet partitions, no pipeline to babysit.

Can tiered storage stay fast enough for online serving? On 100M rows and 1000-key batch reads, Murrdb answers a Python client in 1.1 ms end-to-end, versus 2.5 ms for Redis packed blobs and 8.5 ms for a Feast-style hash layout at three times the RAM. Capped to 2 GiB, it still serves from NVMe in under 7 ms. We'll walk through the numbers, and be honest about where the in-memory crowd still wins.