Architecture of Fennel’s realtime feature platform

Fennel is a modern feature engineering platform built by an ex-Facebook team and is based on several unique architectural choices. As one example, unlike other feature platforms, Fennel doesn’t rely on Spark/Flink for realtime computation but instead uses a home grown streaming system written in Rust. As another example, Fennel has an explicit read/write separation. In this talk, we will describe the architecture powering Fennel and the various design decisions & tradeoffs that led to this architecture. We will also describe the powerful capabilities unlocked by some of these design decisions.