GitHub user vinothchandar edited a discussion: RocksDB as The Replica of MDT/RLI
### RocksDB as The Replica of MDT The initial Flink RLI RFC: https://github.com/apache/hudi/pull/17610 ### The update to RocksDB The rocksDB instantces are initialized and bootstrapped from scratch by reading the full MDT RLI index for each job restart or task failover. The incremental index upserts inferred from the data inputs are applied directly on these RocksDB instances, these index upserts are passed along with the data payloads altogether to the `IndexWrite` op for actual MDT update. The MDT update happens in the same lifecycle of data records write and the incremental upserts are a replica image of the upserts into the RocksDB. The RLI would be utilized for two cases: - serves as the source of truth of the index mapping and been utilized in the RocksDB bootstrap - cross engine compatibility The new write flow with RockDB replica: <img width="4696" height="1386" alt="image" src="https://github.com/user-attachments/assets/d99a624e-84b8-4c46-9a60-2799b7cb79d3"; /> ### The Clean/Eviction of Index Payloads in RocksDB For global RLI, the rocksDB instance would be closed and removed each time a task fails over or got a job restart. For partitioned RLI, for local RocksDB instance per `BucketAssign` task, the paylods under the same data partition is stored as a separate column family, when the data partition is based on datetime, the column family can be dropped very efficiently with a configurable partition lookup TTL. ### The Additional Storage Cost A local transcoding from HFile to RocksDB sst shows that the storage size diff mainly comes from the compression codec, Hudi uses gzip as default while RocksDB uses snappy, and the RocksDB storage size turns out to be a nearly 2x size storage against the native HFile(the RocksDB WAL has been disabled explicitly since it is not necessary in our use cases). <img width="616" height="244" alt="image" src="https://github.com/user-attachments/assets/ba01023f-f744-454c-839e-e5bebf0ba28c"; /> <img width="727" height="192" alt="image" src="https://github.com/user-attachments/assets/708e1d98-b778-43f9-a157-7d3fa131ee5f"; /> GitHub link: https://github.com/apache/hudi/discussions/18296 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
