GitHub user vinothchandar added a comment to the discussion: RLI support for Flink streaming
### 1. Writing structure and missing details. - please separate design and implementation into two separate sections. Not mix them, makes it hard to focus on the high-level design alone (which I think is what we should agree on first). This will make it easy to follow for any contributor/developer. - Also, can we cleanup the writing to e.g the goal I am not sure what you intend to convey by "RLI may not work well for .." . - Please spell out bullets for goals and non-goals in more detailed fashion. e.g. Can the RLI written by Flink be used by a Spark/Trino query for data skipping. Just an example. But we need to clearly state the intended functional and performance goals. ### 2. Precise answers to 'What exactly happens during a Flink checkpoint? When do we exactly commit the write ?' > And when the commit to the data table, the MDT is committed firstly with the > partial RLI write metadata list. I don't fully follow this. I am specifically interested in this. The key problem to solve here IMO is : when a Flink checkpoint happens it can give us a list of data, mt files written. But some data files returned, its corresponding write to RLI may not have propagated. IIUC, in the proposed approach, you are just writing MT ahead of writing the data files (just like the Flink state backend approach rn), to keep this synchronized? ### 3. Caching parts of RLI / partitioned RLI We are not addressing how we scale the cache. And what is expected to be hot. IMO we need partitioned RLI and the cache should be such that the "hot" partition paths's RLI entries are what are cached.. Missing these details. > We need a memory cache for the index mappings of current checkpoint because > it is not committed to Hudi table yet so invisible. we need to invalidate/refresh the cache as well after each write is successful . How does this all happen. I agree that we need some caching. but we need to flesh this out more. We can come back to the caching after we align on the original approach. ### 4. Consider Secondary Index also in scope The current design is not very extensible to secondary index updates. I'd like for that to be included in scope. In case of SI, the main challenge is that the index entries cannot be produced until we actually perform the writing of the data files. MT is updated in a downstream operator from the current operator that writes data files.. GitHub link: https://github.com/apache/hudi/discussions/17452#discussioncomment-15177474 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
