Ytimetravel opened a new issue, #10803: URL: https://github.com/apache/hudi/issues/10803
Dear community: I found an issue while using hudi, which is that there may be data loss when compacting incremental data.Through investigation, it is found that the problem is introduced by the following bugfix：[[[HUDI-6758](https://issues.apache.org/jira/browse/HUDI-6758)]] Detecting and skipping Spurious log blocks with MOR reads (#9545).After testing, it was found that choosing to speculate on the files generated by execution seems to have a probability of causing data loss. **Expected behavior** While retaining this optimization, it is also necessary to ensure that the selected files are always correct to ensure the accuracy of the data. **Environment Description** * Hudi version : 0.14.0 **Additional context** The main logic that led to this issue is as follows： <img width="1385" alt="image" src="https://github.com/apache/hudi/assets/112808605/c83e5314-b9b1-4b18-935d-74681ff55fe9";> I temporarily avoided this issue by retaining all log files with the same number of blocks. I would like to know if there is a better way to solve this problem. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
