boneanxs opened a new pull request, #7627: URL: https://github.com/apache/hudi/pull/7627
### Change Logs _Describe context and summary for this change. Highlight if any code was copied._ 1. `HoodieInstant` add state transition time support 2. Add some methods relating to state transition time in HoodieTimeline 3. Add new configure to allow incremental pulling data by state transition time 4. Spark `HoodieStreamSource` need to use state transition time to incremental pulling data ### Impact Hudi timeline can actually miss some instants if we incremental pulling from upstream hudi table, which is written by several writers. For example, say we have 2 writers writing data to the hudi table, and the last success incremental pulling end timestamp is 001 w1 is writing 002, w2 is writing 003, if w2 is finished earlier than the w1, then the incremental pulling end timestamp will be updated to 003, and actually w1's commit: 002 will be skipped since it's instant time is earlier than the w2's. We actually needs to use commit end time(state transition time) to filter the commits if using incremental pulling. As w2's state transition time is earlier than the w1's, so w1's data won't be filtered. This relates to the [HUDI-1623](https://issues.apache.org/jira/browse/HUDI-1623) but not adding end time to the end of each commit, instead use `FileStatus.getModificationTime` to represent the end time. ### Risk level (write none, low medium or high below) low ### Documentation Update _Describe any necessary documentation update if there is any new feature, config, or user-facing change_ - _The config description must be updated if new configs are added or the default value of the configs are changed_ - _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make changes to the website._ ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Change Logs and Impact were stated clearly - [ ] Adequate tests were added if applicable - [ ] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
