danny0405 commented on a change in pull request #4812:
URL: https://github.com/apache/hudi/pull/4812#discussion_r818514246
##########
File path:
hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/RequestHandler.java
##########
@@ -134,7 +136,9 @@ private boolean isLocalViewBehind(Context ctx) {
}
String localTimelineHash = localTimeline.getTimelineHash();
- if (!localTimelineHash.equals(timelineHashFromClient)) {
+ // refresh if timeline hash mismatches and if local's last known instant <
client's last known instant
+ if (!localTimelineHash.equals(timelineHashFromClient)
+ && HoodieTimeline.compareTimestamps(localLastKnownInstant,
HoodieTimeline.LESSER_THAN, lastKnownInstantFromClient)) {
Review comment:
Finally i got the reason:
We have a start commit method that may generate a rollback instant with
greater timestamp that the actual passed in instant time:
https://github.com/apache/hudi/blob/907e60c252e80be5ef3a848d773e0f866eb609f9/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/BaseHoodieWriteClient.java#L862
And unfortunately, flink uses that (spark uses that too), here is how the
problem comes:
delta_commit compaction delta_commit rollback_commit
--- t1 --------------- t2 ------------ t3 ------------- t4 ------------
The t4 was created before t3 was created and it was with the highest
timestamp t4, then the following sequence happens:
1. the rollback action would then refresh the remote timeline service with
the latest timestamp t4 (remember the fs view as **V1**)
2. the t3 delta commit start to execute and commit, say the commit was
successful
3. then we want to trigger the compaction after the commit of t3
And the tricky things happens:
the compaction scheduler takes the client, the client uses the latest
timestamps on timeline and tries to fetch all the fileslices, but because the
client timestamp t4 equals with the remote timeline time t4, the view does not
sync and we still got **V1** fs view here and we can not find any compaction
plan because there was no log files in the view.
Here is my fix patch to make sure the rollback timestamp not greater than
the delta commit time.
[HUDI-2761.patch.zip](https://github.com/apache/hudi/files/8176836/HUDI-2761.patch.zip)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
