nsivabalan commented on code in PR #20651:
URL: https://github.com/apache/hudi/pull/20651#discussion_r4566551059
##########
hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/RequestHandler.java:
##########
@@ -749,7 +749,7 @@ private boolean isLocalViewBehind(Context ctx) {
return false;
}
- String localTimelineHash = localTimeline.getTimelineHash();
+ String localTimelineHash =
localTimeline.getTimelineHash(getTimelineHashNumInstantsParam(ctx));
Review Comment:
**Blocking: a late-completing instant outside the window lets a stale server
skip the reload.**
The timeline is ordered by requested time, so an instant that started early
but completed late lands outside the latest N. Example with the server's view
cached from the previous write, while clustering `r005` is still inflight:
```text
server cached: [..., c008, c009, c010] (r005 inflight)
client (new): [..., r005 completed, ..., c008, c009, c010]
N = 3 -> both sides hash [c008, c009, c010] -> no sync, final check passes
```
The server still lists the file groups that `r005` replaced as live. The
writer's merge or append handles can then write into replaced file groups, and
those updates become invisible. A late-completing concurrent commit fails the
same way (Danny's `001/002/clean003` example: the merge reads the base file
from `001` instead of `002`). Async clustering and compaction routinely
complete late, so the person setting N can't rule this out.
##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/BaseHoodieTimeline.java:
##########
@@ -652,6 +652,14 @@ public HoodieTimeline mergeTimeline(HoodieTimeline
timeline) {
/**
* Computes the timeline hash and returns.
*/
+ @Override
+ public String getTimelineHash(int numLatestInstants) {
Review Comment:
nit: this was inserted between the existing Javadoc ("Computes the timeline
hash and returns.") and `computeTimelineHash`, so that Javadoc now describes
this method instead.
##########
hudi-timeline-service/src/test/java/org/apache/hudi/timeline/service/TestTimelineViewRefresh.java:
##########
@@ -429,11 +431,91 @@ void testRealViewRefreshesOutOfOrderCompletion() throws
Exception {
verify(view).sync();
}
+ @Test
+ void testArchivedInstantsOutsideHashedInstantsDoNotSync() throws IOException
{
+ HoodieInstant archived = instant(COMPLETED, COMMIT_ACTION, "001");
+ HoodieInstant second = instant(COMPLETED, COMMIT_ACTION, "002");
+ HoodieInstant boundary = instant(COMPLETED, COMMIT_ACTION, "003");
+ HoodieTimeline clientTimeline = timeline(archived, second, boundary);
+ // archival removed the client's oldest instant
+ when(view.getTimeline()).thenReturn(timeline(second, boundary));
+ request(clientTimeline, "003", "partition", 2);
+ // archival removed the client's oldest instant and a clean completed
+ when(view.getTimeline()).thenReturn(timeline(second, boundary,
instant(COMPLETED, CLEAN_ACTION, "004")));
+ request(clientTimeline, "003", "partition", 2);
+ verify(view, never()).sync();
+ }
+
+ @ParameterizedTest
+ @ValueSource(ints = {0, 3})
+ void testArchivedInstantsWithinHashedInstantsSyncOnEveryRequest(int
hashNumInstants) {
+ HoodieInstant second = instant(COMPLETED, COMMIT_ACTION, "002");
+ HoodieInstant boundary = instant(COMPLETED, COMMIT_ACTION, "003");
+ HoodieTimeline clientTimeline = timeline(instant(COMPLETED, COMMIT_ACTION,
"001"), second, boundary);
+ // a sync reads the same archived timeline again, so the server never
matches the client
+ when(view.getTimeline()).thenReturn(timeline(second, boundary,
instant(COMPLETED, CLEAN_ACTION, "004")));
+ for (int i = 0; i < 3; i++) {
+ assertThrows(IOException.class, () -> request(clientTimeline, "003",
"partition", hashNumInstants));
+ }
+ verify(view, times(3)).sync();
+ }
+
+ @Test
+ void testNewerInstantsWithinHashedInstantsStillSync() {
+ HoodieInstant second = instant(COMPLETED, COMMIT_ACTION, "002");
+ HoodieInstant boundary = instant(COMPLETED, COMMIT_ACTION, "003");
+ HoodieTimeline clientTimeline = timeline(instant(COMPLETED, COMMIT_ACTION,
"001"), second, boundary);
+ when(view.getTimeline()).thenReturn(timeline(second, boundary,
instant(COMPLETED, COMMIT_ACTION, "004")));
+ assertThrows(IOException.class, () -> request(clientTimeline, "003",
"partition", 2));
+ verify(view).sync();
+ }
+
+ @Test
+ void testInstantsBeforeHashedInstantsAreNotCompared() throws IOException {
Review Comment:
This test makes the unsafe case expected behaviour: an older replacecommit
that the server sees and the client doesn't is accepted without a sync. In the
reverse direction (stale server, fresh client), the same mismatch outside the
window means the server serves file groups that the replacecommit already
replaced. Could we instead have real-file tests that assert the files returned
when an instant completes late?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]