nsivabalan commented on code in PR #20651:
URL: https://github.com/apache/hudi/pull/20651#discussion_r4566551059


##########
hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/RequestHandler.java:
##########
@@ -749,7 +749,7 @@ private boolean isLocalViewBehind(Context ctx) {
         return false;
       }
 
-      String localTimelineHash = localTimeline.getTimelineHash();
+      String localTimelineHash = 
localTimeline.getTimelineHash(getTimelineHashNumInstantsParam(ctx));

Review Comment:
   **Blocking: a late-completing instant outside the window lets a stale server 
skip the reload.**
   
   The timeline is ordered by requested time, so an instant that started early 
but completed late lands outside the latest N. Example with the server's view 
cached from the previous write, while clustering `r005` is still inflight:
   
   ```text
   server cached: [..., c008, c009, c010]               (r005 inflight)
   client (new):  [..., r005 completed, ..., c008, c009, c010]
   N = 3 -> both sides hash [c008, c009, c010] -> no sync, final check passes
   ```
   
   The server still lists the file groups that `r005` replaced as live. The 
writer's merge or append handles can then write into replaced file groups, and 
those updates become invisible. A late-completing concurrent commit fails the 
same way (Danny's `001/002/clean003` example: the merge reads the base file 
from `001` instead of `002`). Async clustering and compaction routinely 
complete late, so the person setting N can't rule this out.



##########
hudi-common/src/main/java/org/apache/hudi/common/table/timeline/BaseHoodieTimeline.java:
##########
@@ -652,6 +652,14 @@ public HoodieTimeline mergeTimeline(HoodieTimeline 
timeline) {
   /**
    * Computes the timeline hash and returns.
    */
+  @Override
+  public String getTimelineHash(int numLatestInstants) {

Review Comment:
   nit: this was inserted between the existing Javadoc ("Computes the timeline 
hash and returns.") and `computeTimelineHash`, so that Javadoc now describes 
this method instead.



##########
hudi-timeline-service/src/test/java/org/apache/hudi/timeline/service/TestTimelineViewRefresh.java:
##########
@@ -429,11 +431,91 @@ void testRealViewRefreshesOutOfOrderCompletion() throws 
Exception {
     verify(view).sync();
   }
 
+  @Test
+  void testArchivedInstantsOutsideHashedInstantsDoNotSync() throws IOException 
{
+    HoodieInstant archived = instant(COMPLETED, COMMIT_ACTION, "001");
+    HoodieInstant second = instant(COMPLETED, COMMIT_ACTION, "002");
+    HoodieInstant boundary = instant(COMPLETED, COMMIT_ACTION, "003");
+    HoodieTimeline clientTimeline = timeline(archived, second, boundary);
+    // archival removed the client's oldest instant
+    when(view.getTimeline()).thenReturn(timeline(second, boundary));
+    request(clientTimeline, "003", "partition", 2);
+    // archival removed the client's oldest instant and a clean completed
+    when(view.getTimeline()).thenReturn(timeline(second, boundary, 
instant(COMPLETED, CLEAN_ACTION, "004")));
+    request(clientTimeline, "003", "partition", 2);
+    verify(view, never()).sync();
+  }
+
+  @ParameterizedTest
+  @ValueSource(ints = {0, 3})
+  void testArchivedInstantsWithinHashedInstantsSyncOnEveryRequest(int 
hashNumInstants) {
+    HoodieInstant second = instant(COMPLETED, COMMIT_ACTION, "002");
+    HoodieInstant boundary = instant(COMPLETED, COMMIT_ACTION, "003");
+    HoodieTimeline clientTimeline = timeline(instant(COMPLETED, COMMIT_ACTION, 
"001"), second, boundary);
+    // a sync reads the same archived timeline again, so the server never 
matches the client
+    when(view.getTimeline()).thenReturn(timeline(second, boundary, 
instant(COMPLETED, CLEAN_ACTION, "004")));
+    for (int i = 0; i < 3; i++) {
+      assertThrows(IOException.class, () -> request(clientTimeline, "003", 
"partition", hashNumInstants));
+    }
+    verify(view, times(3)).sync();
+  }
+
+  @Test
+  void testNewerInstantsWithinHashedInstantsStillSync() {
+    HoodieInstant second = instant(COMPLETED, COMMIT_ACTION, "002");
+    HoodieInstant boundary = instant(COMPLETED, COMMIT_ACTION, "003");
+    HoodieTimeline clientTimeline = timeline(instant(COMPLETED, COMMIT_ACTION, 
"001"), second, boundary);
+    when(view.getTimeline()).thenReturn(timeline(second, boundary, 
instant(COMPLETED, COMMIT_ACTION, "004")));
+    assertThrows(IOException.class, () -> request(clientTimeline, "003", 
"partition", 2));
+    verify(view).sync();
+  }
+
+  @Test
+  void testInstantsBeforeHashedInstantsAreNotCompared() throws IOException {

Review Comment:
   This test makes the unsafe case expected behaviour: an older replacecommit 
that the server sees and the client doesn't is accepted without a sync. In the 
reverse direction (stale server, fresh client), the same mismatch outside the 
window means the server serves file groups that the replacecommit already 
replaced. Could we instead have real-file tests that assert the files returned 
when an instant completes late?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to