danny0405 commented on issue #20650:
URL: https://github.com/apache/hudi/issues/20650#issuecomment-6847412966

   Following up with a correctness-focused comparison of #20651 (head 
`300756f645b6963a63ef55ef4c728cecc841eda0`) and the [archived-prefix 
reconstruction 
proposal](https://github.com/apache/hudi/issues/20650#issuecomment-6847315815).
   
   The main difference is **ignoring older history versus verifying that the 
missing history moved into the archive**. Both can serve requests remotely in 
the reported archival case, but they establish different guarantees.
   
   | Aspect / scenario | #20651: latest-N hash, when enabled | Proposed: client 
first instant + archived-prefix reconstruction |
   | --- | --- | --- |
   | What is compared | Only the latest N instants, including their requested 
times, actions, and states. N = 0 retains the existing full hash. | The full 
client hash against the server active timeline plus archived identities in 
`[clientFirst, serverFirst)`, using consistent ordering/filtering. |
   | Pure prefix archival, optionally with one trailing clean | Can serve 
remotely if the selected suffix survives archival and matches. Requires 
choosing N. | Can serve remotely if the reconstructed full hash matches. No N 
to tune; the existing single-trailing-clean exception is retained. |
   | Server cache missed an older commit that the client has observed | Can 
incorrectly skip refresh if the difference is outside the latest N instants; 
final validation also passes. | The difference remains part of the comparison. 
Unless archive reconstruction genuinely explains it, retain the existing 
sync-and-validation path. |
   | Server has observed an older late completion that the client has not | Can 
accept it when outside the window. This changes the existing consistency 
contract, but a newer committed view is not by itself proof of incorrect 
writes. | Detects the additional instant, including in the reconstructed 
prefix, and preserves the existing mismatch handling. It does not introduce 
general acceptance of newer snapshots. |
   | Older compaction/clustering action or state changes | Can miss transitions 
outside the window, even though pending-operation state affects file selection. 
| No intentional blind window: retain action/state checking across the 
reconstructed history. Real-file tests are still needed to establish file-view 
behavior. |
   | Restore / rollback leaves extra or changed instants in the cached server 
view | Changes outside the selected suffix can be hidden. Changes inside it 
remain detectable. | Do not truncate the server's tail or remove non-archived 
differences to manufacture a match. Such differences retain the existing 
refresh path; restore regression tests remain required. |
   | Archive unavailable, holes in the timeline, or no active overlap | Does 
not consult archive evidence; can accept any matching suffix. If the selected 
suffix was itself archived, the original mismatch remains. | Conservatively 
retain existing behavior when evidence is unavailable/incomplete. Initially 
support only prefix archival with active overlap. |
   | Compatibility | Changes the meaning of `timelinehash` for positive N; an 
older server does not understand N and may reject the request. | Keeps the 
original hash meaning and adds `firstinstantts`. Old clients/servers can 
continue the existing behavior, without gaining archival tolerance. |
   | Cost and implementation | Smaller patch; no archive I/O. Current 
implementation recomputes the client suffix hash per request. | More 
implementation work and a bounded archive lookup on the exceptional path. Reuse 
range-filtered `LoadMode.ACTION`; coalesce lookups and cache successful 
comparisons for a particular client snapshot and server view generation. |
   
   **The clearest correctness distinction is a stale server, not merely a newer 
server.** For N = 1:
   
   ```text
   Server cached: [commit001,            clean003]
   Client fresh:  [commit001, commit002, clean003]
                             ^ completed late
   ```
   
   Latest-N hashing compares only `clean003`, so the server can skip a 
necessary refresh and return its cached file from `001`. Full hashing detects 
the missing `002`. In this example the first instants are equal, so the 
proposed archival shortcut does not apply and the existing refresh path remains 
in force. `testRealViewRefreshesOutOfOrderCompletion` is a useful real-file 
starting point for testing this distinction.
   
   **Implementation conditions for the proposal:** reconstruction must be 
comparison-only; archive reads must normalize instant identities consistently; 
cache entries must be scoped to the relevant view generation and invalidated on 
refresh/recreation; and the post-handler check must re-evaluate equivalence if 
the view changes. A successful hash comparison establishes equivalence to the 
captured server view, not an independent guarantee that storage has not changed 
since that view was loaded. The proposal also does not extend data-file 
retention or make cleaned-away files readable.
   
   My preference is archived-prefix reconstruction if the goal is to serve 
remotely while preserving the existing full-history validation contract. 
Latest-N hashing is lighter, but its relaxation applies to non-archival 
differences too. Since archival mismatch is the exceptional path, a bounded, 
shared archive lookup seems a reasonable cost to evaluate before weakening that 
contract globally for enabled clients.
   
   This comparison is based on the code and proposed algorithm, not new 
end-to-end test results. Before accepting the reconstruction design, validate 
correct remote file selection with actual archival on both archive layouts, 
warm/cold views, late completion, compaction/clustering, restore, and 
concurrent view changes. Verify both initial and final checks, and verify that 
repeated requests do not repeat archive reads or syncs after successful 
validation.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to