danny0405 commented on issue #20650: URL: https://github.com/apache/hudi/issues/20650#issuecomment-6847412966
Following up with a correctness-focused comparison of #20651 (head `300756f645b6963a63ef55ef4c728cecc841eda0`) and the [archived-prefix reconstruction proposal](https://github.com/apache/hudi/issues/20650#issuecomment-6847315815). The main difference is **ignoring older history versus verifying that the missing history moved into the archive**. Both can serve requests remotely in the reported archival case, but they establish different guarantees. | Aspect / scenario | #20651: latest-N hash, when enabled | Proposed: client first instant + archived-prefix reconstruction | | --- | --- | --- | | What is compared | Only the latest N instants, including their requested times, actions, and states. N = 0 retains the existing full hash. | The full client hash against the server active timeline plus archived identities in `[clientFirst, serverFirst)`, using consistent ordering/filtering. | | Pure prefix archival, optionally with one trailing clean | Can serve remotely if the selected suffix survives archival and matches. Requires choosing N. | Can serve remotely if the reconstructed full hash matches. No N to tune; the existing single-trailing-clean exception is retained. | | Server cache missed an older commit that the client has observed | Can incorrectly skip refresh if the difference is outside the latest N instants; final validation also passes. | The difference remains part of the comparison. Unless archive reconstruction genuinely explains it, retain the existing sync-and-validation path. | | Server has observed an older late completion that the client has not | Can accept it when outside the window. This changes the existing consistency contract, but a newer committed view is not by itself proof of incorrect writes. | Detects the additional instant, including in the reconstructed prefix, and preserves the existing mismatch handling. It does not introduce general acceptance of newer snapshots. | | Older compaction/clustering action or state changes | Can miss transitions outside the window, even though pending-operation state affects file selection. | No intentional blind window: retain action/state checking across the reconstructed history. Real-file tests are still needed to establish file-view behavior. | | Restore / rollback leaves extra or changed instants in the cached server view | Changes outside the selected suffix can be hidden. Changes inside it remain detectable. | Do not truncate the server's tail or remove non-archived differences to manufacture a match. Such differences retain the existing refresh path; restore regression tests remain required. | | Archive unavailable, holes in the timeline, or no active overlap | Does not consult archive evidence; can accept any matching suffix. If the selected suffix was itself archived, the original mismatch remains. | Conservatively retain existing behavior when evidence is unavailable/incomplete. Initially support only prefix archival with active overlap. | | Compatibility | Changes the meaning of `timelinehash` for positive N; an older server does not understand N and may reject the request. | Keeps the original hash meaning and adds `firstinstantts`. Old clients/servers can continue the existing behavior, without gaining archival tolerance. | | Cost and implementation | Smaller patch; no archive I/O. Current implementation recomputes the client suffix hash per request. | More implementation work and a bounded archive lookup on the exceptional path. Reuse range-filtered `LoadMode.ACTION`; coalesce lookups and cache successful comparisons for a particular client snapshot and server view generation. | **The clearest correctness distinction is a stale server, not merely a newer server.** For N = 1: ```text Server cached: [commit001, clean003] Client fresh: [commit001, commit002, clean003] ^ completed late ``` Latest-N hashing compares only `clean003`, so the server can skip a necessary refresh and return its cached file from `001`. Full hashing detects the missing `002`. In this example the first instants are equal, so the proposed archival shortcut does not apply and the existing refresh path remains in force. `testRealViewRefreshesOutOfOrderCompletion` is a useful real-file starting point for testing this distinction. **Implementation conditions for the proposal:** reconstruction must be comparison-only; archive reads must normalize instant identities consistently; cache entries must be scoped to the relevant view generation and invalidated on refresh/recreation; and the post-handler check must re-evaluate equivalence if the view changes. A successful hash comparison establishes equivalence to the captured server view, not an independent guarantee that storage has not changed since that view was loaded. The proposal also does not extend data-file retention or make cleaned-away files readable. My preference is archived-prefix reconstruction if the goal is to serve remotely while preserving the existing full-history validation contract. Latest-N hashing is lighter, but its relaxation applies to non-archival differences too. Since archival mismatch is the exceptional path, a bounded, shared archive lookup seems a reasonable cost to evaluate before weakening that contract globally for enabled clients. This comparison is based on the code and proposed algorithm, not new end-to-end test results. Before accepting the reconstruction design, validate correct remote file selection with actual archival on both archive layouts, warm/cold views, late completion, compaction/clustering, restore, and concurrent view changes. Verify both initial and final checks, and verify that repeated requests do not repeat archive reads or syncs after successful validation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
