danny0405 commented on issue #20650: URL: https://github.com/apache/hudi/issues/20650#issuecomment-6847315815
A possible fix is to reconstruct the missing archived prefix on the server and keep the existing full-timeline hash comparison. **Proposed protocol and comparison** 1. Add `firstinstantts` to remote-view requests, taken from the same captured, filtered client timeline used to compute `timelinehash`. Keep the existing full hash and `lastinstantts`. 2. Keep the existing exact-match and trailing-clean fast paths. If they fail and the client's first instant precedes the server's first instant, with overlap remaining in the active timeline, load archived instant identities in the half-open range `[clientFirstInstant, serverFirstInstant)`. 3. Prepend those archived instants to the server's active timeline for comparison only, using the same ordering, filtering, and instant representation as the full-hash check. 4. If the reconstructed timeline's hash matches the client's hash, optionally applying the existing single-trailing-clean exception, serve the request remotely without syncing the view. Otherwise, retain the existing sync-and-validation path. For example: ```text Client snapshot: [001, 002, 003] hash H Server active timeline: [002, 003, clean004] Archived prefix loaded: [001] Reconstructed comparison: [001, 002, 003, clean004] After clean exception: [001, 002, 003] hash H -> serve remotely ``` The reconstructed timeline is only used to establish equivalence; it does not replace the filesystem view's active timeline. In particular, do not truncate the server's tail at the client's last instant to force a match. Extra data commits or action/state differences should continue to prevent acceptance. Compared with hashing only the latest N instants in #20651, this retains checking of the discarded prefix: an older instant that completed after the client captured its timeline would appear in the reconstructed history and prevent a match. It also needs no client/server negotiation round trip or choice of N. **Scope and performance** - Hudi already has range-filtered archive loading and `LoadMode.ACTION`, so this should read instant identities rather than full commit metadata. - Cache successful equivalence checks for the client snapshot and the particular server view generation. Invalidate on view changes/recreation or explicit refresh, and use the same equivalence predicate in the post-handler consistency check. Concurrent requests sharing a snapshot should share the archive lookup. - Preserve existing behavior when the new parameter is absent or archive evidence is unavailable/incomplete. Initially scope this to ordinary prefix archival with active-timeline overlap; timelines with holes and entirely archived client snapshots can retain the existing path. This is a design proposal, not an implemented or tested fix. Validation should include real-file archival before lazy server initialization, archival plus a trailing clean, repeated concurrent requests returning correct remote results without fallback or repeated sync/archive reads, late completion, compaction/clustering transitions, and restore/deleted-instant recovery across the supported archive layouts. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
