danny0405 commented on issue #20650: URL: https://github.com/apache/hudi/issues/20650#issuecomment-6907108922
@nsivabalan On the [previous-snapshot proposal](https://github.com/apache/hudi/issues/20650#issuecomment-6860215221): retaining a previous snapshot could avoid archive reads for an important subset of requests. I think the equivalence claim needs one more case covered, though: **a late completion followed by archival of that instant before the next observed snapshot**. For example, with enough retained commits to satisfy the configured retention policy (timestamps below are schematic): ```text Previous/client hashed timeline: [001, 003, 004] 002 exists but is inflight 002 completes; archival subsequently removes 001 and 002. Current server timeline: [003, 004, clean005] ``` Ordinary inflight commits are excluded from the remote client's `filterCompletedAndCompactionInstants()` timeline. Thus monotonically increasing instant creation does not prevent this ordering. The proposed predicate evaluates to true: filtering the previous timeline from `003` and appending `clean005` produces exactly the current timeline. However, the transition included completion of `002`, not just archival of the previously observed completed instants. Reconstructing the archived prefix would include `002` and produce a different full hash. **This is a counterexample to the claim that the predicate establishes the same equivalence as archive reconstruction, not an end-to-end data-corruption reproduction.** A newer committed server view can be usable by a writer; accepting it needs a separate argument about the affected write/concurrency paths. I checked the predicate with a small hash calculation, but have not run a real-file reproduction of this sequence. There is a related case for the pending-compaction argument: an instant pending in the previous snapshot can complete and then become eligible for archival before the next snapshot. “Do not archive currently pending compactions” does not by itself rule out that transition. Please add both complete-then-archive cases to the proposed real-file tests; the current overlap-only late-completion test would not exercise them. A few implementation details also need explicit rules: - **Cold-start registration:** it needs to register the exact snapshot used by the executors and validate its relationship to the newly initialized server view. Merely storing the supplied hash as `prevHash` is not enough. This avoids an HTTP protocol change, but the registration lifecycle is essential to fixing the reported cold-start race. - **No-op reloads and multiple clients:** replacing `prevHash` on every reload can discard a useful compatibility record even when the current timeline is unchanged. Another client's registration can also displace the single slot. These are optimization-coverage issues rather than evidence of incorrect answers; please define the replacement policy and test it. - **View lifecycle and concurrency:** scope the compatibility record to the actual view generation. Explicit refresh currently removes/closes the view. Locking the fields makes each check coherent, but separate pre/post checks do not by themselves pin the generation used during handling; the final check needs to validate the generation/compatibility of the view used to produce the response. - **Restart wording:** a new host/port is not guaranteed: the embedded server port is configurable, and an automatically selected port can be reused. Clearing the compatibility state and validating arriving requests is the right safety argument; it should not depend on old requests being unable to reach the endpoint. I would keep this approach under consideration, but either add evidence/guards that distinguish the complete-then-archive transitions, or explicitly justify accepting those newer views. Until then, I would describe the suffix comparison as a candidate archival transition rather than proof that archival was the only change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
