danny0405 commented on issue #20650:
URL: https://github.com/apache/hudi/issues/20650#issuecomment-6907108922

   @nsivabalan On the [previous-snapshot 
proposal](https://github.com/apache/hudi/issues/20650#issuecomment-6860215221): 
retaining a previous snapshot could avoid archive reads for an important subset 
of requests. I think the equivalence claim needs one more case covered, though: 
**a late completion followed by archival of that instant before the next 
observed snapshot**.
   
   For example, with enough retained commits to satisfy the configured 
retention policy (timestamps below are schematic):
   
   ```text
   Previous/client hashed timeline: [001,      003, 004]
                                   002 exists but is inflight
   
   002 completes; archival subsequently removes 001 and 002.
   
   Current server timeline:                  [003, 004, clean005]
   ```
   
   Ordinary inflight commits are excluded from the remote client's 
`filterCompletedAndCompactionInstants()` timeline. Thus monotonically 
increasing instant creation does not prevent this ordering.
   
   The proposed predicate evaluates to true: filtering the previous timeline 
from `003` and appending `clean005` produces exactly the current timeline. 
However, the transition included completion of `002`, not just archival of the 
previously observed completed instants. Reconstructing the archived prefix 
would include `002` and produce a different full hash.
   
   **This is a counterexample to the claim that the predicate establishes the 
same equivalence as archive reconstruction, not an end-to-end data-corruption 
reproduction.** A newer committed server view can be usable by a writer; 
accepting it needs a separate argument about the affected write/concurrency 
paths. I checked the predicate with a small hash calculation, but have not run 
a real-file reproduction of this sequence.
   
   There is a related case for the pending-compaction argument: an instant 
pending in the previous snapshot can complete and then become eligible for 
archival before the next snapshot. “Do not archive currently pending 
compactions” does not by itself rule out that transition. Please add both 
complete-then-archive cases to the proposed real-file tests; the current 
overlap-only late-completion test would not exercise them.
   
   A few implementation details also need explicit rules:
   
   - **Cold-start registration:** it needs to register the exact snapshot used 
by the executors and validate its relationship to the newly initialized server 
view. Merely storing the supplied hash as `prevHash` is not enough. This avoids 
an HTTP protocol change, but the registration lifecycle is essential to fixing 
the reported cold-start race.
   - **No-op reloads and multiple clients:** replacing `prevHash` on every 
reload can discard a useful compatibility record even when the current timeline 
is unchanged. Another client's registration can also displace the single slot. 
These are optimization-coverage issues rather than evidence of incorrect 
answers; please define the replacement policy and test it.
   - **View lifecycle and concurrency:** scope the compatibility record to the 
actual view generation. Explicit refresh currently removes/closes the view. 
Locking the fields makes each check coherent, but separate pre/post checks do 
not by themselves pin the generation used during handling; the final check 
needs to validate the generation/compatibility of the view used to produce the 
response.
   - **Restart wording:** a new host/port is not guaranteed: the embedded 
server port is configurable, and an automatically selected port can be reused. 
Clearing the compatibility state and validating arriving requests is the right 
safety argument; it should not depend on old requests being unable to reach the 
endpoint.
   
   I would keep this approach under consideration, but either add 
evidence/guards that distinguish the complete-then-archive transitions, or 
explicitly justify accepting those newer views. Until then, I would describe 
the suffix comparison as a candidate archival transition rather than proof that 
archival was the only change.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to