linliu-code opened a new pull request, #678: URL: https://github.com/apache/hudi-rs/pull/678
## Why `forward_scan_pass1` has five gates, and Gate 3 checks whether a log block's instant actually completed. It never ran: `completion_gate_inputs` was `None` at every construction site this crate reaches, so the gate was inert and its docs read as if it were wired. Raised on #660 ([resolver.rs:89](https://github.com/apache/hudi-rs/pull/660#discussion_r3788329725), [again](https://github.com/apache/hudi-rs/pull/660#discussion_r3787843003)). The gap is the straddling case. A log file is admitted to a file slice on its own committed instant, but the blocks *inside* it carry their own instants. A writer that opens an instant and never finishes leaves blocks whose instant sorts below the latest committed one — so they clear the future gate and the range gate, and nothing asks whether that instant completed. Those rows merge as if committed. Not a regression: version 1 has no committed-set check either. But version 2 is the default now, `Table` already holds the timeline the gate needs, and the fix is cheap. ## What changed `Timeline::new_from_storage` already lists every active instant in every state to compute the archival boundary, then discards all but the completed ones. It now keeps the rest as `pending_instants`, and the gate inputs travel `Table` → `FileGroupReader` → adapter the same way the data schema already does. Callers that supply no timeline (the cxx bridge) pass `None` and the gate stays off — unchanged behavior. ## One correctness trap, called out because it fails silently The timeline lists an instant **once per state file**, so a completed instant also appears as requested and inflight. Partitioning on state alone marks every committed instant pending — the gate then rejects the entire timeline and reads degrade to base-file-only with **no error at all**. My first attempt did exactly this; the gold sweep caught it (`[1,2,3,4]` instead of `[3,4,11,5000000000]`). Pending is therefore the non-completed rows *minus* the completed set, pinned by `test_completion_gate_inputs_do_not_report_completed_instants_as_pending`. ## Tests - `test_pass1_admits_an_uncommitted_instant_without_the_completion_gate` — the reproduction: a gate-less scan admits an inflight instant's blocks, a gated scan skips them. - `test_table_reader_carries_the_completion_gate_inputs` — the wiring is actually populated, which is the thing that was missing. - `test_completion_gate_inputs_do_not_report_completed_instants_as_pending` — the trap above. Each was mutation-checked: broken on purpose once and observed to fail, then restored. ## Verification - `make check-rust` clean - `cargo clippy -p hudi-core --lib --no-default-features -- -D warnings` clean - 13/13 workspace targets green (1303 lib tests), 70 Python tests - gold parity sweep green ## Behavior changes Reads of a table with an inflight writer appending to an already-sliced log file will now exclude those uncommitted blocks instead of merging them. That is the intended fix. No config surface changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
