etseidl commented on PR #11278:
URL: https://github.com/apache/arrow-rs/pull/11278#issuecomment-6279075415
Here's the latest:
Original blocker
Resolved. The former null-to-default corruption is fixed for:
- required REE schema with nullable dense input;
- required dense schema with nullable REE input;
- nested/list cases;
- parent-masked and unreferenced nulls;
- ordinary and CDC writes.
Remaining issue
A later invalid leaf partially mutates the row group.
At parquet/src/arrow/arrow_writer/mod.rs:1273-1299, buffered_rows is
incremented and leaves are written sequentially. If column 1 succeeds and
column 2 fails required-null validation:
- column 1 is already encoded;
- buffered_rows is already incremented;
- CDC state may already be advanced;
- reusing or closing the writer can fail with mismatched column lengths.
I reproduced this: after a rejected two-column batch, in_progress_rows()
was 2, not 0.
Mitigation verified
I tested a private helper that:
1. Computes all leaves into Vec<ArrowLeafColumn>.
2. Validates every leaf.
3. Only then increments buffered_rows and writes columns/chunkers.
A regression test covered ordinary and CDC modes, checked unchanged row
count and encoded size after failure, then successfully reused and closed the
writer with a valid batch.
Validation with the mitigation:
- 76 Arrow writer tests passed.
- 28 level-generation tests passed.
- Formatting and git diff --check passed.
The mitigation has no public API or lifetime issues, but temporarily
retains level data for the whole batch rather than one field at a time,
increasing peak memory. Also, row-group splitting can still commit valid
leading slices before discovering an invalid later slice; full write(batch)
transactional behavior would require preflighting the original unsplit batch.
The temporary worktree has been restored and is clean. My current verdict
is: the original corruption is fixed, but the row-group preflight issue should
be addressed before merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
