alamb commented on code in PR #9374:
URL: https://github.com/apache/arrow-rs/pull/9374#discussion_r2811993541
##########
parquet/src/column/reader.rs:
##########
@@ -309,6 +309,20 @@ where
});
if let Some(rows) = rows {
+ // If there is a pending partial record from a previous
page,
+ // count it before considering the whole-page skip. When
the
+ // next page provides num_rows (e.g. a V2 data page or via
+ // offset index), its records are self-contained, so the
+ // partial from the previous page is complete at this
boundary.
+ if let Some(decoder) = self.rep_level_decoder.as_mut() {
+ if decoder.flush_partial() {
Review Comment:
> Datapoint: The file that lead to the original error message was written
with arrow-rs version 57.1.0: parquet viewer and more verbose debug dumps.
Given that we can reproduce this issue with the current parquet writer (e.g.
`test_list_struct_page_boundary_desync_produces_length_mismatch`) it would
imply this is still an issue
> In any case, at least at my company we probably have a few PiB of data
written with this or an even earlier version.
BTW this is so cool to hear
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
