mightsleep opened a new pull request, #11828: URL: https://github.com/apache/arrow-rs/pull/11828
# Which issue does this PR close? N/A, found while checking level histograms against the page index for nested columns. # Rationale for this change The writer marks a null page when `num_buffered_rows == num_page_nulls`, but `num_page_nulls` counts levels, not rows. For repeated columns this marks pages like `[null, 7]` per row as null pages with empty min/max, and drops the whole column index for pages like `[null, null]`. DataFusion does not prune nested columns yet, but pages always start at a record boundary, so pruning e.g. `array_has` on list elements is possible and would rely on `null_pages` being right. # What changes are included in this PR? Compare `num_page_nulls` with the number of levels (`num_buffered_values`). Non-repeated columns are unchanged. # Are these changes tested? Two new tests for both cases; both fail without the fix. # Are there any user-facing changes? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
