AnuragRaut08 opened a new pull request, #51357: URL: https://github.com/apache/arrow/pull/51357
### Rationale for this change Parquet null-count statistics can be incorrect for fixed-width leaf columns nested below a repeated ancestor, such as `list<struct<...>>`. When a list is null or empty, its descendant leaf does not produce a value in the leaf values buffer. `DefLevelsToBitmap` intentionally excludes these repeated-ancestor nulls from its `null_count`. `MaybeCalculateValidityBits` was using that value directly for the Parquet column statistics, causing the null count to be undercounted. For example, a `list<struct<string, int32>>` column can report an incorrect `null_count` for the `int32` leaf even though the encoded definition levels correctly represent the null and empty list entries. ### What changes are included in this PR? * Update `MaybeCalculateValidityBits` to calculate the total null count from `batch_size - out_values_to_write`. * This includes nulls represented by repeated ancestors, such as null or empty lists, while preserving the existing validity bitmap behavior. * Add a regression test covering a fixed-width `int32` leaf under `list<struct<...>>`. ### Are these changes tested? Yes. Added a regression test that verifies the `int32` leaf reports: * `null_count = 3` * `num_values = 2` The focused regression test passes: `StatisticsTest.FixedWidthLeafUnderListStructNullCount` The existing `parquet-writer-test` also passes. ### Are there any user-facing changes? Yes. This fixes incorrect Parquet column statistics for affected nested fixed-width columns. The change does not alter the encoded data or public APIs. ### This PR contains a "Critical Fix". This fixes a bug that produces incorrect Parquet statistics. The underlying data remains correct, but the reported `null_count` for affected fixed-width leaf columns can be incorrect. ### Was AI used for this PR? **PR code and description written by:** * [x] Human * [ ] AI **Reviewed before submission by:** * [x] Human * [ ] AI * [ ] Not reviewed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
