peterxcli opened a new issue, #25239: URL: https://github.com/apache/datafusion/issues/25239
### Describe the bug Parquet runtime row-group pruning treats an absent `null_count` as zero. This can incorrectly discard nulls needed by a TopK with `NULLS FIRST`. The assumption is present in [DataFusion 55.1.0's `RowGroupPruner::should_prune`](https://github.com/apache/datafusion/blob/55.1.0/datafusion/datasource-parquet/src/push_decoder.rs#L212-L225), and remains on [current main](https://github.com/apache/datafusion/blob/main/datafusion/datasource-parquet/src/push_decoder.rs#L247-L265). It sets `missing_null_counts_as_zero: true`, which the [statistics adapter](https://github.com/apache/datafusion/blob/55.1.0/datafusion/datasource-parquet/src/row_group_filter.rs#L477-L509) passes to Parquet's `StatisticsConverter`. The [Parquet specification](https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L283-L291) requires an absent null count to remain distinct from a known zero. ### Concrete case to reproduce Use one file and one scan partition, with a nullable INT column `k` and these row groups in scan order: | Row group | Values | Column-chunk statistics | | --- | --- | --- | | 0 | `0, 1, 2` | Ordinary statistics | | 1 | `NULL, 100, 101` | `min=100`, `max=101`, but `null_count` absent | | 2 | `200, 201, 202` | Ordinary statistics | Keep min/max statistics on group 1 and remove only its optional null-count field. Disabling all statistics is a different case. Run: ```sql SELECT k FROM t ORDER BY k ASC NULLS FIRST LIMIT 1; ``` Connect TopK's live predicate to the Parquet scan, use small batches, and disable page-index and row-filter pushdown to isolate runtime row-group pruning. After the first group, TopK retains `0` and [publishes](https://github.com/apache/datafusion/blob/55.1.0/datafusion/physical-plan/src/topk/mod.rs#L605-L635): ```sql k IS NULL OR k < 0 ``` For group 1, treating the unknown null count as zero makes `IS NULL` appear impossible, while `min=100` rules out `k < 0`. The group can therefore be pruned even though its null must win. The expected result is `NULL`; source tracing predicts `0` if that group is skipped. ### Expected behavior and fix direction Preserve unknown null counts during pruning. A group must remain readable when its statistics cannot rule out matching nulls. Review the other callers that enable the same assumption, including static statistics pruning. Add a regression with min/max present and null count omitted, covering TopK filtering off/on and both ascending and descending `NULLS FIRST`. Keep a known-zero-null-count control to verify that valid pruning remains effective. ### Validation and downstream context This is a source-derived report against DataFusion 55.1.0 and the current main implementation. An isolated native pruning probe compiled, but its process was killed before producing output. I have not completed an end-to-end reproduction, so the predicted wrong result above is not a measured test result. Found while reviewing [apache/datafusion-comet#5785](https://github.com/apache/datafusion-comet/pull/5785), which connects Spark's local TopK to this reader path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
