mhatipog opened a new issue, #27456:
URL: https://github.com/apache/datafusion/issues/27456
`PartitionedFile::with_statistics` sets exact `min = max = partition_value`
for partition columns regardless of `num_rows`. With a 0-row file in a
partition, the aggregate-statistics rule answers MIN/MAX from those statistics
without reading data:
```python
import os, pyarrow as pa, pyarrow.parquet as pq, datafusion
for d, rows in (("q=1", [1]), ("q=7", [])):
os.makedirs(f"/tmp/x2/{d}"); pq.write_table(pa.table({"k":
pa.array(rows, pa.int32())}), f"/tmp/x2/{d}/f.parquet")
ctx = datafusion.SessionContext()
ctx.register_parquet("t", "/tmp/x2", table_partition_cols=[("q",
pa.int32())], file_extension=".parquet")
ctx.sql("SELECT count(*), min(q), max(q) FROM t").show() # 1, 1, 7 —
expected 1, 1, 1
```
Correct with `datafusion.execution.collect_statistics = false` or with
`GROUP BY q`. Empty files in partitions are common (empty tasks, filtered
writes).
DataFusion 55.0.0. On main, #26310 changed the NULL-partition case, but a
non-null value still gets exact min/max when `num_rows` is `Exact(0)`. Those
could be `Absent` for empty files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]