jayzhan211 commented on PR #24273: URL: https://github.com/apache/datafusion/pull/24273#issuecomment-6225935752
Thanks @Theodus for tracking this down. While looking at the same code I found a more severe variant of the same root cause, and I opened #26434 with a broader fix. I'd like your view on which way to go. With default settings (no filter, `pushdown_filters` off), `ORDER BY ... LIMIT` over a `WITH ORDER` parquet table can return the wrong rows. With 8 sorted files plus one backfill file holding the smallest values and `target_partitions = 8`, `SELECT a FROM t ORDER BY a LIMIT 3` returns `0, 1, 2` while `min(a)` is `-100000`. The cause is the same `preserve_order = false` that `LimitPushdown` pushes into the scan: besides limit pruning, it also enables file work stealing, so a partition can read a higher file before a lower one. #26434 makes the scan the owner of that decision: a scan that advertises an output ordering always reads its files in order (`FileScanConfig::preserves_file_order()`). The `required_ordering` / `with_preserve_order` plumbing between `EnforceSorting`, `LimitPushdown` and the scan is no longer needed, so it is deprecated. That also covers the case in #24215. I adapted your row-group repro idea for the slt in #26434. If you agree with this direction, would you be OK with closing this PR in favor of #26434? Happy to add you as a co-author there. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
