pingzh commented on issue #5774: URL: https://github.com/apache/datafusion-comet/issues/5774#issuecomment-6782151492
I plan to implement this in three small, dependent PRs: 1. **Native MIN/MAX reader filtering.** Add an internal `MinMaxReaderFilterExec` that creates a fresh aggregate and live predicate for each execution and attaches that predicate to the existing native Parquet reader. Include attachment metrics and core native correctness, pruning, and execution-lifecycle tests. Spark planner wiring is deferred to PR 2. 2. **Spark integration, disabled by default.** Add `spark.comet.exec.aggregate.dynamicFilter.enabled`, capture it in the serialized aggregate plan, wire the native wrapper into eligible aggregate planning, and expose attachment metrics in Spark. Check filtering off/on against Spark, including multiple Spark partitions and unsupported-plan fallback. 3. **Benchmark and documentation.** Compare favorable and unfavorable layouts for MIN and MAX, recording scan rows, row groups pruned, bytes scanned, and elapsed query time. Document configuration and supported plans; keep the default disabled. The first version will support one unfiltered MIN or MAX over a direct signed-integer column, with no grouping, in raw-input Partial mode, reading an eligible native Parquet scan. Bounds remain local to each execution. Mixed aggregates, Final/PartialMerge stages, intervening expressions/filters/limits, and JVM/exchange boundaries remain ineligible. Reader-internal column mapping can use the existing attachment logic. The native lifecycle will follow the existing TopK reader-filter wrapper while retaining DataFusion's aggregate bound updates. Each execution must reconstruct the aggregate as well as its predicate, because cloning or replacing only the predicate can retain shared accumulator-bound state. Aggregate metrics remain with the aggregate and pruning/byte metrics remain with the scan. Core checks will cover empty/all-null input, duplicates and negative values, missing null-count statistics, repeated execution, and actual reader pruning. Later scope expansions can be separate PRs after this initial path is validated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
