pingzh opened a new pull request, #7683: URL: https://github.com/apache/datafusion-comet/pull/7683
## Which issue does this PR close? Part of #5774. Follows the merged reader support (#7598) and Spark integration (#7657). ## Rationale for this change MIN/MAX reader filtering can skip later Parquet row groups after a task observes a useful bound. The benefit depends on data order and task boundaries, while footer inspection and bound updates add overhead. This adds an off/on benchmark and documents eligibility, fallback limits, and metrics before considering broader enablement. ## What changes are included in this PR? - `CometMinMaxBenchmark`: separate MIN and MAX queries over ascending, descending, deterministic random, and constant INT data, with one or four scan tasks by default. It checks Spark result parity, the direct scan/partial aggregate plan, actual task count, and per-task filter attachment before timing. - Scan and attachment metrics from untimed executions, plus elapsed query time in both configurable mode orders. Uncompressed Parquet v1 pages, disabled dictionaries, and disabled page-index/decoder filtering provide a controlled row-group comparison, including constant values. - User-guide configuration, supported plans, footer limits, fallback behavior, and metric interpretation; contributor instructions for reproducing measurements. The existing configuration remains disabled by default. Runtime behavior and supported scope are unchanged. The two commits separate the benchmark from documentation. ## How are these changes tested? - Native release build passed, with `-Ctarget-cpu=native -Clink-arg=-fuse-ld=bfd` and the standard release profile/default features. - JVM reactor `test-compile`, including formatting and style checks, passed. Packaged binary/source jar-content checks passed (the final check was rerun with JDK tools on PATH). - Both complete benchmark runs passed: 16 query/layout/task-count combinations × two modes × two mode orders. Results matched Spark, enabled filters attached once per scan task, and all pruning metrics agreed across mode orders. - `git diff --check` passed. ### Local release measurements Environment: AMD EPYC-Genoa, 32 logical CPUs, about 98 GiB RAM, Ubuntu 24.04.4/Linux 6.8; Spark 4.1.3, OpenJDK 21.0.13.0.3, Rust 1.99.1. Spark used `local[1]` and an 8 GiB benchmark JVM heap. Production code is from `d187e6823`; the benchmark is in this PR's `d2f9fdfc6` commit. Each fixture contained 8,388,608 rows and nine row groups per file. Four scan tasks ran serially under `local[1]`. Equivalent standard invocations after the same release build: ```sh COMET_BENCHMARK_MASTER='local[1]' make BENCH_HEAP=8g benchmark-org.apache.spark.sql.benchmark.CometMinMaxBenchmark -- 8388608 1,4 ascending,descending,random,constant 5 off,on COMET_BENCHMARK_MASTER='local[1]' make BENCH_HEAP=8g benchmark-org.apache.spark.sql.benchmark.CometMinMaxBenchmark -- 8388608 1,4 ascending,descending,random,constant 5 on,off ``` Favorable ordering and constant values reduced emitted scan rows and requested data bytes by about 87.5%. The unfavorable ordered cases pruned no groups and expose filtering overhead. Random layouts pruned some groups but did not consistently improve elapsed time. These short local runs are sensitive to warmup and long-tail outliers; the first ascending-MIN timing even changes direction when case order reverses. This is evidence of scan-work reduction and overhead cases, not a general speedup claim or a basis to enable the feature by default. <details> <summary>All elapsed times and scan-work counters</summary> Times are mean ± standard deviation in milliseconds, always shown as **off → on**, regardless of execution order. Every off execution emitted 8,388,608 rows and pruned zero groups. The byte column is `bytes_scanned` in bytes, excluding metadata. All on attachment counts equaled the scan-task count; skipped counts were zero. Statistics, page-index, and decoder pruning counters were zero. | Tasks | Layout | Aggregate | On scan rows | Groups pruned | Data bytes off → on | Mean ± SD, off-first | Mean ± SD, on-first | | ---: | --- | --- | ---: | ---: | --- | --- | --- | | 1 | ascending | MIN | 1,048,287 | 8 | 33,566,729 → 4,194,680 | 176 ± 21 → 101 ± 22 | 77 ± 10 → 135 ± 96 | | 1 | ascending | MAX | 8,388,608 | 0 | 33,566,729 → 33,566,729 | 105 ± 4 → 98 ± 5 | 81 ± 3 → 96 ± 7 | | 1 | descending | MIN | 8,388,608 | 0 | 33,566,728 → 33,566,728 | 73 ± 4 → 84 ± 6 | 69 ± 3 → 68 ± 6 | | 1 | descending | MAX | 1,048,287 | 8 | 33,566,728 → 4,194,682 | 74 ± 5 → 48 ± 5 | 87 ± 12 → 44 ± 3 | | 1 | random | MIN | 3,144,861 | 6 | 33,570,132 → 12,585,317 | 63 ± 4 → 67 ± 10 | 62 ± 4 → 54 ± 6 | | 1 | random | MAX | 5,241,435 | 4 | 33,570,132 → 20,975,530 | 62 ± 2 → 62 ± 1 | 65 ± 3 → 63 ± 3 | | 1 | constant | MIN | 1,048,287 | 8 | 33,566,756 → 4,194,685 | 53 ± 9 → 102 ± 152 | 60 ± 4 → 42 ± 8 | | 1 | constant | MAX | 1,048,287 | 8 | 33,566,756 → 4,194,685 | 56 ± 4 → 59 ± 14 | 64 ± 9 → 48 ± 1 | | 4 | ascending | MIN | 1,048,196 | 32 | 33,567,471 → 4,194,401 | 78 ± 3 → 73 ± 15 | 76 ± 1 → 66 ± 3 | | 4 | ascending | MAX | 8,388,608 | 0 | 33,567,471 → 33,567,471 | 70 ± 3 → 87 ± 18 | 79 ± 5 → 107 ± 21 | | 4 | descending | MIN | 8,388,608 | 0 | 33,567,469 → 33,567,469 | 73 ± 2 → 88 ± 15 | 79 ± 2 → 94 ± 5 | | 4 | descending | MAX | 1,048,196 | 32 | 33,567,469 → 4,194,400 | 59 ± 2 → 69 ± 17 | 74 ± 4 → 68 ± 15 | | 4 | random | MIN | 3,144,588 | 24 | 33,571,051 → 12,584,536 | 61 ± 2 → 72 ± 17 | 61 ± 3 → 64 ± 2 | | 4 | random | MAX | 3,406,637 | 23 | 33,571,051 → 13,633,247 | 63 ± 4 → 179 ± 196 | 67 ± 2 → 125 ± 46 | | 4 | constant | MIN | 1,048,196 | 32 | 33,567,496 → 4,194,404 | 75 ± 3 → 64 ± 14 | 95 ± 9 → 50 ± 14 | | 4 | constant | MAX | 1,048,196 | 32 | 33,567,496 → 4,194,404 | 68 ± 2 → 58 ± 1 | 81 ± 11 → 68 ± 13 | Reader `scan_io_data_bytes` equaled `bytes_scanned`. `scan_io_metadata_bytes` was 524,288 bytes per task in each mode (524,288 for one task and 2,097,152 for four); attachment does not eliminate footer inspection. Timings include planning and result collection, while fixture generation, Spark parity checks, plan validation, and metric reporting are outside the timed cases. </details> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
