Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
alamb commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4481207976 This is pretty epic -- thank you @zhuqi-lucas and @adriangb -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
zhuqi-lucas merged PR #21956: URL: https://github.com/apache/datafusion/pull/21956 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
zhuqi-lucas commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4469377563 Thanks again for @adriangb and @alamb for review, merged now! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
zhuqi-lucas commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4469284541 > @zhuqi-lucas let me know if you’d like to merge this, if I should, etc Thank you @adriangb , i resolving conflicts and merging it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
adriangb commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4469257858 @zhuqi-lucas let me know if you’d like to merge this, if I should, etc -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
adriangb commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4468562373 Makes sense. I think we should merge this as is then. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
zhuqi-lucas commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4466266314 > [#21956 (comment)](https://github.com/apache/datafusion/pull/21956#issuecomment-4458821315) > > @zhuqi-lucas these benches are good but not all that impressive. I'm not sure if it's just that they are already trivially fast (<10ms) or if we need to tweak them. Yeah @adriangb , this is expected: The wins are bounded by what `Inexact` + `TopK` (and reverse-`TopK`) can do — there's an inherent ceiling, and this PR pushes against it rather than breaking past it. Concretely: in the `Inexact` path, `LIMIT N` doesn't propagate as a static stop signal to the source. The dynamic-filter pushdown can stats-prune *subsequent* row groups once the heap-driven threshold tightens, which already does a lot of work — but inside the row group the source is currently reading, the sort column still has to be **fully decoded** so the filter can be evaluated row by row. **Phase 3's stats reorder this PR** makes that threshold tighten on the first RG instead of after a few, but the per-RG cost floor stays. That's the ceiling. Internally we still keep our own RG-level `Exact` reverse running in production — it's net faster end-to-end on this shape because deleting `SortExec` turns `LIMIT` into a static fetch on the source and removes the heap / per-row filter / final-ordering overhead. We can't land RG-level `Exact` upstream though: parquet doesn't allow partial row-group reads, so it inherits the same ~128 MB peak that closed #18817. The fix is **page-level `Exact` reverse via apache/arrow-rs#9937**. Seek to the last page via `OffsetIndex`, decode forward, reverse the batch, emit — peak memory drops to one page (~1 MB), and the static-fetch early termination is preserved at page granularity. Once that primitive ships, the DataFusion-side integration can replace the `Inexact` + `TopK` shape for `ORDER BY ts DESC LIMIT N` outright, and the ceiling we're hitting today goes away — without the RG-level memory regression. Separately for the forward `Inexact` path (**Phase 3 stats reorder, not reverse-specific this PR**): if we get a **row-group-granular morsel API** later — the shared work queue (#21351) currently holds `PartitionedFile`, but if it held `(file, rg_idx)` instead, an idle partition could release its current file's remaining RGs back to the pool once a global "K satisfied" signal fires — Phase 3's reorder path itself would get true cross-partition early termination too. That removes the "currently-decoding RG must finish" floor on the `Inexact` side as well. Not in this PR, clean follow-up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
adriangbot commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4466252446 🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21956#issuecomment-4466211013) **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) CPU Details (lscpu) ``` Architecture:aarch64 CPU op-mode(s): 64-bit Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Vendor ID: ARM Model name: Neoverse-V2 Model: 1 Thread(s) per core: 1 Core(s) per cluster: 16 Socket(s): - Cluster(s): 1 Stepping:r0p1 BogoMIPS:2000.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti L1d cache: 1 MiB (16 instances) L1i cache: 1 MiB (16 instances) L2 cache:32 MiB (16 instances) L3 cache:80 MiB (1 instance) NUMA node(s):1 NUMA node0 CPU(s): 0-15 Vulnerability Gather data sampling: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling:Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1:Mitigation; __user pointer sanitization Vulnerability Spectre v2:Mitigation; CSV2, BHB Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Not affected ``` Details ``` Comparing HEAD and feat_rg-reorder-by-statistics Benchmark sort_pushdown_inexact_unsorted.json ┏━━━┳━━┳━━━┳━━┓ ┃ Query ┃ HEAD ┃ feat_rg-reorder-by-statistics ┃ Change ┃ ┡━━━╇━━╇━━━╇━━┩ │ Q1│ FAIL │ FAIL │ incomparable │ │ Q2│ FAIL │ FAIL │ incomparable │ │ Q3│ FAIL │ FAIL │ incomparable │ │ Q4│ FAIL │ FAIL │ incomparable │ │ Q5│ FAIL │ FAIL │ incomparable │ │ Q6│ FAIL │ FAIL │ incomparable │ └───┴──┴───┴──┘ ┏━━┳┓ ┃ Benchmark Summary┃┃ ┡━━╇┩ │ Total Time (HEAD)│ 0.00ms │ │ Total Time (feat_rg-reorder-by-statistics) │ 0.00ms │ │ Average Time (HEAD) │ 0.00ms │ │ Average Time (feat_rg-reorder-by-statistics) │ 0.00ms │ │ Queries Faster │ 0 │ │ Queries Slower │ 0 │ │ Queries with No Change │ 0 │ │ Queries with Failure │ 6 │ └──┴┘ ``` Resource Usage **sort_pushdown_inexact_unsorted — base (merge-base)** | Metric | Value | ||---| | Wall time | 5.0s | | Peak memory | 4.1 GiB | | Avg memory | 4.1 GiB | | CPU user | 0.2s | | CPU sys | 0.1s | | Peak spill | 0 B | **sort_pushdown_inexact_unsorted — branch** | Metric | Value | ||---| | Wall time | 5.0s | | Peak memory | 4.1 GiB | | Avg memory | 4.1 GiB | | CPU user | 0.2s | | CPU sys | 0.1s | | Peak spill | 0 B | --- [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) a
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
adriangbot commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4466217317 🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21956#issuecomment-4466211013) **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4466211013-164-6nrzn 6.12.68+ #1 SMP Wed Apr 1 02:23:28 UTC 2026 aarch64 GNU/Linux` CPU Details (lscpu) ``` Architecture:aarch64 CPU op-mode(s): 64-bit Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Vendor ID: ARM Model name: Neoverse-V2 Model: 1 Thread(s) per core: 1 Core(s) per cluster: 16 Socket(s): - Cluster(s): 1 Stepping:r0p1 BogoMIPS:2000.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti L1d cache: 1 MiB (16 instances) L1i cache: 1 MiB (16 instances) L2 cache:32 MiB (16 instances) L3 cache:80 MiB (1 instance) NUMA node(s):1 NUMA node0 CPU(s): 0-15 Vulnerability Gather data sampling: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling:Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1:Mitigation; __user pointer sanitization Vulnerability Spectre v2:Mitigation; CSV2, BHB Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Not affected ``` Comparing feat/rg-reorder-by-statistics (7c5e5fb8ba83b37b09725548b1c11f7da6c4c11d) to 1ab146a (merge-base) [diff](https://github.com/apache/datafusion/compare/1ab146ad6cc119c7656ae1def75fd40697e5f94a..7c5e5fb8ba83b37b09725548b1c11f7da6c4c11d) using: sort_pushdown_inexact_unsorted Results will be posted here when complete --- [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
zhuqi-lucas commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4466211013 run benchmark sort_pushdown_inexact_unsorted -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
zhuqi-lucas commented on code in PR #21956:
URL: https://github.com/apache/datafusion/pull/21956#discussion_r3252443486
##
datafusion/datasource-parquet/src/source.rs:
##
@@ -581,9 +615,63 @@ impl FileSource for ParquetSource {
encryption_factory: self.get_encryption_factory_with_config(),
max_predicate_cache_size: self.max_predicate_cache_size(),
reverse_row_groups: self.reverse_row_groups,
+sort_order_for_reorder: self.sort_order_for_reorder.clone(),
}))
}
+/// Reorder the files in the shared work queue so the most
+/// "promising" files are read first, matching the strategy of
+/// `PreparedAccessPlan::reorder_by_statistics` at the row-group
+/// level: key off the file's `min`, and let the sort direction
+/// follow the request (ASC by `min` for ASC requests, DESC by
+/// `min` for DESC requests).
+///
+/// Keeping both layers consistent matters because they share the
+/// same convergence story for TopK's dynamic filter: file `i`'s
+/// `min` is a lower bound on every row group inside it, so the
+/// order chosen here is a natural prefix of the order
+/// `reorder_by_statistics` will produce within each file.
+///
+/// Files missing statistics sort to the end so present-stats
+/// files run first.
+fn reorder_files(
Review Comment:
Thanks @adriangb good point, addressed in latest PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
zhuqi-lucas commented on code in PR #21956:
URL: https://github.com/apache/datafusion/pull/21956#discussion_r3252432620
##
datafusion/core/tests/dataframe/mod.rs:
##
@@ -3268,7 +3268,7 @@ async fn
union_with_mix_of_presorted_and_explicitly_resorted_inputs_with_reparti
UnionExec
DataSourceExec: file_groups={1 group:
[[{testdata}/alltypes_tiny_pages.parquet]]}, projection=[id],
output_ordering=[id@0 ASC NULLS LAST], file_type=parquet
SortExec: expr=[id@0 ASC NULLS LAST], preserve_partitioning=[false]
- DataSourceExec: file_groups={1 group:
[[{testdata}/alltypes_tiny_pages.parquet]]}, projection=[id], file_type=parquet
+ DataSourceExec: file_groups={1 group:
[[{testdata}/alltypes_tiny_pages.parquet]]}, projection=[id],
file_type=parquet, sort_order_for_reorder=[id@0 ASC NULLS LAST]
Review Comment:
Thanks for review! Thought about it — kept `sort_order_for_reorder` rather
than going to `inexact_output_ordering`. The latter parallels the existing
`output_ordering` (which is a contract downstream relies on), but this field is
a runtime *instruction* to the opener — we explicitly drop the source's
`output_ordering` when this fires precisely because the runtime reorder
invalidates the contract. Keeping the "…for_reorder" naming makes the
instruction-vs-claim distinction explicit and pairs cleanly with
`reverse_row_groups` (also an instruction).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
adriangb commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4462386530 https://github.com/apache/datafusion/pull/21956#issuecomment-4458821315 @zhuqi-lucas these benches are good but not all that impressive. I'm not sure if it's just that they are already trivially fast (<10ms) or if we need to tweak them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
adriangb commented on code in PR #21956:
URL: https://github.com/apache/datafusion/pull/21956#discussion_r3250229819
##
datafusion/datasource-parquet/src/source.rs:
##
@@ -581,9 +615,63 @@ impl FileSource for ParquetSource {
encryption_factory: self.get_encryption_factory_with_config(),
max_predicate_cache_size: self.max_predicate_cache_size(),
reverse_row_groups: self.reverse_row_groups,
+sort_order_for_reorder: self.sort_order_for_reorder.clone(),
}))
}
+/// Reorder the files in the shared work queue so the most
+/// "promising" files are read first, matching the strategy of
+/// `PreparedAccessPlan::reorder_by_statistics` at the row-group
+/// level: key off the file's `min`, and let the sort direction
+/// follow the request (ASC by `min` for ASC requests, DESC by
+/// `min` for DESC requests).
+///
+/// Keeping both layers consistent matters because they share the
+/// same convergence story for TopK's dynamic filter: file `i`'s
+/// `min` is a lower bound on every row group inside it, so the
+/// order chosen here is a natural prefix of the order
+/// `reorder_by_statistics` will produce within each file.
+///
+/// Files missing statistics sort to the end so present-stats
+/// files run first.
+fn reorder_files(
Review Comment:
Not required in this PR but I wonder if we could move some of these helpers
out into a `sort.rs` module in the same package or something to keep
`source.rs` simpler.
##
datafusion/core/tests/dataframe/mod.rs:
##
@@ -3268,7 +3268,7 @@ async fn
union_with_mix_of_presorted_and_explicitly_resorted_inputs_with_reparti
UnionExec
DataSourceExec: file_groups={1 group:
[[{testdata}/alltypes_tiny_pages.parquet]]}, projection=[id],
output_ordering=[id@0 ASC NULLS LAST], file_type=parquet
SortExec: expr=[id@0 ASC NULLS LAST], preserve_partitioning=[false]
- DataSourceExec: file_groups={1 group:
[[{testdata}/alltypes_tiny_pages.parquet]]}, projection=[id], file_type=parquet
+ DataSourceExec: file_groups={1 group:
[[{testdata}/alltypes_tiny_pages.parquet]]}, projection=[id],
file_type=parquet, sort_order_for_reorder=[id@0 ASC NULLS LAST]
Review Comment:
maybe `inexact_output_ordering` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
adriangbot commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4461263372 🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21956#issuecomment-4461127705) **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) CPU Details (lscpu) ``` Architecture:aarch64 CPU op-mode(s): 64-bit Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Vendor ID: ARM Model name: Neoverse-V2 Model: 1 Thread(s) per core: 1 Core(s) per cluster: 16 Socket(s): - Cluster(s): 1 Stepping:r0p1 BogoMIPS:2000.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti L1d cache: 1 MiB (16 instances) L1i cache: 1 MiB (16 instances) L2 cache:32 MiB (16 instances) L3 cache:80 MiB (1 instance) NUMA node(s):1 NUMA node0 CPU(s): 0-15 Vulnerability Gather data sampling: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling:Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1:Mitigation; __user pointer sanitization Vulnerability Spectre v2:Mitigation; CSV2, BHB Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Not affected ``` Details ``` Comparing HEAD and feat_rg-reorder-by-statistics Benchmark clickbench_partitioned.json ┏━━━┳━━━┳━━━┳━━━┓ ┃ Query ┃ HEAD ┃ feat_rg-reorder-by-statistics ┃Change ┃ ┡━━━╇━━━╇━━━╇━━━┩ │ QQuery 0 │ 1.20 / 4.60 ±6.71 / 18.01 ms │ 1.30 / 4.86 ±6.97 / 18.81 ms │ 1.06x slower │ │ QQuery 1 │12.56 / 12.71 ±0.12 / 12.85 ms │12.79 / 13.05 ±0.18 / 13.31 ms │ no change │ │ QQuery 2 │35.59 / 35.94 ±0.33 / 36.38 ms │35.94 / 36.24 ±0.34 / 36.84 ms │ no change │ │ QQuery 3 │30.01 / 30.78 ±0.65 / 31.90 ms │31.22 / 31.65 ±0.49 / 32.59 ms │ no change │ │ QQuery 4 │ 229.76 / 236.18 ±7.24 / 250.00 ms │ 250.06 / 257.20 ±5.19 / 266.18 ms │ 1.09x slower │ │ QQuery 5 │ 278.50 / 285.58 ±5.72 / 292.74 ms │ 280.17 / 287.02 ±4.67 / 294.58 ms │ no change │ │ QQuery 6 │ 6.83 / 7.79 ±0.55 / 8.34 ms │ 6.54 / 7.34 ±0.57 / 8.20 ms │ +1.06x faster │ │ QQuery 7 │13.82 / 14.67 ±1.48 / 17.62 ms │13.25 / 13.73 ±0.34 / 14.30 ms │ +1.07x faster │ │ QQuery 8 │329.44 / 339.99 ±10.64 / 354.02 ms │326.89 / 336.41 ±11.92 / 359.21 ms │ no change │ │ QQuery 9 │ 450.75 / 461.75 ±7.31 / 470.94 ms │ 491.59 / 498.22 ±6.07 / 509.42 ms │ 1.08x slower │ │ QQuery 10 │70.03 / 71.64 ±0.99 / 72.66 ms │70.73 / 72.02 ±0.77 / 72.95 ms │ no change │ │ QQuery 11 │80.78 / 83.95 ±2.20 / 86.58 ms │81.90 / 82.68 ±0.86 / 83.76 ms │ no change │ │ QQuery 12 │ 274.38 / 281.33 ±4.26 / 286.69 ms │ 290.85 / 300.75 ±6.38 / 309.86 ms │ 1.07x slower │ │ QQuery 13 │384.80 / 399.05 ±12.08 / 413.77 ms │385.73 / 403.24 ±12.62 / 422.43 ms │ no change │ │ QQuery 14 │ 290.25 / 303.64 ±7.80 / 311.59 ms │ 287.90 / 294.31 ±6.89 / 305.95 ms │ no change │ │
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
adriangbot commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4461224853 🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21956#issuecomment-4461127705) **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) CPU Details (lscpu) ``` Architecture:aarch64 CPU op-mode(s): 64-bit Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Vendor ID: ARM Model name: Neoverse-V2 Model: 1 Thread(s) per core: 1 Core(s) per cluster: 16 Socket(s): - Cluster(s): 1 Stepping:r0p1 BogoMIPS:2000.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti L1d cache: 1 MiB (16 instances) L1i cache: 1 MiB (16 instances) L2 cache:32 MiB (16 instances) L3 cache:80 MiB (1 instance) NUMA node(s):1 NUMA node0 CPU(s): 0-15 Vulnerability Gather data sampling: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling:Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1:Mitigation; __user pointer sanitization Vulnerability Spectre v2:Mitigation; CSV2, BHB Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Not affected ``` Details ``` Comparing HEAD and feat_rg-reorder-by-statistics Benchmark tpch_sf1.json ┏━━━┳┳┳━━━┓ ┃ Query ┃ HEAD ┃ feat_rg-reorder-by-statistics ┃Change ┃ ┡━━━╇╇╇━━━┩ │ QQuery 1 │ 39.59 / 40.24 ±0.76 / 41.53 ms │ 38.78 / 39.54 ±0.87 / 41.01 ms │ no change │ │ QQuery 2 │ 20.87 / 21.02 ±0.16 / 21.32 ms │ 20.35 / 20.84 ±0.28 / 21.09 ms │ no change │ │ QQuery 3 │ 35.15 / 37.46 ±1.20 / 38.38 ms │ 35.47 / 37.51 ±1.79 / 40.66 ms │ no change │ │ QQuery 4 │ 17.50 / 18.02 ±0.60 / 19.18 ms │ 17.45 / 17.54 ±0.08 / 17.64 ms │ no change │ │ QQuery 5 │ 43.80 / 44.77 ±0.81 / 45.88 ms │ 43.58 / 44.90 ±1.10 / 46.28 ms │ no change │ │ QQuery 6 │ 16.76 / 17.08 ±0.46 / 17.99 ms │ 16.46 / 16.74 ±0.30 / 17.29 ms │ no change │ │ QQuery 7 │ 49.92 / 51.57 ±1.56 / 54.38 ms │ 48.91 / 49.96 ±0.73 / 51.20 ms │ no change │ │ QQuery 8 │ 46.20 / 46.79 ±0.64 / 48.03 ms │ 45.97 / 46.03 ±0.05 / 46.11 ms │ no change │ │ QQuery 9 │ 50.87 / 51.86 ±0.90 / 53.50 ms │ 51.06 / 51.90 ±0.91 / 53.30 ms │ no change │ │ QQuery 10 │ 64.54 / 64.65 ±0.08 / 64.77 ms │ 64.34 / 64.94 ±0.60 / 65.96 ms │ no change │ │ QQuery 11 │ 13.95 / 14.53 ±1.00 / 16.53 ms │ 13.70 / 14.12 ±0.53 / 15.10 ms │ no change │ │ QQuery 12 │ 25.26 / 26.17 ±1.03 / 28.09 ms │ 25.33 / 26.01 ±0.79 / 27.49 ms │ no change │ │ QQuery 13 │ 35.57 / 36.80 ±0.88 / 38.30 ms │ 35.88 / 36.43 ±0.41 / 37.08 ms │ no change │ │ QQuery 14 │ 26.09 / 26.34 ±0.21 / 26.71 ms │ 25.84 / 26.24 ±0.32 / 26.68 ms │ no change │ │ QQuery 15 │ 32.40 / 33.03 ±0.77 / 34.53 ms │ 31.70 / 32.52 ±0.65 / 33.42 ms │ no change │ │ QQuery 16 │ 15.21 / 15.43 ±0.21 / 15.81 ms │ 15.28 / 15.34 ±0.04 / 15.41 ms │ no change │ │ QQuery 17 │ 76.46 / 78.16 ±1.67 / 80.61 ms │ 76.57 / 76.84 ±0.26 / 77.22 ms │ no change │ │ QQuery 18 │ 69.65 / 70.63 ±0.66 / 71.63 ms │ 69.11 / 69.94 ±0.61 / 70.47 ms │ no change │ │ QQuery 19 │ 35.98 / 37.08 ±1.34 / 39.69 ms │ 35.4
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
adriangbot commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4461221290 🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21956#issuecomment-4461127705) **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) CPU Details (lscpu) ``` Architecture:aarch64 CPU op-mode(s): 64-bit Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Vendor ID: ARM Model name: Neoverse-V2 Model: 1 Thread(s) per core: 1 Core(s) per cluster: 16 Socket(s): - Cluster(s): 1 Stepping:r0p1 BogoMIPS:2000.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti L1d cache: 1 MiB (16 instances) L1i cache: 1 MiB (16 instances) L2 cache:32 MiB (16 instances) L3 cache:80 MiB (1 instance) NUMA node(s):1 NUMA node0 CPU(s): 0-15 Vulnerability Gather data sampling: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling:Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1:Mitigation; __user pointer sanitization Vulnerability Spectre v2:Mitigation; CSV2, BHB Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Not affected ``` Details ``` Comparing HEAD and feat_rg-reorder-by-statistics Benchmark tpcds_sf1.json ┏━━━┳━━━┳━━━┳━━━┓ ┃ Query ┃ HEAD ┃ feat_rg-reorder-by-statistics ┃Change ┃ ┡━━━╇━━━╇━━━╇━━━┩ │ QQuery 1 │ 6.36 / 7.00 ±0.81 / 8.52 ms │ 6.39 / 6.93 ±0.86 / 8.63 ms │ no change │ │ QQuery 2 │82.00 / 82.34 ±0.24 / 82.75 ms │81.05 / 81.53 ±0.37 / 81.91 ms │ no change │ │ QQuery 3 │29.26 / 29.47 ±0.24 / 29.82 ms │28.86 / 29.43 ±0.36 / 29.79 ms │ no change │ │ QQuery 4 │ 523.05 / 528.70 ±6.62 / 540.67 ms │ 520.59 / 526.38 ±5.15 / 535.52 ms │ no change │ │ QQuery 5 │52.91 / 53.25 ±0.31 / 53.82 ms │53.21 / 53.40 ±0.15 / 53.63 ms │ no change │ │ QQuery 6 │36.00 / 36.33 ±0.25 / 36.66 ms │36.36 / 36.68 ±0.27 / 37.06 ms │ no change │ │ QQuery 7 │ 110.63 / 112.82 ±2.36 / 117.17 ms │ 109.94 / 112.49 ±1.82 / 115.49 ms │ no change │ │ QQuery 8 │39.54 / 39.73 ±0.18 / 40.05 ms │39.26 / 39.80 ±0.33 / 40.23 ms │ no change │ │ QQuery 9 │54.26 / 55.65 ±1.31 / 58.12 ms │54.51 / 56.14 ±2.00 / 59.99 ms │ no change │ │ QQuery 10 │82.27 / 83.43 ±1.69 / 86.78 ms │81.21 / 82.77 ±2.26 / 87.25 ms │ no change │ │ QQuery 11 │ 325.13 / 328.78 ±3.94 / 336.45 ms │ 318.10 / 324.77 ±4.89 / 332.86 ms │ no change │ │ QQuery 12 │28.87 / 29.24 ±0.42 / 30.02 ms │28.54 / 29.03 ±0.34 / 29.56 ms │ no change │ │ QQuery 13 │ 129.21 / 129.80 ±0.35 / 130.32 ms │ 129.07 / 129.71 ±0.64 / 130.93 ms │ no change │ │ QQuery 14 │ 517.29 / 519.90 ±2.10 / 522.53 ms │ 514.41 / 516.12 ±1.10 / 517.40 ms │ no change │ │ QQuery 15 │61.05 / 62.38 ±1.12 / 64.07 ms │60.28 / 61.60 ±1.33 / 63.93 ms │ no change │ │ QQuery 16 │ 6.88 / 7.06 ±0.18 / 7.37 ms │ 7.04 / 7.22 ±0.28 / 7
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
adriangbot commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4461159131 🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21956#issuecomment-4461127705) **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4461127705-143-b666m 6.12.68+ #1 SMP Wed Apr 1 02:23:28 UTC 2026 aarch64 GNU/Linux` CPU Details (lscpu) ``` Architecture:aarch64 CPU op-mode(s): 64-bit Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Vendor ID: ARM Model name: Neoverse-V2 Model: 1 Thread(s) per core: 1 Core(s) per cluster: 16 Socket(s): - Cluster(s): 1 Stepping:r0p1 BogoMIPS:2000.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti L1d cache: 1 MiB (16 instances) L1i cache: 1 MiB (16 instances) L2 cache:32 MiB (16 instances) L3 cache:80 MiB (1 instance) NUMA node(s):1 NUMA node0 CPU(s): 0-15 Vulnerability Gather data sampling: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling:Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1:Mitigation; __user pointer sanitization Vulnerability Spectre v2:Mitigation; CSV2, BHB Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Not affected ``` Comparing feat/rg-reorder-by-statistics (58a654bb9afb7fcee23be21839f727a77fbc31fd) to 9d92944 (merge-base) [diff](https://github.com/apache/datafusion/compare/9d92944900f8a50b688a4bd8685629ebd76999f3..58a654bb9afb7fcee23be21839f727a77fbc31fd) using: clickbench_partitioned Results will be posted here when complete --- [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
adriangbot commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4461158311 🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21956#issuecomment-4461127705) **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4461127705-145-mqcpm 6.12.68+ #1 SMP Wed Apr 1 02:23:28 UTC 2026 aarch64 GNU/Linux` CPU Details (lscpu) ``` Architecture:aarch64 CPU op-mode(s): 64-bit Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Vendor ID: ARM Model name: Neoverse-V2 Model: 1 Thread(s) per core: 1 Core(s) per cluster: 16 Socket(s): - Cluster(s): 1 Stepping:r0p1 BogoMIPS:2000.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti L1d cache: 1 MiB (16 instances) L1i cache: 1 MiB (16 instances) L2 cache:32 MiB (16 instances) L3 cache:80 MiB (1 instance) NUMA node(s):1 NUMA node0 CPU(s): 0-15 Vulnerability Gather data sampling: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling:Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1:Mitigation; __user pointer sanitization Vulnerability Spectre v2:Mitigation; CSV2, BHB Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Not affected ``` Comparing feat/rg-reorder-by-statistics (58a654bb9afb7fcee23be21839f727a77fbc31fd) to 9d92944 (merge-base) [diff](https://github.com/apache/datafusion/compare/9d92944900f8a50b688a4bd8685629ebd76999f3..58a654bb9afb7fcee23be21839f727a77fbc31fd) using: tpch Results will be posted here when complete --- [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
adriangbot commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4461145365 🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21956#issuecomment-4461127705) **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4461127705-144-nplqp 6.12.68+ #1 SMP Wed Apr 1 02:23:28 UTC 2026 aarch64 GNU/Linux` CPU Details (lscpu) ``` Architecture:aarch64 CPU op-mode(s): 64-bit Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Vendor ID: ARM Model name: Neoverse-V2 Model: 1 Thread(s) per core: 1 Core(s) per cluster: 16 Socket(s): - Cluster(s): 1 Stepping:r0p1 BogoMIPS:2000.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti L1d cache: 1 MiB (16 instances) L1i cache: 1 MiB (16 instances) L2 cache:32 MiB (16 instances) L3 cache:80 MiB (1 instance) NUMA node(s):1 NUMA node0 CPU(s): 0-15 Vulnerability Gather data sampling: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling:Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1:Mitigation; __user pointer sanitization Vulnerability Spectre v2:Mitigation; CSV2, BHB Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Not affected ``` Comparing feat/rg-reorder-by-statistics (58a654bb9afb7fcee23be21839f727a77fbc31fd) to 9d92944 (merge-base) [diff](https://github.com/apache/datafusion/compare/9d92944900f8a50b688a4bd8685629ebd76999f3..58a654bb9afb7fcee23be21839f727a77fbc31fd) using: tpcds Results will be posted here when complete --- [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
adriangb commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4461127705 run benchmarks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
adriangbot commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4458821315 🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21956#issuecomment-4458695623) **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) CPU Details (lscpu) ``` Architecture:aarch64 CPU op-mode(s): 64-bit Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Vendor ID: ARM Model name: Neoverse-V2 Model: 1 Thread(s) per core: 1 Core(s) per cluster: 16 Socket(s): - Cluster(s): 1 Stepping:r0p1 BogoMIPS:2000.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti L1d cache: 1 MiB (16 instances) L1i cache: 1 MiB (16 instances) L2 cache:32 MiB (16 instances) L3 cache:80 MiB (1 instance) NUMA node(s):1 NUMA node0 CPU(s): 0-15 Vulnerability Gather data sampling: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling:Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1:Mitigation; __user pointer sanitization Vulnerability Spectre v2:Mitigation; CSV2, BHB Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Not affected ``` Details ``` Comparing HEAD and feat_rg-reorder-by-statistics Benchmark sort_pushdown_inexact.json ┏━━━┳┳┳━━━┓ ┃ Query ┃ HEAD ┃ feat_rg-reorder-by-statistics ┃ Change ┃ ┡━━━╇╇╇━━━┩ │ Q1│6.98 / 8.61 ±1.02 / 9.90 ms │6.77 / 7.46 ±1.03 / 9.48 ms │ +1.15x faster │ │ Q2│6.97 / 8.10 ±0.74 / 9.29 ms │6.55 / 7.03 ±0.85 / 8.73 ms │ +1.15x faster │ │ Q3│ 21.93 / 22.95 ±0.76 / 24.05 ms │ 21.72 / 22.17 ±0.31 / 22.65 ms │ no change │ │ Q4│ 21.85 / 22.61 ±0.83 / 24.22 ms │ 20.10 / 20.94 ±0.65 / 21.76 ms │ +1.08x faster │ └───┴┴┴───┘ ┏━━┳━┓ ┃ Benchmark Summary┃ ┃ ┡━━╇━┩ │ Total Time (HEAD)│ 62.27ms │ │ Total Time (feat_rg-reorder-by-statistics) │ 57.60ms │ │ Average Time (HEAD) │ 15.57ms │ │ Average Time (feat_rg-reorder-by-statistics) │ 14.40ms │ │ Queries Faster │ 3 │ │ Queries Slower │ 0 │ │ Queries with No Change │ 1 │ │ Queries with Failure │ 0 │ └──┴─┘ ``` Resource Usage **sort_pushdown_inexact — base (merge-base)** | Metric | Value | ||---| | Wall time | 5.0s | | Peak memory | 4.6 GiB | | Avg memory | 4.6 GiB | | CPU user | 2.6s | | CPU sys | 0.4s | | Peak spill | 0 B | **sort_pushdown_inexact — branch** | Metric | Value | ||---| | Wall time | 5.0s | | Peak memory | 4.6 GiB | | Avg memory | 4.6 GiB | | CPU user | 2.5s | | CPU sys | 0.3s | | Peak spill | 0 B | --
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
adriangbot commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4458716112 🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21956#issuecomment-4458695623) **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4458695623-123-j4qgs 6.12.68+ #1 SMP Wed Apr 1 02:23:28 UTC 2026 aarch64 GNU/Linux` CPU Details (lscpu) ``` Architecture:aarch64 CPU op-mode(s): 64-bit Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Vendor ID: ARM Model name: Neoverse-V2 Model: 1 Thread(s) per core: 1 Core(s) per cluster: 16 Socket(s): - Cluster(s): 1 Stepping:r0p1 BogoMIPS:2000.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti L1d cache: 1 MiB (16 instances) L1i cache: 1 MiB (16 instances) L2 cache:32 MiB (16 instances) L3 cache:80 MiB (1 instance) NUMA node(s):1 NUMA node0 CPU(s): 0-15 Vulnerability Gather data sampling: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling:Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1:Mitigation; __user pointer sanitization Vulnerability Spectre v2:Mitigation; CSV2, BHB Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Not affected ``` Comparing feat/rg-reorder-by-statistics (58a654bb9afb7fcee23be21839f727a77fbc31fd) to 9d92944 (merge-base) [diff](https://github.com/apache/datafusion/compare/9d92944900f8a50b688a4bd8685629ebd76999f3..58a654bb9afb7fcee23be21839f727a77fbc31fd) using: sort_pushdown_inexact Results will be posted here when complete --- [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
zhuqi-lucas commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4458695623 run benchmark sort_pushdown_inexact -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
zhuqi-lucas commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4458351797 https://github.com/apache/datafusion/issues/22198 I also created some follow-up feature after this done, or do you think we need to do part of those in this PR, i am also ok to implement in this PR. cc @adriangb @alamb -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
adriangb commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4458010977 Thanks @zhuqi-lucas. Will take one last look tomorrow!! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
zhuqi-lucas commented on PR #21956:
URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4457944094
Thanks @adriangb! Adopted all five , landed in PR already.
One tweak in #3: your `reverse_row_groups = reversed_satisfies ||
is_descending` re-introduces a latent bug for source `[a DESC]` + request `[a
ASC]` — both flags are true while the request is ASC, so the formula
flips iteration to DESC after the stats-based reorder, wrong direction.
Kept the fix from the earlier "route stats-based RG reorder through
try_pushdown_sort" commit:
```rust
new_source.reverse_row_groups = if column_in_file_schema {
is_descending // stats reorder gives ASC-by-min; flip iff request
DESC
} else {
true// flipping the file's natural order
};
What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
zhuqi-lucas commented on code in PR #21956:
URL: https://github.com/apache/datafusion/pull/21956#discussion_r3246405372
##
datafusion/datasource-parquet/src/source.rs:
##
@@ -815,17 +975,53 @@ impl FileSource for ParquetSource {
new
};
-// Check if the reversed ordering satisfies the requested ordering
-if !reversed_eq_properties.ordering_satisfy(order.iter().cloned())? {
-return Ok(SortOrderPushdownResult::Unsupported);
+// Check if the reversed ordering satisfies the requested ordering.
+// If yes, this is the "Inexact via row-group reversal" case: source
+// declares ASC ordering, request is DESC (or vice versa), so iterating
+// RGs in reverse approximates the requested order.
+if reversed_eq_properties.ordering_satisfy(order.iter().cloned())? {
+let sort_order = LexOrdering::new(order.iter().cloned());
+let mut new_source = self.clone().with_reverse_row_groups(true);
+new_source.sort_order_for_reorder = sort_order;
+return Ok(SortOrderPushdownResult::Inexact {
+inner: Arc::new(new_source) as Arc,
+});
Review Comment:
Took a closer look while refactoring — partially merged, but kept a narrow
fallback. Rationale spelled out in the new doc on `try_pushdown_sort`:
- **Plain `Column` leading sort**: merged into the stats-based path as you
suggested. This branch now sets both `sort_order_for_reorder` and
`reverse_row_groups`, with the latter taken from the request's direction (which
also fixes a latent source-DESC + request-ASC mismatch).
- **Function-wrapped sort** (e.g. `date_trunc('day', ts) DESC`, `ceil(value)
DESC`): retained as a reverse-only branch. `reorder_by_statistics` calls
`StatisticsConverter::try_new(column_name, ...)` — parquet stores min/max keyed
by physical column name, so it fundamentally needs a plain `Column` reference.
The branches encode two genuinely different runtime operations:
| | `reorder_by_statistics` | `reverse` |
|---|---|---|
| Cost | O(n log n) + metadata reads | O(n) iteration flip |
| Needs parquet stats | yes | no |
| Sort-expression constraint | leading must map to a physical column |
none |
So even if we extended `reorder_by_statistics` to multi-column or to
monotonic function wrappers (good follow-up — currently it only uses
`sort_order.first()` and only handles `Column`), the two would still need to
coexist for the cases where one operation is available and the other isn't.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
zhuqi-lucas commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4457707801 > Sorry if you started working on my comments already — I left you a PR with the suggestions at [zhuqi-lucas#1](https://github.com/zhuqi-lucas/arrow-datafusion/pull/1). Feel free to cherry-pick, ignore, or push back on any of the five commits. Thanks @adriangb , i will take a look further. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
adriangb commented on code in PR #21956:
URL: https://github.com/apache/datafusion/pull/21956#discussion_r324693
##
datafusion/datasource-parquet/src/source.rs:
##
@@ -815,17 +975,53 @@ impl FileSource for ParquetSource {
new
};
-// Check if the reversed ordering satisfies the requested ordering
-if !reversed_eq_properties.ordering_satisfy(order.iter().cloned())? {
-return Ok(SortOrderPushdownResult::Unsupported);
+// Check if the reversed ordering satisfies the requested ordering.
+// If yes, this is the "Inexact via row-group reversal" case: source
+// declares ASC ordering, request is DESC (or vice versa), so iterating
+// RGs in reverse approximates the requested order.
+if reversed_eq_properties.ordering_satisfy(order.iter().cloned())? {
+let sort_order = LexOrdering::new(order.iter().cloned());
+let mut new_source = self.clone().with_reverse_row_groups(true);
+new_source.sort_order_for_reorder = sort_order;
+return Ok(SortOrderPushdownResult::Inexact {
+inner: Arc::new(new_source) as Arc,
+});
Review Comment:
All good, let's go with this / keep the fallback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
adriangb commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4457670798 Sorry if you started working on my comments already — I left you a PR with the suggestions at https://github.com/zhuqi-lucas/arrow-datafusion/pull/1. Feel free to cherry-pick, ignore, or push back on any of the five commits. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
zhuqi-lucas commented on code in PR #21956:
URL: https://github.com/apache/datafusion/pull/21956#discussion_r3246405372
##
datafusion/datasource-parquet/src/source.rs:
##
@@ -815,17 +975,53 @@ impl FileSource for ParquetSource {
new
};
-// Check if the reversed ordering satisfies the requested ordering
-if !reversed_eq_properties.ordering_satisfy(order.iter().cloned())? {
-return Ok(SortOrderPushdownResult::Unsupported);
+// Check if the reversed ordering satisfies the requested ordering.
+// If yes, this is the "Inexact via row-group reversal" case: source
+// declares ASC ordering, request is DESC (or vice versa), so iterating
+// RGs in reverse approximates the requested order.
+if reversed_eq_properties.ordering_satisfy(order.iter().cloned())? {
+let sort_order = LexOrdering::new(order.iter().cloned());
+let mut new_source = self.clone().with_reverse_row_groups(true);
+new_source.sort_order_for_reorder = sort_order;
+return Ok(SortOrderPushdownResult::Inexact {
+inner: Arc::new(new_source) as Arc,
+});
Review Comment:
Took a closer look while refactoring — partially merged, but kept a narrow
fallback.
- **Plain `Column` leading sort**: merged into the stats-based path as you
suggested. The column-in-schema branch now sets both `sort_order_for_reorder`
and `reverse_row_groups` (the latter taken from the
request's direction, which also fixes a latent source-DESC + request-ASC
mismatch).
- **Function-wrapped sort** (e.g. `date_trunc('day', ts) DESC`,
`ceil(value) DESC`): kept as a reverse-only fallback. `reorder_by_statistics`
needs a plain column name to look up min/max in parquet metadata, so
it can't subsume these. The fallback only sets `reverse_row_groups=true`
and no longer wastes a `sort_order_for_reorder` hint that would just be skipped.
Rationale spelled out in the new doc on `try_pushdown_sort`. Open to
dropping the fallback entirely if you'd rather not pay that complexity for
function-wrapped sorts — let me know.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
zhuqi-lucas commented on code in PR #21956:
URL: https://github.com/apache/datafusion/pull/21956#discussion_r3246190490
##
datafusion/datasource-parquet/src/access_plan.rs:
##
@@ -16,10 +16,28 @@
// under the License.
use crate::sort::reverse_row_selection;
+use arrow::array::{Array, ArrayRef, BooleanArray};
+use arrow::datatypes::Schema;
use datafusion_common::{Result, assert_eq_or_internal_err};
+use datafusion_physical_expr::expressions::Column;
+use datafusion_physical_expr_common::sort_expr::LexOrdering;
+use log::debug;
+use parquet::arrow::arrow_reader::statistics::StatisticsConverter;
use parquet::arrow::arrow_reader::{RowSelection, RowSelector};
use parquet::file::metadata::{ParquetMetaData, RowGroupMetaData};
+/// Fraction of adjacent (in sorted-by-min order) row group pairs whose
+/// `[min, max]` ranges overlap above which `reorder_by_statistics` will
+/// bail out without reordering.
+///
+/// When stats overlap heavily (e.g. unsorted columns like ClickBench's
+/// `EventTime` on `hits_partitioned`), reordering by min cannot enable
+/// row-group-level pruning — every "later" RG still has values that
+/// could appear in TopK. The reorder cost (CPU sort + lost IO sequential
+/// locality + parallel scheduling pessimization across workers all
+/// pulling "best" RGs first) then dominates, producing a net regression.
Review Comment:
Yes, this logic was added because the clickbench benchmark result several
topk have regression, but i can remove those logic and trigger again, if it
works well, we don't need to add those logic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
zhuqi-lucas commented on code in PR #21956:
URL: https://github.com/apache/datafusion/pull/21956#discussion_r3246146008
##
datafusion/datasource-parquet/src/source.rs:
##
@@ -815,17 +975,53 @@ impl FileSource for ParquetSource {
new
};
-// Check if the reversed ordering satisfies the requested ordering
-if !reversed_eq_properties.ordering_satisfy(order.iter().cloned())? {
-return Ok(SortOrderPushdownResult::Unsupported);
+// Check if the reversed ordering satisfies the requested ordering.
+// If yes, this is the "Inexact via row-group reversal" case: source
+// declares ASC ordering, request is DESC (or vice versa), so iterating
+// RGs in reverse approximates the requested order.
+if reversed_eq_properties.ordering_satisfy(order.iter().cloned())? {
+let sort_order = LexOrdering::new(order.iter().cloned());
+let mut new_source = self.clone().with_reverse_row_groups(true);
+new_source.sort_order_for_reorder = sort_order;
+return Ok(SortOrderPushdownResult::Inexact {
+inner: Arc::new(new_source) as Arc,
+});
Review Comment:
Thanks, i think it's same, will merge those logic soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
zhuqi-lucas commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4457183435 > Some nits: > > 1. I still think those traits should go. > 2. Several references to overheads that don't really make sense to me. Are those justified by benchmarks? Yes, those changes are from previous benchmark regression, but i am not sure if it's noise, we can trigger benchmark again, i will address it first. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
zhuqi-lucas commented on code in PR #21956:
URL: https://github.com/apache/datafusion/pull/21956#discussion_r3246110701
##
datafusion/core/tests/physical_optimizer/pushdown_sort.rs:
##
@@ -310,7 +317,7 @@ fn test_no_prefix_match_longer_than_source() {
output:
Ok:
- SortExec: expr=[a@0 ASC, b@1 DESC NULLS LAST],
preserve_partitioning=[false]
- - DataSourceExec: file_groups={1 group: [[x]]}, projection=[a, b,
c, d, e], output_ordering=[a@0 DESC NULLS LAST], file_type=parquet
+ - DataSourceExec: file_groups={1 group: [[x]]}, projection=[a, b,
c, d, e], file_type=parquet
Review Comment:
Good suggestion!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
zhuqi-lucas commented on code in PR #21956:
URL: https://github.com/apache/datafusion/pull/21956#discussion_r3246109227
##
datafusion/datasource-parquet/src/source.rs:
##
@@ -482,6 +485,107 @@ impl ParquetSource {
pub(crate) fn reverse_row_groups(&self) -> bool {
self.reverse_row_groups
}
+
+/// Extract the (column name, descending) tuple used by file-level
+/// reordering. Driven entirely from the sort-pushdown channel
+/// (`sort_order_for_reorder` + `reverse_row_groups`) — set by
+/// `try_pushdown_sort`. We do not consult any dynamic-filter
+/// metadata here: `DynamicFilterPhysicalExpr` is for runtime
+/// threshold pruning, not for telling the source how to schedule
+/// reads.
+fn extract_topk_sort_info(&self) -> Option<(String, bool)> {
+let sort_order = self.sort_order_for_reorder.as_ref()?;
+let first = sort_order.first();
+let col = first
+.expr
+.downcast_ref::()?;
+Some((col.name().to_string(), self.reverse_row_groups))
+}
+
+/// Extract the sort key from a file's statistics for reordering.
+fn sort_key_for_file(
+file: &datafusion_datasource::PartitionedFile,
+col_idx: usize,
+descending: bool,
+) -> Option {
+let stats = file.statistics.as_ref()?;
+let col_stats = stats.column_statistics.get(col_idx)?;
+if descending {
+col_stats.min_value.get_value().cloned()
+} else {
+col_stats.max_value.get_value().cloned()
+}
+}
+}
+
+/// Threshold (fraction in `[0, 1]`) for the overlap guard in
+/// [`ParquetSource::reorder_files`]. When at least this fraction of
+/// adjacent file pairs (in sorted-by-min order) have overlapping
+/// `[min, max]` ranges, file reorder is skipped — file-level pruning
+/// cannot help and the reorder cost would dominate.
Review Comment:
I think it was coming from the benchmark data from previous PR trigger, i
can remove it and trigger again to see it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
zhuqi-lucas commented on code in PR #21956:
URL: https://github.com/apache/datafusion/pull/21956#discussion_r3246103116
##
datafusion/datasource-parquet/src/source.rs:
##
@@ -482,6 +485,107 @@ impl ParquetSource {
pub(crate) fn reverse_row_groups(&self) -> bool {
self.reverse_row_groups
}
+
+/// Extract the (column name, descending) tuple used by file-level
+/// reordering. Driven entirely from the sort-pushdown channel
+/// (`sort_order_for_reorder` + `reverse_row_groups`) — set by
+/// `try_pushdown_sort`. We do not consult any dynamic-filter
+/// metadata here: `DynamicFilterPhysicalExpr` is for runtime
+/// threshold pruning, not for telling the source how to schedule
+/// reads.
Review Comment:
Yes, will update the comment here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
adriangb commented on code in PR #21956:
URL: https://github.com/apache/datafusion/pull/21956#discussion_r3246022325
##
datafusion/core/tests/physical_optimizer/pushdown_sort.rs:
##
@@ -310,7 +317,7 @@ fn test_no_prefix_match_longer_than_source() {
output:
Ok:
- SortExec: expr=[a@0 ASC, b@1 DESC NULLS LAST],
preserve_partitioning=[false]
- - DataSourceExec: file_groups={1 group: [[x]]}, projection=[a, b,
c, d, e], output_ordering=[a@0 DESC NULLS LAST], file_type=parquet
+ - DataSourceExec: file_groups={1 group: [[x]]}, projection=[a, b,
c, d, e], file_type=parquet
Review Comment:
I wonder if we could include the inexact ordering, perhaps under an `EXPLAIN
ANALYZE` option?
##
datafusion/datasource-parquet/src/source.rs:
##
@@ -482,6 +485,107 @@ impl ParquetSource {
pub(crate) fn reverse_row_groups(&self) -> bool {
self.reverse_row_groups
}
+
+/// Extract the (column name, descending) tuple used by file-level
+/// reordering. Driven entirely from the sort-pushdown channel
+/// (`sort_order_for_reorder` + `reverse_row_groups`) — set by
+/// `try_pushdown_sort`. We do not consult any dynamic-filter
+/// metadata here: `DynamicFilterPhysicalExpr` is for runtime
+/// threshold pruning, not for telling the source how to schedule
+/// reads.
+fn extract_topk_sort_info(&self) -> Option<(String, bool)> {
+let sort_order = self.sort_order_for_reorder.as_ref()?;
+let first = sort_order.first();
+let col = first
+.expr
+.downcast_ref::()?;
+Some((col.name().to_string(), self.reverse_row_groups))
+}
+
+/// Extract the sort key from a file's statistics for reordering.
+fn sort_key_for_file(
+file: &datafusion_datasource::PartitionedFile,
+col_idx: usize,
+descending: bool,
+) -> Option {
+let stats = file.statistics.as_ref()?;
+let col_stats = stats.column_statistics.get(col_idx)?;
+if descending {
+col_stats.min_value.get_value().cloned()
+} else {
+col_stats.max_value.get_value().cloned()
+}
+}
+}
+
+/// Threshold (fraction in `[0, 1]`) for the overlap guard in
+/// [`ParquetSource::reorder_files`]. When at least this fraction of
+/// adjacent file pairs (in sorted-by-min order) have overlapping
+/// `[min, max]` ranges, file reorder is skipped — file-level pruning
+/// cannot help and the reorder cost would dominate.
Review Comment:
Curious what that cost is?
##
datafusion/datasource-parquet/src/source.rs:
##
@@ -482,6 +485,107 @@ impl ParquetSource {
pub(crate) fn reverse_row_groups(&self) -> bool {
self.reverse_row_groups
}
+
+/// Extract the (column name, descending) tuple used by file-level
+/// reordering. Driven entirely from the sort-pushdown channel
+/// (`sort_order_for_reorder` + `reverse_row_groups`) — set by
+/// `try_pushdown_sort`. We do not consult any dynamic-filter
+/// metadata here: `DynamicFilterPhysicalExpr` is for runtime
+/// threshold pruning, not for telling the source how to schedule
+/// reads.
Review Comment:
This seems like a typical LLM comment: it's referencing the previous
implementation in this PR. That won't be helpful to readers once this code is
merged. Please update the comment to explain what the current / final code is
doing, not what previous drafts did.
##
datafusion/datasource-parquet/src/access_plan.rs:
##
@@ -16,10 +16,28 @@
// under the License.
use crate::sort::reverse_row_selection;
+use arrow::array::{Array, ArrayRef, BooleanArray};
+use arrow::datatypes::Schema;
use datafusion_common::{Result, assert_eq_or_internal_err};
+use datafusion_physical_expr::expressions::Column;
+use datafusion_physical_expr_common::sort_expr::LexOrdering;
+use log::debug;
+use parquet::arrow::arrow_reader::statistics::StatisticsConverter;
use parquet::arrow::arrow_reader::{RowSelection, RowSelector};
use parquet::file::metadata::{ParquetMetaData, RowGroupMetaData};
+/// Fraction of adjacent (in sorted-by-min order) row group pairs whose
+/// `[min, max]` ranges overlap above which `reorder_by_statistics` will
+/// bail out without reordering.
+///
+/// When stats overlap heavily (e.g. unsorted columns like ClickBench's
+/// `EventTime` on `hits_partitioned`), reordering by min cannot enable
+/// row-group-level pruning — every "later" RG still has values that
+/// could appear in TopK. The reorder cost (CPU sort + lost IO sequential
+/// locality + parallel scheduling pessimization across workers all
+/// pulling "best" RGs first) then dominates, producing a net regression.
Review Comment:
I wonder how much each of these effects contributes? I'd imaging CPU to sort
row groups (few containers) should be pretty minimal. If it's row groups being
sorted I don't know how much data lo
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
zhuqi-lucas commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4456649095 Thanks @adriangb, you're right. Routing through `try_pushdown_sort` matches what that channel is for, decouples the reorder from TopK (so it helps any ORDER BY on a sorted source, not just LIMIT-K), and lets the `sort_options`/`fetch` fields on `DynamicFilterPhysicalExpr` go away — along with the `find_dynamic_filter` walk under AND/wrappers and the `with_fetch`/`create_filter` ordering coupling, which were the parts I was least happy about anyway. Can't think of a case where the sort metadata reaches parquet through the dynamic filter but not through `try_pushdown_sort` — same SortExec → DataSource shape, dynamic filter just happens to be what SortExec attaches to its predicate. Let me refactor along the lines you sketched. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
adriangb commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4455515011 I wonder if this belongs on `try_pushdown_sort` rather than on `DynamicFilterPhysicalExpr`. The `PushdownSort` rule already visits `SortExec → DataSourceExec` (TopK or not) and forwards the required ordering to the source via `try_pushdown_sort(order, eq_properties)`. Today parquet's implementation only accepts when the request matches the file's declared natural ordering (or its reverse). The case this PR is targeting — request doesn't match declared ordering (or the table declares none), but the source can still reorder row groups/files by min/max stats — fits the existing `Inexact` contract: keep the `SortExec`, but feed it data in a better order. The rule already preserves `sort_exec.fetch()` on the rebuilt `SortExec`, so TopK keeps working. Concretely: in `ParquetSource::try_pushdown_sort`, after the existing `Exact`/reverse-`Inexact` checks fall through, instead of returning `Unsupported`, check whether the requested sort column exists in the file schema (the same check the opener fallback is already doing) and return `Inexact` with `sort_order_for_reorder` set. The opener already prefers `sort_order_for_reorder` over the dynamic filter — that whole branch in `opener.rs` (the `find_dynamic_filter` / `find_column_in_expr` fallback) can go away, along with the `sort_options`/`fetch` fields on `DynamicFilterPhysicalExpr` and the reordering of `with_fetch` to call `create_filter` after setting fetch. A few reasons I think this factoring is cleaner: - `DynamicFilterPhysicalExpr`'s job is "live threshold for runtime pruning." Adding sort-direction and K fields to it conflates that with "how should the source schedule its reads." `try_pushdown_sort` is the channel that already means the latter. - The reorder is useful even without a `LIMIT` — less sorting work and better locality help any `ORDER BY`. Routing through the dynamic filter ties the optimization to TopK; routing through sort pushdown makes it apply to any `SortExec → DataSource` shape. - The opener never actually reads `df.fetch()` — only `sort_options` and the child column — so the only piece of metadata being moved is one the sort-pushdown call already passes as its `order` argument. - Fewer places to maintain: no proto-roundtrip carve-out, no `find_dynamic_filter` tree walk under `AND`/wrappers (which is the kind of thing that quietly breaks the next time someone wraps the predicate), and no ordering coupling between `with_fetch` and `create_filter`. Happy to be wrong here — is there a case I'm missing where the sort metadata can reach the parquet source through the dynamic filter but *not* through `try_pushdown_sort`? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
adriangb commented on code in PR #21956:
URL: https://github.com/apache/datafusion/pull/21956#discussion_r3244796568
##
datafusion/datasource-parquet/src/access_plan_optimizer.rs:
##
@@ -0,0 +1,107 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`AccessPlanOptimizer`] trait and implementations for optimizing
+//! row group access order during parquet scans.
+//!
+//! Applied after row group pruning but before building the decoder,
+//! these optimizers reorder (or reverse) the row groups to improve
+//! query performance — e.g., placing the "best" row groups first
+//! so TopK's dynamic filter threshold tightens quickly.
+
+use crate::access_plan::PreparedAccessPlan;
+use arrow::datatypes::Schema;
+use datafusion_common::Result;
+use datafusion_physical_expr_common::sort_expr::LexOrdering;
+use parquet::file::metadata::ParquetMetaData;
+use std::fmt::Debug;
+
+/// Optimizes the row group access order for a prepared access plan.
+///
+/// Implementations can reorder, reverse, or otherwise transform the
+/// row group read order to improve scan performance. The optimizer
+/// is applied once per file, after all pruning passes are complete.
+///
+/// # Examples
+///
+/// - [`ReverseRowGroups`]: simple O(n) reversal for DESC on ASC-sorted data
+/// - [`ReorderByStatistics`]: sort row groups by min/max statistics
+/// so TopK queries find optimal values first
+pub(crate) trait AccessPlanOptimizer: Send + Sync + Debug {
+/// Transform the prepared access plan.
+///
+/// Implementations should return the plan unchanged if they cannot
+/// apply their optimization (e.g., missing statistics).
+fn optimize(
+&self,
+plan: PreparedAccessPlan,
+file_metadata: &ParquetMetaData,
+arrow_schema: &Schema,
+) -> Result;
+}
+
+/// Reverse the row group order — simple O(n) reversal.
+///
+/// Used as a fallback when the sort column has no statistics available.
+/// For ASC-sorted files with a DESC query, reversing row groups places
+/// the highest-value row groups first.
+#[derive(Debug)]
+pub(crate) struct ReverseRowGroups;
+
+impl AccessPlanOptimizer for ReverseRowGroups {
+fn optimize(
+&self,
+plan: PreparedAccessPlan,
+file_metadata: &ParquetMetaData,
+_arrow_schema: &Schema,
+) -> Result {
+plan.reverse(file_metadata)
Review Comment:
I like the idea but is it really worth having a trait just to hide 1 line
behind it?
##
datafusion/datasource-parquet/src/opener.rs:
##
@@ -1123,13 +1129,107 @@ impl RowGroupsPrunedParquetOpen {
);
}
-// Prepare the access plan (extract row groups and row selection)
-let mut prepared_plan = access_plan.prepare(rg_metadata)?;
+// Row group ordering optimization (two composable steps):
+//
+// 1. reorder_by_statistics: sort RGs by min values (ASC) to align
+//with the file's declared output ordering. This fixes out-of-order
Review Comment:
This comment doesn't make sense to me. A file either declares an output
ordering or not. If it does the row groups should already be sorted by it.
Should this say the _query's desired_ output ordering?
##
datafusion/datasource-parquet/src/access_plan_optimizer.rs:
##
@@ -0,0 +1,107 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! [`AccessPlanOptimizer`] trait and implementations for
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
zhuqi-lucas commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4396960901 The benchmark is ok now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
adriangbot commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4396252305 🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21956#issuecomment-4396120969) **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) CPU Details (lscpu) ``` Architecture:aarch64 CPU op-mode(s): 64-bit Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Vendor ID: ARM Model name: Neoverse-V2 Model: 1 Thread(s) per core: 1 Core(s) per cluster: 16 Socket(s): - Cluster(s): 1 Stepping:r0p1 BogoMIPS:2000.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti L1d cache: 1 MiB (16 instances) L1i cache: 1 MiB (16 instances) L2 cache:32 MiB (16 instances) L3 cache:80 MiB (1 instance) NUMA node(s):1 NUMA node0 CPU(s): 0-15 Vulnerability Gather data sampling: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling:Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1:Mitigation; __user pointer sanitization Vulnerability Spectre v2:Mitigation; CSV2, BHB Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Not affected ``` Details ``` Comparing HEAD and feat_rg-reorder-by-statistics Benchmark clickbench_partitioned.json ┏━━━┳━━━┳━━━┳━━┓ ┃ Query ┃ HEAD ┃ feat_rg-reorder-by-statistics ┃ Change ┃ ┡━━━╇━━━╇━━━╇━━┩ │ QQuery 0 │ 1.20 / 4.54 ±6.64 / 17.82 ms │ 1.18 / 4.53 ±6.64 / 17.80 ms │no change │ │ QQuery 1 │12.65 / 12.95 ±0.21 / 13.29 ms │12.69 / 12.80 ±0.08 / 12.94 ms │no change │ │ QQuery 2 │35.28 / 35.76 ±0.40 / 36.38 ms │36.42 / 36.85 ±0.24 / 37.13 ms │no change │ │ QQuery 3 │30.06 / 31.22 ±0.67 / 32.11 ms │30.29 / 30.98 ±0.64 / 32.10 ms │no change │ │ QQuery 4 │ 226.08 / 231.19 ±4.52 / 238.42 ms │ 224.28 / 230.47 ±3.90 / 235.85 ms │no change │ │ QQuery 5 │ 274.93 / 277.88 ±2.84 / 282.48 ms │ 271.91 / 274.81 ±2.46 / 278.80 ms │no change │ │ QQuery 6 │ 6.70 / 7.19 ±0.38 / 7.88 ms │ 6.96 / 7.19 ±0.20 / 7.45 ms │no change │ │ QQuery 7 │14.65 / 14.73 ±0.05 / 14.80 ms │13.89 / 14.00 ±0.07 / 14.07 ms │no change │ │ QQuery 8 │ 310.14 / 311.16 ±1.04 / 312.56 ms │ 309.27 / 311.98 ±2.40 / 316.25 ms │no change │ │ QQuery 9 │ 439.85 / 445.48 ±3.05 / 448.27 ms │ 441.40 / 447.12 ±3.32 / 450.62 ms │no change │ │ QQuery 10 │69.32 / 70.75 ±1.75 / 74.14 ms │67.98 / 68.91 ±0.74 / 70.01 ms │no change │ │ QQuery 11 │79.46 / 81.95 ±2.52 / 86.49 ms │79.14 / 81.80 ±3.87 / 89.44 ms │no change │ │ QQuery 12 │ 266.87 / 272.45 ±4.43 / 278.74 ms │ 266.41 / 271.18 ±3.75 / 277.01 ms │no change │ │ QQuery 13 │ 379.90 / 383.16 ±3.69 / 389.79 ms │ 375.59 / 387.25 ±8.17 / 395.91 ms │no change │ │ QQuery 14 │ 277.36 / 279.44 ±2.48 / 284.23 ms │ 275.12 / 277.48 ±3.51 / 284.43 ms │no change │ │ QQuery 15 │ 2
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
adriangbot commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4396144336 🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21956#issuecomment-4396120969) **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4396120969-2047-dtfjq 6.12.68+ #1 SMP Wed Apr 1 02:23:28 UTC 2026 aarch64 GNU/Linux` CPU Details (lscpu) ``` Architecture:aarch64 CPU op-mode(s): 64-bit Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Vendor ID: ARM Model name: Neoverse-V2 Model: 1 Thread(s) per core: 1 Core(s) per cluster: 16 Socket(s): - Cluster(s): 1 Stepping:r0p1 BogoMIPS:2000.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti L1d cache: 1 MiB (16 instances) L1i cache: 1 MiB (16 instances) L2 cache:32 MiB (16 instances) L3 cache:80 MiB (1 instance) NUMA node(s):1 NUMA node0 CPU(s): 0-15 Vulnerability Gather data sampling: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling:Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1:Mitigation; __user pointer sanitization Vulnerability Spectre v2:Mitigation; CSV2, BHB Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Not affected ``` Comparing feat/rg-reorder-by-statistics (6987aa226bd3076067c7c76991d6f0106b9ca028) to 3b634aa (merge-base) [diff](https://github.com/apache/datafusion/compare/3b634aaef9c2b82b41180d230807c4658e30e2f4..6987aa226bd3076067c7c76991d6f0106b9ca028) using: clickbench_partitioned Results will be posted here when complete --- [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
zhuqi-lucas commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4396120969 run benchmark clickbench_partitioned -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
adriangbot commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4395406332 🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21956#issuecomment-4395242557) **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) CPU Details (lscpu) ``` Architecture:aarch64 CPU op-mode(s): 64-bit Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Vendor ID: ARM Model name: Neoverse-V2 Model: 1 Thread(s) per core: 1 Core(s) per cluster: 16 Socket(s): - Cluster(s): 1 Stepping:r0p1 BogoMIPS:2000.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti L1d cache: 1 MiB (16 instances) L1i cache: 1 MiB (16 instances) L2 cache:32 MiB (16 instances) L3 cache:80 MiB (1 instance) NUMA node(s):1 NUMA node0 CPU(s): 0-15 Vulnerability Gather data sampling: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling:Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1:Mitigation; __user pointer sanitization Vulnerability Spectre v2:Mitigation; CSV2, BHB Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Not affected ``` Details ``` Comparing HEAD and feat_rg-reorder-by-statistics Benchmark clickbench_partitioned.json ┏━━━┳━━━┳━━━┳━━━┓ ┃ Query ┃ HEAD ┃ feat_rg-reorder-by-statistics ┃Change ┃ ┡━━━╇━━━╇━━━╇━━━┩ │ QQuery 0 │ 1.17 / 4.64 ±6.78 / 18.19 ms │ 1.18 / 4.64 ±6.77 / 18.18 ms │ no change │ │ QQuery 1 │12.74 / 12.91 ±0.12 / 13.07 ms │13.03 / 13.28 ±0.14 / 13.46 ms │ no change │ │ QQuery 2 │35.86 / 36.47 ±0.48 / 37.25 ms │36.47 / 36.68 ±0.19 / 37.02 ms │ no change │ │ QQuery 3 │30.84 / 31.38 ±0.64 / 32.65 ms │30.91 / 31.05 ±0.08 / 31.14 ms │ no change │ │ QQuery 4 │ 232.70 / 236.25 ±3.06 / 241.83 ms │ 233.22 / 234.32 ±0.79 / 235.30 ms │ no change │ │ QQuery 5 │ 279.72 / 281.78 ±1.81 / 284.54 ms │ 276.91 / 280.80 ±3.00 / 285.58 ms │ no change │ │ QQuery 6 │ 6.77 / 7.03 ±0.19 / 7.28 ms │ 6.67 / 7.32 ±0.40 / 7.93 ms │ no change │ │ QQuery 7 │13.90 / 13.95 ±0.04 / 14.01 ms │14.09 / 14.27 ±0.12 / 14.45 ms │ no change │ │ QQuery 8 │ 318.01 / 323.88 ±3.52 / 327.40 ms │ 315.04 / 318.04 ±2.01 / 320.95 ms │ no change │ │ QQuery 9 │ 455.54 / 459.86 ±3.17 / 463.50 ms │ 446.87 / 455.95 ±4.81 / 461.11 ms │ no change │ │ QQuery 10 │70.11 / 71.04 ±0.81 / 72.49 ms │69.85 / 70.59 ±0.75 / 71.72 ms │ no change │ │ QQuery 11 │79.31 / 81.02 ±1.24 / 82.69 ms │81.09 / 81.49 ±0.30 / 81.82 ms │ no change │ │ QQuery 12 │ 271.17 / 277.33 ±5.06 / 286.31 ms │ 271.78 / 276.31 ±3.98 / 282.18 ms │ no change │ │ QQuery 13 │ 385.11 / 389.27 ±4.84 / 398.56 ms │379.76 / 391.53 ±11.83 / 413.39 ms │ no change │ │ QQuery 14 │ 277.49 / 281.96 ±5.86 / 293.26 ms │ 279.02 / 281.91 ±3.46 / 288.60 ms │ no change │ │
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
adriangbot commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4395264378 🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21956#issuecomment-4395242557) **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4395242557-2045-x5dxd 6.12.68+ #1 SMP Wed Apr 1 02:23:28 UTC 2026 aarch64 GNU/Linux` CPU Details (lscpu) ``` Architecture:aarch64 CPU op-mode(s): 64-bit Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Vendor ID: ARM Model name: Neoverse-V2 Model: 1 Thread(s) per core: 1 Core(s) per cluster: 16 Socket(s): - Cluster(s): 1 Stepping:r0p1 BogoMIPS:2000.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti L1d cache: 1 MiB (16 instances) L1i cache: 1 MiB (16 instances) L2 cache:32 MiB (16 instances) L3 cache:80 MiB (1 instance) NUMA node(s):1 NUMA node0 CPU(s): 0-15 Vulnerability Gather data sampling: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling:Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1:Mitigation; __user pointer sanitization Vulnerability Spectre v2:Mitigation; CSV2, BHB Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Not affected ``` Comparing feat/rg-reorder-by-statistics (4fac37b08551f666b7377bb44a88a6cf86f22eee) to 3b634aa (merge-base) [diff](https://github.com/apache/datafusion/compare/3b634aaef9c2b82b41180d230807c4658e30e2f4..4fac37b08551f666b7377bb44a88a6cf86f22eee) using: clickbench_partitioned Results will be posted here when complete --- [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
zhuqi-lucas commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4395242557 run benchmark clickbench_partitioned -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
adriangbot commented on PR #21956:
URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4395050013
Benchmark for [this
request](https://github.com/apache/datafusion/pull/21956#issuecomment-4395019893)
failed.
Last 20 lines of output:
Click to expand
```
Compiling async-compression v0.4.42
Compiling mimalloc v0.1.50
Compiling parquet v58.2.0
Compiling datafusion-common v53.1.0
(/workspace/datafusion-branch/datafusion/common)
Compiling datafusion-expr-common v53.1.0
(/workspace/datafusion-branch/datafusion/expr-common)
Compiling datafusion-physical-expr-common v53.1.0
(/workspace/datafusion-branch/datafusion/physical-expr-common)
Compiling datafusion-functions-aggregate-common v53.1.0
(/workspace/datafusion-branch/datafusion/functions-aggregate-common)
Compiling datafusion-functions-window-common v53.1.0
(/workspace/datafusion-branch/datafusion/functions-window-common)
Compiling datafusion-expr v53.1.0
(/workspace/datafusion-branch/datafusion/expr)
Compiling datafusion-physical-expr v53.1.0
(/workspace/datafusion-branch/datafusion/physical-expr)
Compiling datafusion-execution v53.1.0
(/workspace/datafusion-branch/datafusion/execution)
error[E0063]: missing fields `fetch` and `sort_options` in initializer of
`DynamicFilterPhysicalExpr`
--> datafusion/physical-expr/src/expressions/dynamic_filters.rs:457:9
|
457 | Self {
| missing `fetch` and `sort_options`
For more information about this error, try `rustc --explain E0063`.
error: could not compile `datafusion-physical-expr` (lib) due to 1 previous
error
warning: build failed, waiting for other jobs to finish...
```
---
[File an issue](https://github.com/adriangb/datafusion-benchmarking/issues)
against this benchmark runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
github-actions[bot] commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4395048896 Thank you for opening this pull request! Reviewer note: [cargo-semver-checks](https://github.com/obi1kenobi/cargo-semver-checks) reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch). Details ``` Cloning apache/main Building datafusion v53.1.0 (current) error: running cargo-doc on crate 'datafusion' failed with output: - Compiling proc-macro2 v1.0.106 Compiling quote v1.0.45 Compiling unicode-ident v1.0.24 Compiling libc v0.2.186 Checking cfg-if v1.0.4 Compiling autocfg v1.5.0 Compiling shlex v1.3.0 Compiling find-msvc-tools v0.1.9 Compiling libm v0.2.16 Compiling num-traits v0.2.19 Compiling syn v2.0.117 Checking memchr v2.8.0 Compiling jobserver v0.1.34 Compiling version_check v0.9.5 Compiling cc v1.2.61 Compiling zerocopy v0.8.48 Checking bytes v1.11.1 Compiling serde_core v1.0.228 Checking itoa v1.0.18 Compiling getrandom v0.3.4 Checking once_cell v1.21.4 Compiling zmij v1.0.21 Checking num-integer v0.1.46 Compiling serde v1.0.228 Compiling serde_json v1.0.149 Checking num-bigint v0.4.6 Checking equivalent v1.0.2 Checking iana-time-zone v0.1.65 Checking allocator-api2 v0.2.21 Checking foldhash v0.2.0 Compiling pkg-config v0.3.33 Checking siphasher v1.0.3 Checking phf_shared v0.12.1 Checking hashbrown v0.17.0 Checking chrono v0.4.44 Compiling synstructure v0.13.2 Compiling ahash v0.8.12 Checking stable_deref_trait v1.2.1 Compiling chrono-tz v0.10.4 Checking phf v0.12.1 Checking num-complex v0.4.6 Compiling zstd-sys v2.0.16+zstd.1.5.7 Checking pin-project-lite v0.2.17 Compiling zstd-safe v7.2.4 Checking futures-sink v0.3.32 Checking smallvec v1.15.1 Checking lexical-util v1.0.7 Compiling zerocopy-derive v0.8.48 Compiling zerofrom-derive v0.1.7 Compiling serde_derive v1.0.228 Checking zerofrom v0.1.7 Compiling yoke-derive v0.8.2 Checking yoke v0.8.2 Compiling zerovec-derive v0.11.3 Checking arrow-schema v58.2.0 Checking half v2.7.1 Checking arrow-buffer v58.2.0 Compiling displaydoc v0.2.5 Checking zerovec v0.11.6 Checking arrow-data v58.2.0 Checking writeable v0.6.3 Checking tinystr v0.8.3 Compiling object v0.37.3 Checking litemap v0.8.2 Checking futures-core v0.3.32 Checking icu_locale_core v2.2.0 Checking arrow-array v58.2.0 Checking potential_utf v0.1.5 Checking zerotrie v0.2.4 Compiling icu_properties_data v2.2.0 Compiling icu_normalizer_data v2.2.0 Checking utf8_iter v1.0.4 Checking icu_collections v2.2.0 Checking icu_provider v2.2.0 Compiling tokio-macros v2.7.0 Checking arrow-select v58.2.0 Compiling crc32fast v1.5.0 Compiling semver v1.0.28 Compiling rustc_version v0.4.1 Checking tokio v1.52.2 Checking futures-channel v0.3.32 Checking lexical-parse-integer v1.0.6 Checking lexical-write-integer v1.0.6 Compiling futures-macro v0.3.32 Checking futures-io v0.3.32 Checking futures-task v0.3.32 Checking adler2 v2.0.1 Checking bitflags v2.11.1 Compiling parking_lot_core v0.9.12 Checking slab v0.4.12 Compiling ar_archive_writer v0.5.1 Checking simd-adler32 v0.3.9 Checking futures-util v0.3.32 Checking miniz_oxide v0.8.9 Checking lexical-write-float v1.0.6 Checking lexical-parse-float v1.0.6 Compiling psm v0.1.31 Checking icu_normalizer v2.2.0 Checking icu_properties v2.2.0 Compiling flatbuffers v25.12.19 Checking aho-corasick v1.1.4 Checking base64 v0.22.1 Checking regex-syntax v0.8.10 Checking zlib-rs v0.6.3 Checking unicode-width v0.2.2 Checking zstd v0.13.3 Checking ryu v1.0.23 Compiling getrandom v0.4.2 Checking scopeguard v1.2.0 Checking unicode-segmentation v1.13.2 Checking lock_api v0.4.14 Checking idna_adapter v1.2.2 Checking comfy-table v7.2.2 Checking lexical-core v1.0.6 Checking arrow-ord v58.2.0 Checking indexmap v2.14.0 Checking flate2 v1.1.9 Compiling stacker v0.1.24 Checking atoi v2.0.0 Checking twox-hash v2.1.2 Compiling thiserror v2.0.18 Checking regex-automata v0.4.14 Checking alloc-no-stdlib v2.0.4 Checking percent-encoding v2.3.2 Compiling snap v1.1.1 Checking form_urlencoded v1.2.2 Checking alloc-std
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
adriangbot commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4395037672 🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21956#issuecomment-4395019893) **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4395019893-2044-wkdsl 6.12.68+ #1 SMP Wed Apr 1 02:23:28 UTC 2026 aarch64 GNU/Linux` CPU Details (lscpu) ``` Architecture:aarch64 CPU op-mode(s): 64-bit Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Vendor ID: ARM Model name: Neoverse-V2 Model: 1 Thread(s) per core: 1 Core(s) per cluster: 16 Socket(s): - Cluster(s): 1 Stepping:r0p1 BogoMIPS:2000.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti L1d cache: 1 MiB (16 instances) L1i cache: 1 MiB (16 instances) L2 cache:32 MiB (16 instances) L3 cache:80 MiB (1 instance) NUMA node(s):1 NUMA node0 CPU(s): 0-15 Vulnerability Gather data sampling: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling:Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1:Mitigation; __user pointer sanitization Vulnerability Spectre v2:Mitigation; CSV2, BHB Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Not affected ``` Comparing feat/rg-reorder-by-statistics (c8fd32187ffb96555a5c37980f8e4b79dcb94f5c) to 0c38ebb (merge-base) [diff](https://github.com/apache/datafusion/compare/0c38ebba110104b84bb923246e04871008684a1d..c8fd32187ffb96555a5c37980f8e4b79dcb94f5c) using: clickbench_partitioned Results will be posted here when complete --- [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
zhuqi-lucas commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4395018952 > the benchmark results look mixed -- I'll see if that reproduces on a second run Thanks @alamb , i update the PR to skip reorder when overlap is heavy, and let me trigger again to see if it's the reason. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
zhuqi-lucas commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4395019893 run benchmark clickbench_partitioned -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
adriangbot commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4391944370 🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21956#issuecomment-4391783062) **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) CPU Details (lscpu) ``` Architecture:aarch64 CPU op-mode(s): 64-bit Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Vendor ID: ARM Model name: Neoverse-V2 Model: 1 Thread(s) per core: 1 Core(s) per cluster: 16 Socket(s): - Cluster(s): 1 Stepping:r0p1 BogoMIPS:2000.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti L1d cache: 1 MiB (16 instances) L1i cache: 1 MiB (16 instances) L2 cache:32 MiB (16 instances) L3 cache:80 MiB (1 instance) NUMA node(s):1 NUMA node0 CPU(s): 0-15 Vulnerability Gather data sampling: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling:Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1:Mitigation; __user pointer sanitization Vulnerability Spectre v2:Mitigation; CSV2, BHB Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Not affected ``` Details ``` Comparing HEAD and feat_rg-reorder-by-statistics Benchmark clickbench_partitioned.json ┏━━━┳━━━┳━━━┳━━━┓ ┃ Query ┃ HEAD ┃ feat_rg-reorder-by-statistics ┃Change ┃ ┡━━━╇━━━╇━━━╇━━━┩ │ QQuery 0 │ 1.20 / 4.70 ±6.87 / 18.44 ms │ 1.22 / 4.79 ±6.97 / 18.74 ms │ no change │ │ QQuery 1 │12.75 / 13.17 ±0.22 / 13.34 ms │12.40 / 12.90 ±0.28 / 13.23 ms │ no change │ │ QQuery 2 │36.81 / 37.18 ±0.33 / 37.61 ms │37.22 / 37.42 ±0.25 / 37.88 ms │ no change │ │ QQuery 3 │31.71 / 32.19 ±0.54 / 33.22 ms │31.94 / 32.44 ±0.39 / 32.96 ms │ no change │ │ QQuery 4 │ 247.78 / 250.01 ±2.06 / 253.50 ms │ 253.69 / 257.03 ±1.73 / 258.56 ms │ no change │ │ QQuery 5 │ 287.78 / 289.52 ±1.03 / 290.85 ms │ 289.42 / 292.64 ±2.07 / 295.08 ms │ no change │ │ QQuery 6 │ 7.29 / 7.81 ±0.67 / 9.10 ms │ 6.90 / 7.16 ±0.18 / 7.46 ms │ +1.09x faster │ │ QQuery 7 │14.28 / 14.36 ±0.07 / 14.49 ms │13.95 / 14.08 ±0.09 / 14.16 ms │ no change │ │ QQuery 8 │ 336.38 / 340.04 ±2.13 / 342.05 ms │ 341.09 / 343.49 ±1.90 / 345.33 ms │ no change │ │ QQuery 9 │ 461.94 / 467.43 ±5.31 / 477.00 ms │ 467.44 / 472.74 ±3.67 / 477.75 ms │ no change │ │ QQuery 10 │74.48 / 76.10 ±1.88 / 79.71 ms │75.52 / 77.16 ±2.19 / 81.23 ms │ no change │ │ QQuery 11 │85.73 / 86.07 ±0.26 / 86.42 ms │85.99 / 87.54 ±1.12 / 89.28 ms │ no change │ │ QQuery 12 │ 280.98 / 285.44 ±3.52 / 290.79 ms │ 282.81 / 287.61 ±5.21 / 296.21 ms │ no change │ │ QQuery 13 │ 404.18 / 411.13 ±9.26 / 429.29 ms │ 405.10 / 414.80 ±8.21 / 425.37 ms │ no change │ │ QQuery 14 │ 292.37 / 297.84 ±3.26 / 301.90 ms │ 293.62 / 295.19 ±1.64 / 297.82 ms │ no change │ │
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
adriangbot commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4391811049 🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21956#issuecomment-4391783062) **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4391783062-2042-6k9px 6.12.68+ #1 SMP Wed Apr 1 02:23:28 UTC 2026 aarch64 GNU/Linux` CPU Details (lscpu) ``` Architecture:aarch64 CPU op-mode(s): 64-bit Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Vendor ID: ARM Model name: Neoverse-V2 Model: 1 Thread(s) per core: 1 Core(s) per cluster: 16 Socket(s): - Cluster(s): 1 Stepping:r0p1 BogoMIPS:2000.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti L1d cache: 1 MiB (16 instances) L1i cache: 1 MiB (16 instances) L2 cache:32 MiB (16 instances) L3 cache:80 MiB (1 instance) NUMA node(s):1 NUMA node0 CPU(s): 0-15 Vulnerability Gather data sampling: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling:Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1:Mitigation; __user pointer sanitization Vulnerability Spectre v2:Mitigation; CSV2, BHB Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Not affected ``` Comparing feat/rg-reorder-by-statistics (f7d9156d647accad823dcfce7c2b95ac386c08c0) to 0144570 (merge-base) [diff](https://github.com/apache/datafusion/compare/014457014d28806cee34c51ebf3587148188a303..f7d9156d647accad823dcfce7c2b95ac386c08c0) using: clickbench_partitioned Results will be posted here when complete --- [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
alamb commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4391783062 run benchmark clickbench_partitioned -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
alamb commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4391781710 the benchmark results look mixed -- I'll see if that reproduces on a second run -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
zhuqi-lucas commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4377675107 > Sorry @zhuqi-lucas - I am really struggling these days tor review all these large PRs. Last year it was very rare to see a more than 1000 line PR and now we get multiple such PRs a day. > > I will try and find the time to review this more carefuly Thanks @alamb, no rush at all, I really appreciate you taking the time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
adriangbot commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4373402320 🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21956#issuecomment-4373271959) **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) CPU Details (lscpu) ``` Architecture:aarch64 CPU op-mode(s): 64-bit Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Vendor ID: ARM Model name: Neoverse-V2 Model: 1 Thread(s) per core: 1 Core(s) per cluster: 16 Socket(s): - Cluster(s): 1 Stepping:r0p1 BogoMIPS:2000.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti L1d cache: 1 MiB (16 instances) L1i cache: 1 MiB (16 instances) L2 cache:32 MiB (16 instances) L3 cache:80 MiB (1 instance) NUMA node(s):1 NUMA node0 CPU(s): 0-15 Vulnerability Gather data sampling: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling:Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1:Mitigation; __user pointer sanitization Vulnerability Spectre v2:Mitigation; CSV2, BHB Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Not affected ``` Details ``` Comparing HEAD and feat_rg-reorder-by-statistics Benchmark clickbench_partitioned.json ┏━━━┳━━━┳━━━┳━━━┓ ┃ Query ┃ HEAD ┃ feat_rg-reorder-by-statistics ┃Change ┃ ┡━━━╇━━━╇━━━╇━━━┩ │ QQuery 0 │ 1.23 / 4.82 ±7.04 / 18.90 ms │ 1.21 / 4.82 ±7.05 / 18.92 ms │ no change │ │ QQuery 1 │12.47 / 13.00 ±0.29 / 13.32 ms │12.72 / 12.98 ±0.18 / 13.19 ms │ no change │ │ QQuery 2 │38.63 / 39.11 ±0.35 / 39.47 ms │37.97 / 38.31 ±0.34 / 38.80 ms │ no change │ │ QQuery 3 │33.07 / 33.52 ±0.51 / 34.49 ms │33.41 / 33.64 ±0.22 / 33.98 ms │ no change │ │ QQuery 4 │ 242.23 / 249.99 ±9.73 / 267.45 ms │ 271.06 / 274.49 ±3.09 / 278.63 ms │ 1.10x slower │ │ QQuery 5 │ 284.47 / 296.56 ±9.28 / 306.71 ms │ 299.65 / 308.25 ±4.93 / 314.88 ms │ no change │ │ QQuery 6 │ 6.96 / 7.23 ±0.18 / 7.46 ms │ 6.65 / 7.27 ±0.56 / 8.05 ms │ no change │ │ QQuery 7 │14.21 / 14.72 ±0.31 / 15.02 ms │14.41 / 16.07 ±3.18 / 22.43 ms │ 1.09x slower │ │ QQuery 8 │ 356.10 / 358.94 ±2.33 / 363.04 ms │ 328.55 / 340.34 ±8.92 / 354.19 ms │ +1.05x faster │ │ QQuery 9 │ 487.77 / 497.42 ±8.53 / 511.30 ms │ 439.88 / 453.05 ±9.21 / 467.33 ms │ +1.10x faster │ │ QQuery 10 │76.69 / 80.03 ±5.15 / 90.25 ms │74.55 / 77.13 ±3.18 / 83.38 ms │ no change │ │ QQuery 11 │84.54 / 86.98 ±1.52 / 88.97 ms │86.04 / 87.25 ±1.24 / 89.04 ms │ no change │ │ QQuery 12 │ 282.89 / 284.46 ±1.23 / 285.94 ms │275.05 / 289.89 ±11.97 / 305.74 ms │ no change │ │ QQuery 13 │ 396.27 / 404.06 ±5.67 / 410.90 ms │ 403.21 / 412.12 ±9.24 / 429.06 ms │ no change │ │ QQuery 14 │ 282.35 / 286.07 ±3.02 / 291.18 ms │ 287.05 / 296.09 ±7.05 / 306.18 ms │ no change │ │
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
adriangbot commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4373296896 🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21956#issuecomment-4373271959) **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4373271959-2014-w8h9b 6.12.68+ #1 SMP Wed Apr 1 02:23:28 UTC 2026 aarch64 GNU/Linux` CPU Details (lscpu) ``` Architecture:aarch64 CPU op-mode(s): 64-bit Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Vendor ID: ARM Model name: Neoverse-V2 Model: 1 Thread(s) per core: 1 Core(s) per cluster: 16 Socket(s): - Cluster(s): 1 Stepping:r0p1 BogoMIPS:2000.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti L1d cache: 1 MiB (16 instances) L1i cache: 1 MiB (16 instances) L2 cache:32 MiB (16 instances) L3 cache:80 MiB (1 instance) NUMA node(s):1 NUMA node0 CPU(s): 0-15 Vulnerability Gather data sampling: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling:Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1:Mitigation; __user pointer sanitization Vulnerability Spectre v2:Mitigation; CSV2, BHB Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Not affected ``` Comparing feat/rg-reorder-by-statistics (f7d9156d647accad823dcfce7c2b95ac386c08c0) to 0144570 (merge-base) [diff](https://github.com/apache/datafusion/compare/014457014d28806cee34c51ebf3587148188a303..f7d9156d647accad823dcfce7c2b95ac386c08c0) using: clickbench_partitioned Results will be posted here when complete --- [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
alamb commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4373271959 run benchmark clickbench_partitioned -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
alamb commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4373268994 Sorry @zhuqi-lucas - I am really struggling these days tor review all these large PRs. Last year it was very rare to see a more than 1000 line PR and now we get multiple such PRs a day. I will try and find the time to review this more carefuly -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
zhuqi-lucas commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4351600543 The benchmark results are expected — RG reorder alone doesn't skip any row groups, it only changes the read order so that TopK's dynamic filter threshold converges faster. The significant speedup (2-3x on `sort_pushdown_inexact`) comes from **stats init + cumulative RG prune** which will be in the follow-up PR. Those optimizations depend on RG reorder as a foundation: 1. **RG reorder**: put best RGs first (this PR) 2. **Stats init**: initialize TopK threshold from RG statistics before reading → prune RGs upfront (next PR) 3. **Cumulative prune**: after reorder, truncate remaining RGs once enough rows are collected (next PR) Without reorder, cumulative prune might truncate the wrong RGs. Reorder ensures the best RGs come first, making truncation safe and effective. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
adriangbot commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4351579855 🤖 Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/21956#issuecomment-4351456416) **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) CPU Details (lscpu) ``` Architecture:aarch64 CPU op-mode(s): 64-bit Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Vendor ID: ARM Model name: Neoverse-V2 Model: 1 Thread(s) per core: 1 Core(s) per cluster: 16 Socket(s): - Cluster(s): 1 Stepping:r0p1 BogoMIPS:2000.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti L1d cache: 1 MiB (16 instances) L1i cache: 1 MiB (16 instances) L2 cache:32 MiB (16 instances) L3 cache:80 MiB (1 instance) NUMA node(s):1 NUMA node0 CPU(s): 0-15 Vulnerability Gather data sampling: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling:Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1:Mitigation; __user pointer sanitization Vulnerability Spectre v2:Mitigation; CSV2, BHB Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Not affected ``` Details ``` Comparing HEAD and feat_rg-reorder-by-statistics Benchmark sort_pushdown_inexact.json ┏━━━┳┳┳━━━┓ ┃ Query ┃ HEAD ┃ feat_rg-reorder-by-statistics ┃ Change ┃ ┡━━━╇╇╇━━━┩ │ Q1│7.22 / 8.08 ±0.90 / 9.82 ms │ 6.71 / 7.47 ±1.27 / 10.00 ms │ +1.08x faster │ │ Q2│6.79 / 6.88 ±0.08 / 7.01 ms │6.87 / 7.08 ±0.23 / 7.48 ms │ no change │ │ Q3│ 21.76 / 22.52 ±0.64 / 23.37 ms │ 21.99 / 22.88 ±0.57 / 23.66 ms │ no change │ │ Q4│ 20.82 / 21.86 ±0.84 / 22.82 ms │ 20.73 / 21.74 ±0.77 / 22.71 ms │ no change │ └───┴┴┴───┘ ┏━━┳━┓ ┃ Benchmark Summary┃ ┃ ┡━━╇━┩ │ Total Time (HEAD)│ 59.34ms │ │ Total Time (feat_rg-reorder-by-statistics) │ 59.17ms │ │ Average Time (HEAD) │ 14.83ms │ │ Average Time (feat_rg-reorder-by-statistics) │ 14.79ms │ │ Queries Faster │ 1 │ │ Queries Slower │ 0 │ │ Queries with No Change │ 3 │ │ Queries with Failure │ 0 │ └──┴─┘ ``` Resource Usage **sort_pushdown_inexact — base (merge-base)** | Metric | Value | ||---| | Wall time | 5.0s | | Peak memory | 4.6 GiB | | Avg memory | 4.6 GiB | | CPU user | 2.5s | | CPU sys | 0.4s | | Peak spill | 0 B | **sort_pushdown_inexact — branch** | Metric | Value | ||---| | Wall time | 5.0s | | Peak memory | 4.6 GiB | | Avg memory | 4.6 GiB | | CPU user | 2.6s | | CPU sys | 0.3s | | Peak spill | 0 B | --
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
adriangbot commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4351480738 🤖 Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/21956#issuecomment-4351456416) **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c4351456416-1947-jscm6 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux` CPU Details (lscpu) ``` Architecture:aarch64 CPU op-mode(s): 64-bit Byte Order: Little Endian CPU(s): 16 On-line CPU(s) list: 0-15 Vendor ID: ARM Model name: Neoverse-V2 Model: 1 Thread(s) per core: 1 Core(s) per cluster: 16 Socket(s): - Cluster(s): 1 Stepping:r0p1 BogoMIPS:2000.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti L1d cache: 1 MiB (16 instances) L1i cache: 1 MiB (16 instances) L2 cache:32 MiB (16 instances) L3 cache:80 MiB (1 instance) NUMA node(s):1 NUMA node0 CPU(s): 0-15 Vulnerability Gather data sampling: Not affected Vulnerability Indirect target selection: Not affected Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Not affected Vulnerability Reg file data sampling:Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Vulnerability Spectre v1:Mitigation; __user pointer sanitization Vulnerability Spectre v2:Mitigation; CSV2, BHB Vulnerability Srbds: Not affected Vulnerability Tsa: Not affected Vulnerability Tsx async abort: Not affected Vulnerability Vmscape: Not affected ``` Comparing feat/rg-reorder-by-statistics (235c4e1f7ba10ba5bb4f3f9464f9ca7e27841ca3) to 0144570 (merge-base) [diff](https://github.com/apache/datafusion/compare/014457014d28806cee34c51ebf3587148188a303..235c4e1f7ba10ba5bb4f3f9464f9ca7e27841ca3) using: sort_pushdown_inexact Results will be posted here when complete --- [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] feat: globally reorder files and row groups by statistics for TopK queries [datafusion]
zhuqi-lucas commented on PR #21956: URL: https://github.com/apache/datafusion/pull/21956#issuecomment-4351456416 run benchmark sort_pushdown_inexact -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
