adriangb commented on PR #25715: URL: https://github.com/apache/datafusion/pull/25715#issuecomment-5820571305
## Benchmark summary PR head 32d0a7b5c2 (stack + `adaptive` default) vs merge-base 95bb0a0 (`main`). PR side always has `pushdown_filters = true`. Times: min of 5 iterations (ms). Ratio = PR / main; **bold** = more than 10% change. Triggers: [main pushdown ON](https://github.com/apache/datafusion/pull/25715#issuecomment-5820225937), [main pushdown OFF](https://github.com/apache/datafusion/pull/25715#issuecomment-5820226337). Both comparisons run in one job each (not cross-run). ### Totals | Suite | main ON | main OFF | PR (ON) vs main ON | faster / slower >7% | PR (ON) vs main OFF | faster / slower >7% | results | |---|---|---|---|---|---|---|---| | TPC-H SF1 | 863 | 751 | 841 (0.97) | 2 / 0 | 841 (**1.12**) | 1 / 14 | [vs ON](https://github.com/apache/datafusion/pull/25715#issuecomment-5820396694), [vs OFF](https://github.com/apache/datafusion/pull/25715#issuecomment-5820396143) | | TPC-H SF10 | 8,060 | 6,351 | 7,322 (**0.91**) | 8 / 2 | 7,082 (**1.12**) | 2 / 16 | [vs ON](https://github.com/apache/datafusion/pull/25715#issuecomment-5820419431), [vs OFF](https://github.com/apache/datafusion/pull/25715#issuecomment-5820413781) | | TPC-DS SF1 | 8,328 | 8,951 | 7,554 (**0.91**) | 18 / 1 | 7,095 (**0.79**) | 33 / 33 | [vs ON](https://github.com/apache/datafusion/pull/25715#issuecomment-5820423392), [vs OFF](https://github.com/apache/datafusion/pull/25715#issuecomment-5820416293) | | ClickBench partitioned | 16,655 | 19,476 | 16,675 (1.00) | 1 / 0 | 16,172 (**0.83**) | 8 / 13 | [vs ON](https://github.com/apache/datafusion/pull/25715#issuecomment-5820476044), [vs OFF](https://github.com/apache/datafusion/pull/25715#issuecomment-5820463824) | ### Key queries `PR (ON)` is from the vs-ON job. `PR / main OFF` uses the PR time of the vs-OFF job. | Query | main ON | main OFF | PR (ON) | PR / main ON | PR / main OFF | |---|---|---|---|---|---| | TPC-H SF1 Q17 | 72.7 | 75.1 | 75.8 | 1.04 | 1.01 | | TPC-H SF1 Q18 | 76.1 | 63.4 | 59.6 | **0.78** | 0.92 | | TPC-H SF10 Q17 | 501 | 574 | 613 | **1.22** | 1.01 | | TPC-H SF10 Q18 | 663 | 703 | 486 | **0.73** | **0.67** | | TPC-DS Q17 | 122 | 78.7 | 91.5 | **0.75** | **1.10** | | TPC-DS Q25 | 120 | 106 | 98.0 | **0.82** | **0.89** | | TPC-DS Q29 | 151 | 92.9 | 106 | **0.71** | 1.08 | | TPC-DS Q50 | 128 | 57.7 | 70.5 | **0.55** | **1.20** | | TPC-DS Q64 | 541 | 357 | 328 | **0.61** | **0.87** | | TPC-DS Q72 | 181 | 1,746 | 112 | **0.62** | **0.06** | | TPC-DS Q2 | 47.9 | 80.5 | 46.7 | 0.98 | **0.56** | | TPC-DS Q24 | 182 | 190 | 184 | 1.01 | **0.85** | | TPC-DS Q45 | 27.5 | 36.4 | 27.1 | 0.99 | **0.68** | | TPC-DS Q65 | 164 | 131 | 159 | 0.97 | **1.20** | | ClickBench Q10 | 94.0 | 65.4 | 89.7 | 0.95 | **1.35** | | ClickBench Q23 | 134 | 3,080 | 142 | 1.06 | **0.05** | | ClickBench Q24 | 44.6 | 40.3 | 44.7 | 1.00 | 1.09 | | ClickBench Q25 | 135 | 106 | 135 | 1.00 | **1.28** | | ClickBench Q26 | 47.9 | 40.8 | 48.3 | 1.01 | **1.17** | <details><summary>All queries with more than 7% change vs main pushdown ON</summary> | Suite | Query | main | PR | ratio | |---|---|---|---|---| | TPC-H SF1 | Q18 | 76.1 | 59.6 | **0.78** | | TPC-H SF1 | Q12 | 40.2 | 34.5 | **0.86** | | TPC-H SF10 | Q22 | 131 | 74.2 | **0.57** | | TPC-H SF10 | Q4 | 294 | 213 | **0.72** | | TPC-H SF10 | Q21 | 1,233 | 900 | **0.73** | | TPC-H SF10 | Q18 | 663 | 486 | **0.73** | | TPC-H SF10 | Q13 | 423 | 327 | **0.77** | | TPC-H SF10 | Q16 | 89.3 | 72.2 | **0.81** | | TPC-H SF10 | Q12 | 301 | 252 | **0.84** | | TPC-H SF10 | Q20 | 281 | 252 | **0.90** | | TPC-H SF10 | Q7 | 440 | 496 | **1.13** | | TPC-H SF10 | Q17 | 501 | 613 | **1.22** | | TPC-DS | Q50 | 128 | 70.5 | **0.55** | | TPC-DS | Q64 | 541 | 328 | **0.61** | | TPC-DS | Q72 | 181 | 112 | **0.62** | | TPC-DS | Q95 | 94.7 | 65.8 | **0.69** | | TPC-DS | Q29 | 151 | 106 | **0.71** | | TPC-DS | Q18 | 170 | 122 | **0.72** | | TPC-DS | Q85 | 169 | 127 | **0.75** | | TPC-DS | Q17 | 122 | 91.5 | **0.75** | | TPC-DS | Q25 | 120 | 98.0 | **0.82** | | TPC-DS | Q6 | 38.5 | 32.4 | **0.84** | | TPC-DS | Q75 | 189 | 164 | **0.87** | | TPC-DS | Q15 | 26.1 | 22.8 | **0.87** | | TPC-DS | Q99 | 62.3 | 55.1 | **0.89** | | TPC-DS | Q58 | 74.9 | 67.2 | **0.90** | | TPC-DS | Q14 | 452 | 409 | **0.91** | | TPC-DS | Q4 | 402 | 364 | **0.91** | | TPC-DS | Q74 | 157 | 144 | 0.92 | | TPC-DS | Q70 | 126 | 117 | 0.92 | | TPC-DS | Q54 | 28.5 | 31.0 | 1.09 | | ClickBench | Q39 | 137 | 123 | **0.89** | </details> <details><summary>All queries with more than 10% change vs main pushdown OFF</summary> | Suite | Query | main | PR | ratio | |---|---|---|---|---| | TPC-H SF1 | Q14 | 24.7 | 27.3 | **1.11** | | TPC-H SF1 | Q3 | 28.6 | 31.6 | **1.11** | | TPC-H SF1 | Q16 | 14.3 | 16.0 | **1.12** | | TPC-H SF1 | Q10 | 40.9 | 46.9 | **1.15** | | TPC-H SF1 | Q8 | 40.3 | 46.5 | **1.15** | | TPC-H SF1 | Q11 | 13.7 | 15.8 | **1.15** | | TPC-H SF1 | Q21 | 55.7 | 66.5 | **1.19** | | TPC-H SF1 | Q22 | 14.2 | 17.0 | **1.20** | | TPC-H SF1 | Q5 | 35.5 | 43.5 | **1.23** | | TPC-H SF1 | Q9 | 50.1 | 62.5 | **1.25** | | TPC-H SF1 | Q15 | 31.1 | 42.0 | **1.35** | | TPC-H SF1 | Q6 | 16.5 | 22.7 | **1.37** | | TPC-H SF1 | Q4 | 17.8 | 26.0 | **1.46** | | TPC-H SF1 | Q12 | 21.2 | 34.5 | **1.63** | | TPC-H SF10 | Q18 | 703 | 468 | **0.67** | | TPC-H SF10 | Q20 | 273 | 245 | **0.89** | | TPC-H SF10 | Q8 | 355 | 391 | **1.10** | | TPC-H SF10 | Q16 | 62.9 | 70.0 | **1.11** | | TPC-H SF10 | Q2 | 90.0 | 103 | **1.15** | | TPC-H SF10 | Q9 | 511 | 587 | **1.15** | | TPC-H SF10 | Q22 | 62.1 | 72.5 | **1.17** | | TPC-H SF10 | Q5 | 349 | 422 | **1.21** | | TPC-H SF10 | Q10 | 292 | 361 | **1.23** | | TPC-H SF10 | Q15 | 297 | 376 | **1.26** | | TPC-H SF10 | Q21 | 645 | 874 | **1.36** | | TPC-H SF10 | Q3 | 211 | 296 | **1.40** | | TPC-H SF10 | Q6 | 125 | 179 | **1.43** | | TPC-H SF10 | Q12 | 150 | 247 | **1.64** | | TPC-H SF10 | Q4 | 112 | 210 | **1.88** | | TPC-DS | Q72 | 1,746 | 104 | **0.06** | | TPC-DS | Q15 | 54.4 | 20.8 | **0.38** | | TPC-DS | Q8 | 36.2 | 14.7 | **0.41** | | TPC-DS | Q54 | 53.5 | 28.4 | **0.53** | | TPC-DS | Q2 | 80.5 | 45.3 | **0.56** | | TPC-DS | Q58 | 109 | 64.1 | **0.59** | | TPC-DS | Q45 | 36.4 | 24.9 | **0.68** | | TPC-DS | Q59 | 117 | 80.5 | **0.69** | | TPC-DS | Q78 | 162 | 117 | **0.72** | | TPC-DS | Q11 | 265 | 202 | **0.76** | | TPC-DS | Q4 | 435 | 340 | **0.78** | | TPC-DS | Q80 | 95.4 | 75.3 | **0.79** | | TPC-DS | Q57 | 167 | 134 | **0.81** | | TPC-DS | Q99 | 65.9 | 53.9 | **0.82** | | TPC-DS | Q95 | 77.7 | 64.2 | **0.83** | | TPC-DS | Q40 | 23.7 | 19.8 | **0.84** | | TPC-DS | Q66 | 76.4 | 64.3 | **0.84** | | TPC-DS | Q74 | 157 | 132 | **0.84** | | TPC-DS | Q6 | 36.0 | 30.6 | **0.85** | | TPC-DS | Q24 | 190 | 162 | **0.85** | | TPC-DS | Q62 | 44.9 | 38.3 | **0.85** | | TPC-DS | Q20 | 35.3 | 30.7 | **0.87** | | TPC-DS | Q64 | 357 | 311 | **0.87** | | TPC-DS | Q83 | 32.9 | 28.8 | **0.87** | | TPC-DS | Q25 | 106 | 94.0 | **0.89** | | TPC-DS | Q98 | 42.4 | 38.0 | **0.90** | | TPC-DS | Q71 | 35.0 | 31.8 | **0.91** | | TPC-DS | Q7 | 75.2 | 82.7 | **1.10** | | TPC-DS | Q17 | 78.7 | 86.8 | **1.10** | | TPC-DS | Q21 | 17.0 | 18.8 | **1.10** | | TPC-DS | Q70 | 102 | 114 | **1.11** | | TPC-DS | Q75 | 141 | 157 | **1.11** | | TPC-DS | Q31 | 108 | 120 | **1.12** | | TPC-DS | Q18 | 103 | 116 | **1.12** | | TPC-DS | Q23 | 307 | 351 | **1.14** | | TPC-DS | Q10 | 62.5 | 72.0 | **1.15** | | TPC-DS | Q35 | 70.9 | 82.4 | **1.16** | | TPC-DS | Q85 | 104 | 122 | **1.18** | | TPC-DS | Q76 | 34.5 | 40.5 | **1.18** | | TPC-DS | Q50 | 57.7 | 69.1 | **1.20** | | TPC-DS | Q65 | 131 | 158 | **1.20** | | TPC-DS | Q86 | 25.0 | 30.2 | **1.21** | | TPC-DS | Q89 | 35.2 | 42.7 | **1.21** | | TPC-DS | Q69 | 56.2 | 69.2 | **1.23** | | TPC-DS | Q28 | 59.7 | 73.6 | **1.23** | | TPC-DS | Q87 | 61.1 | 76.6 | **1.25** | | TPC-DS | Q49 | 69.0 | 87.0 | **1.26** | | TPC-DS | Q38 | 59.9 | 76.3 | **1.27** | | TPC-DS | Q48 | 96.2 | 123 | **1.28** | | TPC-DS | Q26 | 46.9 | 64.1 | **1.36** | | TPC-DS | Q82 | 16.4 | 31.1 | **1.90** | | TPC-DS | Q9 | 52.1 | 111 | **2.12** | | ClickBench | Q23 | 3,080 | 139 | **0.05** | | ClickBench | Q22 | 1,011 | 774 | **0.77** | | ClickBench | Q32 | 1,088 | 950 | **0.87** | | ClickBench | Q38 | 40.8 | 35.9 | **0.88** | | ClickBench | Q39 | 153 | 135 | **0.89** | | ClickBench | Q36 | 70.9 | 63.8 | **0.90** | | ClickBench | Q14 | 280 | 310 | **1.11** | | ClickBench | Q13 | 360 | 400 | **1.11** | | ClickBench | Q12 | 262 | 291 | **1.11** | | ClickBench | Q26 | 40.8 | 47.6 | **1.17** | | ClickBench | Q41 | 13.3 | 16.2 | **1.22** | | ClickBench | Q7 | 12.9 | 16.1 | **1.25** | | ClickBench | Q40 | 13.9 | 17.6 | **1.27** | | ClickBench | Q25 | 106 | 136 | **1.28** | | ClickBench | Q11 | 75.9 | 99.3 | **1.31** | | ClickBench | Q10 | 65.4 | 88.3 | **1.35** | </details> ### Notes - vs main pushdown ON: no suite is slower. TPC-H SF10 Q17 is 1.22x slower; Q7 1.13x. All other >10% changes are wins. - vs main pushdown OFF: queries without dynamic filters also regress (TPC-H Q6, one table, no join: 1.37x; TPC-DS Q9, scalar subqueries on one table: 2.12x). Most of these regressions are equal to main pushdown ON (for example ClickBench Q10, Q25, Q26). This cost comes from row-level evaluation of static (required) filters, which `adaptive` does not gate. **Conclusion:** with `pushdown_filters = true`, `adaptive` beats `main` pushdown ON (TPC-DS −9%, TPC-H SF10 −9%, no suite slower). Against today's default (pushdown OFF) it wins on TPC-DS (−21%) and ClickBench (−17%) but loses on TPC-H (+12%), so the success criterion does not hold yet. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
