Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
morrySnow merged PR #67067: URL: https://github.com/apache/doris/pull/67067 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
github-actions[bot] commented on code in PR #67067:
URL: https://github.com/apache/doris/pull/67067#discussion_r4055867998
##
fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java:
##
@@ -1971,7 +1971,7 @@ public void setMaxJoinNumberOfReorder(int
maxJoinNumberOfReorder) {
}
@VarAttrDef.VarAttr(name = MAX_JOIN_NUMBER_BUSHY_TREE)
-private int maxJoinNumBushyTree = 8;
+private int maxJoinNumBushyTree = 9;
Review Comment:
[P2] Keep the nine-way CBO expansion behind a separate control
For `Filter(equalities) -> (... nine inner/cross-join inputs ...)`,
`ReorderJoin` records `maxNAryInnerJoin=9`. Even with
`enable_join_reorder_before_eager_agg=false` (or with no aggregate), this new
default then makes `OptimizeGroupExpressionJob` select the bushy rule set where
the base default selected zig-zag. That adds both association directions and
exchange exploration for queries outside the new feature; a fully connected
nine-input graph already has 9,330 nontrivial subset splits before
orientations/implementations, near the 10,000-expression memo cap. Please
retain 8, gate this independent expansion separately, or document it and add a
reliable-statistics nine-table feature-off planning-budget regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
feiniaofeiafei commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5746971742 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
feiniaofeiafei commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5746967615 run external -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
hello-stephen commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5728439063 TPC-H: Total hot run time: 27882 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit ae153ab8f4a14ee706d379c62a7fe2d94de2643f, data reload: false -- Round 1 -- q1 17622 382538253825 q2 2229366 307 307 q3 10084 1462824 824 q4 4684479 346 346 q5 7483814 561 561 q6 175 167 135 135 q7 724 800 606 606 q8 9289149314431443 q9 5416413341624133 q10 6819133610161016 q11 422 264 242 242 q12 627 422 290 290 q13 18091 258120232023 q14 267 258 235 235 q15 q16 713 716 658 658 q17 1770109610191019 q18 6463563855505550 q19 1290131711081108 q20 523 404 273 273 q21 5773331429862986 q22 456 371 302 302 Total cold run time: 100920 ms Total hot run time: 27882 ms - Round 2, with runtime_filter_mode=off - q1 4625469344034403 q2 728 595 523 523 q3 5189513346524652 q4 2226231114611461 q5 4576435545134355 q6 236 177 128 128 q7 1880169114621462 q8 2323200020062000 q9 7334723071667166 q10 3712354830643064 q11 513 370 355 355 q12 709 707 503 503 q13 2271259020052005 q14 268 282 247 247 q15 q16 653 689 586 586 q17 7283670866416641 q18 11856 11086 11647 11086 q19 1120983 1015983 q20 2188218919101910 q21 4967405042554050 q22 504 446 394 394 Total cold run time: 65161 ms Total hot run time: 57974 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
hello-stephen commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5726498185 # FE UT Coverage Report Increment line coverage ` 83.64% (322/385)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/67067_e0594aa03f5e70519834ca36c4e7497b4fd3ea7d/fe_increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/67067_e0594aa03f5e70519834ca36c4e7497b4fd3ea7d/fe_report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
hello-stephen commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5729566297 TPC-DS: Total hot run time: 152376 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit ae153ab8f4a14ee706d379c62a7fe2d94de2643f, data reload: false query5 4352628 472 472 query6 424 215 192 192 query7 4820571 294 294 query8 320 180 164 164 query9 8789396940343969 query10 464 306 258 258 query11 5956352232113211 query12 149 90 89 89 query13 1253576 413 413 query14 6520455842454245 query14_13998397939583958 query15 200 200 174 174 query16 969 442 401 401 query17 889 680 524 524 query18 2426461 325 325 query19 197 175 137 137 query20 80 79 81 79 query21 223 138 118 118 query22 12982 13005 12871 12871 query23 14067 12937 12419 12419 query23_112587 12465 12560 12465 query24 72771230672 672 query24_1701 698 746 698 query25 554 408 357 357 query26 1267304 159 159 query27 2742578 323 323 query28 4583197219521952 query29 1608712 506 506 query30 292 217 186 186 query31 914 767 652 652 query32 145 98 102 98 query33 538 305 272 272 query34 13251156648 648 query35 747 761 657 657 query36 797 799 726 726 query37 146 119 96 96 query38 1825175717081708 query39 716 667 676 667 query39_1657 648 643 643 query40 226 122 99 99 query41 65 63 62 62 query42 93 96 91 91 query43 338 348 305 305 query44 1370709 712 709 query45 182 177 161 161 query46 10841155732 732 query47 1500148913931393 query48 378 401 292 292 query49 584 410 299 299 query50 954 371 257 257 query51 10536 10230 10485 10230 query52 88 90 77 77 query53 244 250 181 181 query54 253 201 220 201 query55 83 79 69 69 query56 230 226 209 209 query57 1532135312051205 query58 281 269 249 249 query59 2003204518391839 query60 270 236 220 220 query61 147 139 144 139 query62 405 324 271 271 query63 214 184 179 179 query64 27931000885 885 query65 3483342834103410 query66 1813444 324 324 query67 20478 19993 19922 19922 query68 34711601926 926 query69 439 319 279 279 query70 919 856 827 827 query71 318 252 219 219 query72 2936256322962296 query73 817 841 426 426 query74 4657446642804280 query75 2297229119491949 query76 24121138730 730 query77 373 398 290 290 query78 9001891284368436 query79 14151188766 766 query80 635 465 366 366 query81 548 328 288 288 query82 628 158 121 121 query83 272 227 197 197 query84 314 144 112 112 query85 864 479 392 392 query86 342 233 234 233 query87 1978196818471847 query88 3693273127082708 query89 359 287 248 248 query90 1871196 178 178 query91 173 161 129 129 query92 101 83 91 83 query93 14351484857 857 query94 592 349 302 302 query95 673 375 453 375 query96 1007802 337 337 query97 2448244422962296 query98 161 148 151 148 query99 723 728 616 616 Total cold run time: 237642 ms Total hot run time: 152376 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
hello-stephen commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5729617232 ClickBench: Total hot run time: 23.98 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit ae153ab8f4a14ee706d379c62a7fe2d94de2643f, data reload: false query1 0.010.000.01 query2 0.090.050.06 query3 0.260.140.13 query4 1.620.140.14 query5 0.250.220.22 query6 1.160.940.93 query7 0.050.010.01 query8 0.060.040.04 query9 0.400.340.33 query10 0.540.560.58 query11 0.200.140.15 query12 0.190.150.15 query13 0.460.460.47 query14 0.940.950.94 query15 0.600.590.59 query16 0.310.320.31 query17 1.141.071.11 query18 0.230.210.22 query19 2.041.921.90 query20 0.020.020.01 query21 15.46 0.210.14 query22 4.970.050.06 query23 16.16 0.310.12 query24 3.000.440.33 query25 0.110.040.04 query26 0.750.200.16 query27 0.030.030.03 query28 3.570.820.33 query29 12.51 4.113.20 query30 0.280.150.17 query31 2.780.530.30 query32 3.220.590.49 query33 3.223.173.21 query34 15.57 3.963.29 query35 3.233.213.25 query36 0.560.440.43 query37 0.090.070.06 query38 0.050.030.04 query39 0.040.030.03 query40 0.180.140.14 query41 0.100.030.03 query42 0.040.030.03 query43 0.050.030.03 Total cold run time: 96.54 s Total hot run time: 23.98 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
hello-stephen commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5729465211 TPC-H: Total hot run time: 27791 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit ae153ab8f4a14ee706d379c62a7fe2d94de2643f, data reload: false -- Round 1 -- q1 17869 391339613913 q2 2230378 326 326 q3 10012 1435816 816 q4 4683478 350 350 q5 7475833 557 557 q6 186 179 143 143 q7 754 812 598 598 q8 9286152315481523 q9 5402418041764176 q10 6838131910141014 q11 433 269 254 254 q12 644 429 291 291 q13 18023 263020022002 q14 270 254 236 236 q15 q16 726 715 661 661 q17 1772109110141014 q18 6576560955535553 q19 1176127710421042 q20 510 395 272 272 q21 5580292727532753 q22 424 352 297 297 Total cold run time: 100869 ms Total hot run time: 27791 ms - Round 2, with runtime_filter_mode=off - q1 4241406741564067 q2 712 581 535 535 q3 4474486543014301 q4 2228229014631463 q5 4246417041044104 q6 230 179 132 132 q7 1723163414351435 q8 2298255921922192 q9 7651760977007609 q10 3985367831693169 q11 580 415 372 372 q12 726 744 539 539 q13 2503278721792179 q14 301 304 280 280 q15 q16 720 701 627 627 q17 7706720571757175 q18 11994 11124 11882 11124 q19 1198105210961052 q20 2224220219411941 q21 5468450946264509 q22 536 472 408 408 Total cold run time: 65744 ms Total hot run time: 59213 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
hello-stephen commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5729450205 # FE UT Coverage Report Increment line coverage ` 83.64% (322/385)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/67067_ae153ab8f4a14ee706d379c62a7fe2d94de2643f/fe_increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/67067_ae153ab8f4a14ee706d379c62a7fe2d94de2643f/fe_report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
hello-stephen commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5728599554 ClickBench: Total hot run time: 23.95 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit ae153ab8f4a14ee706d379c62a7fe2d94de2643f, data reload: false query1 0.010.010.00 query2 0.110.060.05 query3 0.260.130.14 query4 1.610.140.14 query5 0.230.220.22 query6 1.160.910.92 query7 0.080.010.01 query8 0.060.030.04 query9 0.390.340.34 query10 0.540.550.54 query11 0.200.140.14 query12 0.190.150.15 query13 0.450.460.46 query14 0.950.950.95 query15 0.600.590.58 query16 0.300.320.31 query17 1.051.051.04 query18 0.220.210.20 query19 2.031.951.90 query20 0.020.020.01 query21 15.43 0.240.13 query22 4.830.060.05 query23 16.14 0.310.12 query24 2.960.430.34 query25 0.110.070.04 query26 0.730.210.17 query27 0.040.030.03 query28 3.480.810.35 query29 12.46 4.043.23 query30 0.270.150.17 query31 2.770.560.30 query32 3.220.600.48 query33 3.133.153.18 query34 15.53 3.953.29 query35 3.243.213.22 query36 0.560.450.44 query37 0.090.060.06 query38 0.050.040.04 query39 0.040.030.03 query40 0.180.150.14 query41 0.080.040.03 query42 0.040.030.03 query43 0.040.030.03 Total cold run time: 95.88 s Total hot run time: 23.95 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
hello-stephen commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5728545896 TPC-DS: Total hot run time: 152300 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit ae153ab8f4a14ee706d379c62a7fe2d94de2643f, data reload: false query5 4358596 454 454 query6 431 204 218 204 query7 4867532 281 281 query8 343 191 167 167 query9 8813397839753975 query10 468 314 271 271 query11 5858354032373237 query12 147 96 86 86 query13 1257600 405 405 query14 6492447342374237 query14_13942396339203920 query15 202 196 179 179 query16 966 448 420 420 query17 926 674 548 548 query18 2418463 333 333 query19 202 184 144 144 query20 82 82 81 81 query21 217 136 115 115 query22 13092 12987 12945 12945 query23 14078 13125 12578 12578 query23_112612 12651 12689 12651 query24 72971078658 658 query24_1651 744 695 695 query25 557 439 367 367 query26 1249320 168 168 query27 2700570 328 328 query28 4554196619931966 query29 1637732 529 529 query30 321 226 172 172 query31 888 752 639 639 query32 148 100 94 94 query33 509 310 256 256 query34 11921144628 628 query35 734 757 642 642 query36 791 824 726 726 query37 146 105 90 90 query38 1824175916671667 query39 686 693 632 632 query39_1646 672 684 672 query40 223 125 105 105 query41 74 73 70 70 query42 96 94 97 94 query43 331 341 296 296 query44 1355712 709 709 query45 180 180 167 167 query46 10771174757 757 query47 1475149214021402 query48 389 411 305 305 query49 620 401 286 286 query50 948 343 252 252 query51 10399 10203 10238 10203 query52 89 88 76 76 query53 232 261 182 182 query54 243 212 178 178 query55 82 72 76 72 query56 220 207 227 207 query57 1526125912481248 query58 281 270 248 248 query59 1961209218111811 query60 270 245 218 218 query61 138 149 138 138 query62 395 323 261 261 query63 215 174 180 174 query64 27811007828 828 query65 3450340134153401 query66 1778404 313 313 query67 20241 20231 19873 19873 query68 34071436866 866 query69 416 297 254 254 query70 907 836 808 808 query71 296 233 218 218 query72 2703254222002200 query73 831 733 397 397 query74 4612447642904290 query75 2280226919411941 query76 24141060722 722 query77 353 390 298 298 query78 9043893484638463 query79 13251130676 676 query80 957 462 363 363 query81 578 322 281 281 query82 629 157 126 126 query83 325 220 195 195 query84 317 141 113 113 query85 885 454 370 370 query86 393 257 233 233 query87 1999200718391839 query88 3589273826882688 query89 346 290 245 245 query90 1798176 177 176 query91 169 160 130 130 query92 105 89 87 87 query93 14411513890 890 query94 640 362 303 303 query95 650 395 425 395 query96 972 786 327 327 query97 2439243423222322 query98 163 154 143 143 query99 711 733 609 609 Total cold run time: 236711 ms Total hot run time: 152300 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
feiniaofeiafei commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5727406316 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
feiniaofeiafei commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5727407082 /reivew -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
feiniaofeiafei commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5726321261 run feut -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
feiniaofeiafei commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5726320394 run cloud_p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
hello-stephen commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5725830220 ClickBench: Total hot run time: 24.03 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit e0594aa03f5e70519834ca36c4e7497b4fd3ea7d, data reload: false query1 0.000.000.01 query2 0.090.050.04 query3 0.260.140.13 query4 1.600.130.15 query5 0.240.220.21 query6 1.160.980.93 query7 0.040.010.00 query8 0.050.040.04 query9 0.380.330.33 query10 0.550.540.55 query11 0.200.140.14 query12 0.180.140.14 query13 0.450.460.47 query14 0.950.950.94 query15 0.610.580.60 query16 0.310.310.32 query17 1.101.131.11 query18 0.210.200.20 query19 2.011.951.97 query20 0.020.010.01 query21 15.43 0.230.14 query22 4.860.050.05 query23 16.11 0.310.12 query24 2.950.420.32 query25 0.120.050.04 query26 0.720.200.14 query27 0.040.040.04 query28 3.560.820.37 query29 12.47 4.093.29 query30 0.280.150.16 query31 2.770.560.30 query32 3.210.580.49 query33 3.213.283.11 query34 15.53 3.973.29 query35 3.243.213.21 query36 0.550.440.42 query37 0.090.060.06 query38 0.050.040.04 query39 0.040.020.02 query40 0.170.150.15 query41 0.090.030.03 query42 0.040.030.03 query43 0.050.030.04 Total cold run time: 95.99 s Total hot run time: 24.03 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
hello-stephen commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5725791952 TPC-DS: Total hot run time: 152558 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit e0594aa03f5e70519834ca36c4e7497b4fd3ea7d, data reload: false query5 4305596 476 476 query6 435 214 203 203 query7 4821549 299 299 query8 327 179 157 157 query9 8773395439683954 query10 452 303 250 250 query11 5858353632153215 query12 146 91 84 84 query13 1286590 384 384 query14 6511448942094209 query14_13966399239523952 query15 200 199 180 180 query16 974 464 453 453 query17 916 669 538 538 query18 2433472 346 346 query19 212 180 140 140 query20 84 82 82 82 query21 220 135 114 114 query22 13061 13034 12849 12849 query23 13942 12912 12516 12516 query23_112479 12465 12494 12465 query24 72691157730 730 query24_1709 698 732 698 query25 592 441 373 373 query26 1266303 164 164 query27 2731588 340 340 query28 4573197719301930 query29 1594746 539 539 query30 300 217 187 187 query31 892 756 650 650 query32 146 102 97 97 query33 525 310 248 248 query34 12121091620 620 query35 736 755 653 653 query36 793 838 727 727 query37 148 109 94 94 query38 1839176017211721 query39 682 685 657 657 query39_1629 645 648 645 query40 234 126 104 104 query41 74 72 68 68 query42 99 96 94 94 query43 337 343 299 299 query44 1369709 709 709 query45 185 178 166 166 query46 10931149700 700 query47 1480149713971397 query48 404 418 288 288 query49 616 468 308 308 query50 950 349 253 253 query51 10313 10326 10459 10326 query52 94 88 74 74 query53 234 246 179 179 query54 248 200 183 183 query55 82 73 74 73 query56 232 221 208 208 query57 1574138813361336 query58 289 265 254 254 query59 2008206018531853 query60 282 244 224 224 query61 150 145 143 143 query62 401 315 265 265 query63 210 174 176 174 query64 28201010813 813 query65 3450338733993387 query66 1803423 312 312 query67 19937 20010 20073 20010 query68 32581516969 969 query69 414 308 264 264 query70 943 811 804 804 query71 299 252 215 215 query72 2566246122192219 query73 836 793 404 404 query74 4590448942984298 query75 2291227819391939 query76 23301099691 691 query77 354 401 299 299 query78 9125899984218421 query79 13811186727 727 query80 632 456 366 366 query81 535 324 276 276 query82 616 163 127 127 query83 290 222 204 204 query84 332 140 114 114 query85 847 468 382 382 query86 324 236 236 236 query87 1993196018401840 query88 3629272327082708 query89 356 287 246 246 query90 1887182 171 171 query91 170 157 125 125 query92 104 88 82 82 query93 14771425921 921 query94 525 343 286 286 query95 663 361 434 361 query96 1013762 329 329 query97 2440241223492349 query98 160 152 145 145 query99 712 721 613 613 Total cold run time: 235781 ms Total hot run time: 152558 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
hello-stephen commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5725713306 TPC-H: Total hot run time: 27614 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit e0594aa03f5e70519834ca36c4e7497b4fd3ea7d, data reload: false -- Round 1 -- q1 17662 389638543854 q2 2161367 291 291 q3 10103 1412776 776 q4 4686481 353 353 q5 7475807 550 550 q6 183 173 138 138 q7 727 785 600 600 q8 9294150915321509 q9 5415418641354135 q10 6822133210181018 q11 420 271 245 245 q12 632 414 292 292 q13 18073 262920072007 q14 260 266 242 242 q15 q16 730 715 655 655 q17 1820109110711071 q18 6525559555185518 q19 1196124310151015 q20 471 387 259 259 q21 5374295727912791 q22 422 347 295 295 Total cold run time: 100451 ms Total hot run time: 27614 ms - Round 2, with runtime_filter_mode=off - q1 4262413641564136 q2 710 555 528 528 q3 4458481742874287 q4 2201229514461446 q5 4184410040874087 q6 223 171 127 127 q7 1707160114351435 q8 2184220621302130 q9 7509732174047321 q10 3690362831543154 q11 541 383 359 359 q12 742 726 521 521 q13 2487282521602160 q14 312 308 269 269 q15 q16 686 709 629 629 q17 7889717570467046 q18 11836 1 11878 1 q19 112610901090 q20 2234220219151915 q21 5290441543774377 q22 511 450 409 409 Total cold run time: 64782 ms Total hot run time: 58537 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
github-actions[bot] commented on code in PR #67067:
URL: https://github.com/apache/doris/pull/67067#discussion_r4043927195
##
fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPlanUtil.java:
##
@@ -131,6 +131,8 @@ public class MTMVPlanUtil {
RuleType.ELIMINATE_GROUP_BY_KEY_BY_UNIFORM,
RuleType.ELIMINATE_GROUP_BY_KEY,
RuleType.ELIMINATE_GROUP_BY,
+// Preserve the join structure used by FK-based MV matching.
+RuleType.REORDER_JOIN_BEFORE_EAGER_AGG,
RuleType.SALT_JOIN
Review Comment:
[P2] Retain initial ordering for MTMV refresh tasks
This entry is in the cache-generation list, but
`DISABLE_RULES_WHEN_RUN_MTMV_TASK` aliases that list, so scheduled/manual
refresh plans skip the rule too. Before this PR, `InitJoinOrder` still ran in
those contexts: for a memo-capped `Aggregate -> Join(A_small, F_huge)`, it put
the small input on the right and unlocked the small-broadcast eager path; now
the task has neither that swap nor its replacement, and later CBO cannot rerun
eager aggregation. The cache-matching rationale does not require this execution
mask—refresh planning disables MV rewrite, and IVM signature generation
finishes in `Analyzer` before this `Rewriter` stage. Please split the
cache/task masks (or retain an equivalent task-side pass) and add an `EXPLAIN
REFRESH`/refresh-plan regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
feiniaofeiafei commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5725061885 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
feiniaofeiafei closed pull request #67067: [improvement](eager-agg) Improve eager aggregation with greedy join reorder URL: https://github.com/apache/doris/pull/67067 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
feiniaofeiafei commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5725060776 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
hello-stephen commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5716988619 # FE UT Coverage Report Increment line coverage ` 83.64% (322/385)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/67067_5465d61ed0599094ec775ab68dec8da8a6ff1e1c/fe_increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/67067_5465d61ed0599094ec775ab68dec8da8a6ff1e1c/fe_report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
github-actions[bot] commented on code in PR #67067:
URL: https://github.com/apache/doris/pull/67067#discussion_r4037407345
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownAggThroughJoinOnPkFk.java:
##
@@ -138,17 +140,29 @@ public List buildRules() {
// select primary_table_pk, primary_table_other from primary_table join
foreign_table on pk = fk
// group by pk, primary_table_other_cols;
private LogicalAggregate eliminatePrimaryOutput(LogicalAggregate
agg, Plan child,
-Plan primary, Plan foreign) {
+PrimaryForeignInfo primaryForeignInfo) {
+Set groupBySlots = agg.getGroupByExpressions().stream()
+.map(Slot.class::cast)
+.collect(ImmutableSet.toImmutableSet());
+DataTrait dataTrait = child.getLogicalProperties().getTrait();
+if (!groupByDeterminesForeignKey(groupBySlots,
primaryForeignInfo.foreignKeys, dataTrait)) {
Review Comment:
[P1] Fence non-movable predicates in the PK/FK cluster rebuild
This indirect-FD admission can now rebuild `(P JOIN F) JOIN C` as `(F JOIN
C) JOIN P`, but `InnerJoinCluster` does not reject hash conjuncts containing
`NoneMovableFunction`. For a lower conjunct such as `assert_true(F.row_id > 0,
'bad') = (P.id > 0)`, an F row with no C match throws in the original lower
join but is discarded before the rebuilt top join evaluates the assertion. The
generic join reorder already fences this expression class; apply the same fence
here (or preserve the original boundary) and add a three-table indirect-FD
regression.
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/eageraggregation/ReorderJoinBeforeEagerAgg.java:
##
@@ -0,0 +1,60 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.eageraggregation;
+
+import org.apache.doris.nereids.jobs.JobContext;
+import org.apache.doris.nereids.rules.rewrite.ColumnPruning;
+import org.apache.doris.nereids.rules.rewrite.joinorder.JoinReorderRule;
+import org.apache.doris.nereids.stats.StatsCalculator;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.algebra.CatalogRelation;
+import org.apache.doris.nereids.trees.plans.visitor.CustomRewriter;
+import org.apache.doris.qe.ConnectContext;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.List;
+
+/** Reorder joins before eager aggregation. */
+public class ReorderJoinBeforeEagerAgg implements CustomRewriter {
+private static final Logger LOG =
LoggerFactory.getLogger(ReorderJoinBeforeEagerAgg.class);
+
+@Override
+public Plan rewriteRoot(Plan plan, JobContext jobContext) {
+List scans =
plan.collectToList(CatalogRelation.class::isInstance);
+StatsCalculator.disableJoinReorderIfStatsInvalid(scans,
jobContext.getCascadesContext());
+ConnectContext connectContext =
jobContext.getCascadesContext().getConnectContext();
+if (connectContext.getSessionVariable().isDisableJoinReorder()
Review Comment:
[P1] Preserve the existing initial-reorder control
This replacement no longer consults `enable_init_join_order`, and removing
the only `InitJoinOrder` registration also makes `INIT_JOIN_ORDER` rule
disabling ineffective. Existing sessions that explicitly disable initial
reordering will now receive this default-on greedy rewrite for any join plan,
even without an aggregate. Gate the replacement through the legacy control (or
provide an explicit compatibility/deprecation migration) and cover both old
disable paths with a plan test.
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/joinorder/JoinReorderRule.java:
##
@@ -0,0 +1,178 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
feiniaofeiafei commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5714443544 run feut -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
feiniaofeiafei commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5714441863 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
hello-stephen commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5709028309 TPC-DS: Total hot run time: 152409 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 5465d61ed0599094ec775ab68dec8da8a6ff1e1c, data reload: false query5 4300581 467 467 query6 432 208 193 193 query7 4826555 295 295 query8 325 173 165 165 query9 8822397540033975 query10 464 313 275 275 query11 5826354432233223 query12 150 90 90 90 query13 1275598 411 411 query14 6532448342304230 query14_13959392239643922 query15 204 191 186 186 query16 962 462 422 422 query17 929 675 549 549 query18 2460466 347 347 query19 219 181 143 143 query20 85 82 81 81 query21 235 132 112 112 query22 12940 12950 12854 12854 query23 13926 13010 12413 12413 query23_112523 12424 12523 12424 query24 72921159699 699 query24_1640 662 717 662 query25 576 445 378 378 query26 1268307 167 167 query27 2702557 342 342 query28 4543196019661960 query29 1662752 534 534 query30 304 214 180 180 query31 905 760 639 639 query32 144 117 113 113 query33 530 325 255 255 query34 11821100648 648 query35 741 742 651 651 query36 820 816 724 724 query37 158 119 101 101 query38 1819175717101710 query39 690 703 673 673 query39_1637 653 668 653 query40 225 124 102 102 query41 72 67 68 67 query42 99 94 95 94 query43 333 343 299 299 query44 1359708 713 708 query45 187 180 166 166 query46 10581152750 750 query47 1492148914541454 query48 410 363 297 297 query49 585 400 296 296 query50 927 354 249 249 query51 10248 10489 10344 10344 query52 87 86 75 75 query53 234 249 175 175 query54 247 208 192 192 query55 82 73 72 72 query56 222 210 200 200 query57 1507134913071307 query58 283 241 253 241 query59 1987205818101810 query60 301 245 223 223 query61 150 151 148 148 query62 413 317 264 264 query63 215 171 176 171 query64 2801982 806 806 query65 3448339334023393 query66 1814417 303 303 query67 20216 19924 19904 19904 query68 33541442950 950 query69 460 302 259 259 query70 941 840 812 812 query71 296 233 213 213 query72 2836260322252225 query73 803 799 426 426 query74 4675451742864286 query75 2293227319221922 query76 24011100713 713 query77 360 392 300 300 query78 9080899484858485 query79 13071187748 748 query80 591 449 372 372 query81 532 312 281 281 query82 629 163 126 126 query83 314 222 200 200 query84 316 139 116 116 query85 844 479 390 390 query86 338 245 225 225 query87 1985194618531853 query88 3601271527322715 query89 353 286 244 244 query90 1935181 177 177 query91 178 168 127 127 query92 105 89 88 88 query93 15321384881 881 query94 548 342 276 276 query95 654 471 333 333 query96 1086803 357 357 query97 2442241023092309 query98 155 148 146 146 query99 728 737 617 617 Total cold run time: 236353 ms Total hot run time: 152409 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
hello-stephen commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5709091821 ClickBench: Total hot run time: 23.85 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 5465d61ed0599094ec775ab68dec8da8a6ff1e1c, data reload: false query1 0.000.000.00 query2 0.090.050.05 query3 0.260.140.14 query4 1.610.130.13 query5 0.250.220.21 query6 1.160.930.91 query7 0.040.010.00 query8 0.060.040.04 query9 0.390.330.34 query10 0.570.540.54 query11 0.200.140.14 query12 0.180.140.14 query13 0.460.460.45 query14 0.950.980.94 query15 0.600.590.58 query16 0.320.310.31 query17 1.061.121.08 query18 0.210.200.20 query19 2.041.911.98 query20 0.020.010.01 query21 15.43 0.250.15 query22 4.770.060.05 query23 16.12 0.310.13 query24 2.980.430.34 query25 0.100.040.04 query26 0.720.200.14 query27 0.040.040.04 query28 3.620.740.36 query29 12.50 4.013.20 query30 0.280.140.16 query31 2.780.570.31 query32 3.230.580.49 query33 3.253.133.23 query34 15.50 3.903.26 query35 3.243.213.22 query36 0.560.410.40 query37 0.090.060.06 query38 0.050.030.03 query39 0.030.040.04 query40 0.180.160.14 query41 0.080.030.03 query42 0.040.020.02 query43 0.050.040.03 Total cold run time: 96.11 s Total hot run time: 23.85 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
hello-stephen commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5708896650 TPC-H: Total hot run time: 27637 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 5465d61ed0599094ec775ab68dec8da8a6ff1e1c, data reload: false -- Round 1 -- q1 17656 396538733873 q2 2189360 318 318 q3 10057 1396796 796 q4 4685484 348 348 q5 7568815 535 535 q6 180 167 137 137 q7 713 774 597 597 q8 9287149815291498 q9 5340417641474147 q10 6837133210231023 q11 427 264 244 244 q12 634 413 291 291 q13 18074 265420262026 q14 259 254 240 240 q15 q16 719 714 656 656 q17 17261114962 962 q18 6541562555135513 q19 1265121410741074 q20 465 390 260 260 q21 5677292928072807 q22 425 357 292 292 Total cold run time: 100724 ms Total hot run time: 27637 ms - Round 2, with runtime_filter_mode=off - q1 4172416740784078 q2 703 554 499 499 q3 4461481143674367 q4 2146232314111411 q5 4192407840514051 q6 224 173 128 128 q7 1701159314051405 q8 2174225120662066 q9 7307738173807380 q10 3703362831693169 q11 556 405 372 372 q12 718 740 507 507 q13 2481281921292129 q14 306 295 284 284 q15 q16 687 723 653 653 q17 7919720770527052 q18 11895 11079 11773 11079 q19 1202111011001100 q20 2234221919361936 q21 5291445843794379 q22 515 449 390 390 Total cold run time: 64587 ms Total hot run time: 58435 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
feiniaofeiafei commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5708219621 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
github-actions[bot] commented on code in PR #67067:
URL: https://github.com/apache/doris/pull/67067#discussion_r4025714427
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/joinorder/JoinReorderRule.java:
##
@@ -0,0 +1,177 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.joinorder;
+
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.visitor.DefaultPlanRewriter;
+
+import com.google.common.collect.Lists;
+
+import java.util.List;
+import java.util.stream.Stream;
+
+/**JoinReorderRule*/
+public class JoinReorderRule extends DefaultPlanRewriter {
+public static final JoinReorderRule INSTANCE = new JoinReorderRule();
+public static final int MAX_ATOM_NUM_FOR_GREEDY = 16;
+
+public Plan rewrite(Plan plan, Void context) {
+return plan.accept(this, context);
+}
+
+@Override
+public Plan visitLogicalJoin(
+LogicalJoin join,
+Void context) {
+if (!isReorderable(join)) {
+// The current join is a boundary, but its children may contain
independent join clusters.
+return DefaultPlanRewriter.visitChildren(this, join, context);
+}
+
+// The current join is the root of a cluster. Reorder the current
cluster and recursively
+// process independent clusters below its boundaries.
+return reorderCluster(join, context);
+}
+
+private Plan reorderCluster(
+LogicalJoin root,
+Void context) {
+JoinCluster cluster = new JoinCluster(root.getOutput());
+Plan fallback = rewriteAndCollectCluster(root, cluster, context);
+
+// Use the fallback when the best candidate increases the number of
cross joins.
+Plan reordered = reorder(cluster);
+return reordered == null ? fallback : reordered;
+}
+
+private int countCrossJoinsInCluster(Plan plan) {
+if (plan instanceof LogicalJoin
+&& isReorderable((LogicalJoin) plan)) {
+LogicalJoin join = (LogicalJoin) plan;
+int currentCrossJoinCount = join.getJoinType().isCrossJoin() ? 1 :
0;
+return currentCrossJoinCount
++ countCrossJoinsInCluster(join.left())
++ countCrossJoinsInCluster(join.right());
+}
+if (plan instanceof LogicalProject
+&& isTransparentProject((LogicalProject) plan)) {
+return countCrossJoinsInCluster(plan.child(0));
+}
+return 0;
+}
+
+/*
+ * Traverses once to collect the current reorderable join cluster and
rewrite independent
+ * clusters below its boundaries.
+ * Collects inputs, predicates, and the cross-join count into the cluster
parameter, and
+ * returns the fallback plan.
+ */
+private Plan rewriteAndCollectCluster(Plan plan, JoinCluster cluster, Void
context) {
+if (plan instanceof LogicalJoin
+&& isReorderable((LogicalJoin) plan)) {
+LogicalJoin join = (LogicalJoin) plan;
+cluster.addPredicates(join.getHashJoinConjuncts());
+cluster.addPredicates(join.getOtherJoinConjuncts());
+if (join.getJoinType().isCrossJoin()) {
+cluster.crossJoinCount++;
+}
+Plan left = rewriteAndCollectCluster(join.left(), cluster,
context);
+Plan right = rewriteAndCollectCluster(join.right(), cluster,
context);
+return left == join.left() && right == join.right()
+? join
+: join.withChildren(left, right);
+}
+if (plan instanceof LogicalProject
+&& isTransparentProject((LogicalProject) plan)) {
+LogicalProject project = (LogicalProject) plan;
+
+/*
+ * The project contains only existing slots and does not replace
any
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
hello-stephen commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5698219352 TPC-DS: Total hot run time: 83396 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 3bcb4095e1bfe9c99e290063c67c7a51fe106113, data reload: false query5 4237416 355 355 query6 393 146 128 128 query7 4911405 234 234 query8 301 127 124 124 query9 8734293029152915 query10 405 221 186 186 query11 5465140312771277 query12 120 69 69 69 query13 1200468 342 342 query14 6188227421522152 query14_12037201620052005 query15 176 123 117 117 query16 916 376 367 367 query17 814 451 351 351 query18 2342330 242 242 query19 171 139 119 119 query20 66 69 63 63 query21 205 104 90 90 query22 5528559254845484 query23 6462605557405740 query23_15805576357655763 query24 7248940 516 516 query24_1520 517 566 517 query25 448 307 270 270 query26 1244223 133 133 query27 2823417 272 272 query28 4669151414911491 query29 1526555 411 411 query30 277 158 126 126 query31 754 406 341 341 query32 132 77 76 76 query33 489 203 180 180 query34 992 831 486 486 query35 417 398 339 339 query36 593 568 527 527 query37 114 89 73 73 query38 1017849 820 820 query39 486 488 483 483 query39_1462 458 466 458 query40 199 88 77 77 query41 53 51 51 51 query42 77 75 71 71 query43 241 242 214 214 query44 1004548 560 548 query45 110 105 103 103 query46 756 864 542 542 query47 780 773 708 708 query48 329 315 221 221 query49 544 242 191 191 query50 713 275 207 207 query51 8139808380498049 query52 68 74 62 62 query53 200 244 159 159 query54 210 170 151 151 query55 64 60 54 54 query56 193 165 167 165 query57 717 697 652 652 query58 228 195 201 195 query59 1262125411311131 query60 231 189 183 183 query61 113 120 111 111 query62 335 197 207 197 query63 178 145 144 144 query64 2837719 582 582 query65 1515144714731447 query66 1813259 205 205 query67 991910048 98889888 query68 30771140782 782 query69 342 223 193 193 query70 664 562 599 562 query71 245 167 171 167 query72 2091165915191519 query73 630 635 345 345 query74 1988124511991199 query75 12031121976 976 query76 2293714 513 513 query77 258 258 214 214 query78 3986386933323332 query79 1527815 568 568 query80 1197340 289 289 query81 524 195 165 165 query82 696 125 101 101 query83 272 168 153 153 query84 300 116 96 96 query85 855 395 332 332 query86 387 182 177 177 query87 1023999 938 938 query88 2739208521082085 query89 296 195 175 175 query90 1936129 136 129 query91 133 120 102 102 query92 81 73 71 71 query93 14411033677 677 query94 656 264 233 233 query95 528 261 298 261 query96 831 574 279 279 query97 1102111010221022 query98 133 116 127 116 query99 416 352 303 303 Total cold run time: 176717 ms Total hot run time: 83396 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
hello-stephen commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5698128656 TPC-H: Total hot run time: 17139 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 3bcb4095e1bfe9c99e290063c67c7a51fe106113, data reload: false -- Round 1 -- q1 17560 303730263026 q2 2068281 251 251 q3 10228 877 540 540 q4 4673266 207 207 q5 7651581 385 385 q6 137 124 97 97 q7 548 521 406 406 q8 9224839 951 839 q9 3523247924372437 q10 6488671 530 530 q11 393 202 183 183 q12 612 270 211 211 q13 18116 156811721172 q14 167 150 140 140 q15 q16 459 405 369 369 q17 1376929 754 754 q18 3239231322892289 q19 1280927 765 765 q20 375 293 196 196 q21 5532220621132113 q22 334 264 229 229 Total cold run time: 93983 ms Total hot run time: 17139 ms - Round 2, with runtime_filter_mode=off - q1 3361329933193299 q2 533 394 384 384 q3 2333235921302130 q4 12581244909 909 q5 2305222121822182 q6 183 120 93 93 q7 1091949 929 929 q8 1627140914291409 q9 3270325732363236 q10 1737163114441444 q11 362 278 255 255 q12 464 442 350 350 q13 1514155411741174 q14 172 175 161 161 q15 q16 395 411 368 368 q17 3649341233253325 q18 5014455951334559 q19 944 880 863 863 q20 10511013850 850 q21 3699316631293129 q22 399 342 318 318 Total cold run time: 35361 ms Total hot run time: 31367 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
hello-stephen commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5698283464 ClickBench: Total hot run time: 14.82 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 3bcb4095e1bfe9c99e290063c67c7a51fe106113, data reload: false query1 0.010.000.00 query2 0.080.040.03 query3 0.240.110.10 query4 1.600.090.09 query5 0.180.160.16 query6 1.280.690.68 query7 0.040.000.01 query8 0.050.030.04 query9 0.290.220.22 query10 0.360.350.35 query11 0.170.110.11 query12 0.150.130.12 query13 0.320.310.30 query14 0.460.460.47 query15 0.360.350.36 query16 0.240.240.24 query17 0.740.680.74 query18 0.170.160.17 query19 1.161.161.13 query20 0.020.010.01 query21 15.52 0.150.13 query22 5.070.040.04 query23 16.08 0.260.10 query24 3.000.310.26 query25 0.110.050.04 query26 0.760.170.13 query27 0.030.030.02 query28 3.670.560.28 query29 12.40 3.202.56 query30 0.250.120.12 query31 2.760.380.16 query32 3.540.330.23 query33 1.471.441.54 query34 15.39 2.291.80 query35 1.791.791.77 query36 0.460.310.29 query37 0.070.040.04 query38 0.040.030.03 query39 0.040.020.02 query40 0.120.080.07 query41 0.080.030.02 query42 0.030.020.02 query43 0.030.030.03 Total cold run time: 90.63 s Total hot run time: 14.82 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
hello-stephen commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5697555254 # FE Regression Coverage Report Increment line coverage ` 65.62% (313/477)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/67067_3bcb4095e1bfe9c99e290063c67c7a51fe106113_merge_fe/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/67067_3bcb4095e1bfe9c99e290063c67c7a51fe106113_merge_fe/report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
feiniaofeiafei commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5696429360 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
feiniaofeiafei commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5694102455 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
hello-stephen commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5681534619 # FE Regression Coverage Report Increment line coverage ` 81.89% (312/381)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/67067_572642825675f9ae70bb2eb768aed9068362f851_merge_fe/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/67067_572642825675f9ae70bb2eb768aed9068362f851_merge_fe/report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
hello-stephen commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5679963166 # FE UT Coverage Report Increment line coverage ` 83.73% (319/381)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/67067_572642825675f9ae70bb2eb768aed9068362f851/fe_increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/67067_572642825675f9ae70bb2eb768aed9068362f851/fe_report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
feiniaofeiafei commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5678141126 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
feiniaofeiafei commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5677644435 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
feiniaofeiafei commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5674750145 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
hello-stephen commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5511354226 # FE Regression Coverage Report Increment line coverage ` 80.67% (313/388)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/67067_a77d52120da471f8d24ab2ca83ed3aaaf936c236_merge_fe/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/67067_a77d52120da471f8d24ab2ca83ed3aaaf936c236_merge_fe/report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
hello-stephen commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5507780248 # FE UT Coverage Report Increment line coverage ` 84.84% (319/376)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/67067_a77d52120da471f8d24ab2ca83ed3aaaf936c236/fe_increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/67067_a77d52120da471f8d24ab2ca83ed3aaaf936c236/fe_report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
hello-stephen commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5508690181 ClickBench: Total hot run time: 14.71 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit a77d52120da471f8d24ab2ca83ed3aaaf936c236, data reload: false query1 0.000.010.00 query2 0.080.040.03 query3 0.240.110.09 query4 1.600.100.09 query5 0.180.160.16 query6 1.230.720.69 query7 0.040.000.00 query8 0.040.040.03 query9 0.290.210.22 query10 0.340.350.33 query11 0.160.120.12 query12 0.150.120.12 query13 0.310.300.30 query14 0.450.460.45 query15 0.360.340.36 query16 0.210.230.22 query17 0.730.720.72 query18 0.160.160.17 query19 1.211.211.19 query20 0.020.010.01 query21 15.46 0.150.11 query22 5.070.040.04 query23 16.17 0.260.10 query24 3.050.340.25 query25 0.110.050.03 query26 0.720.160.12 query27 0.030.030.04 query28 3.670.560.27 query29 12.43 3.222.57 query30 0.280.100.12 query31 2.760.360.17 query32 3.520.330.22 query33 1.451.431.51 query34 15.35 2.101.78 query35 1.701.731.71 query36 0.470.290.28 query37 0.070.040.04 query38 0.040.040.04 query39 0.040.020.02 query40 0.110.080.07 query41 0.070.020.02 query42 0.040.020.02 query43 0.030.030.03 Total cold run time: 90.44 s Total hot run time: 14.71 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
hello-stephen commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5508642266 TPC-DS: Total hot run time: 82426 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit a77d52120da471f8d24ab2ca83ed3aaaf936c236, data reload: false query5 4263407 339 339 query6 398 135 139 135 query7 4930400 221 221 query8 294 131 142 131 query9 8718289228702870 query10 392 227 195 195 query11 5483139712591259 query12 127 72 73 72 query13 1208440 341 341 query14 6106219420882088 query14_11977194519651945 query15 179 122 117 117 query16 926 364 368 364 query17 822 470 408 408 query18 2352347 256 256 query19 179 155 114 114 query20 86 65 63 63 query21 198 100 88 88 query22 5349544352805280 query23 6261564253955395 query23_15636565254745474 query24 72831106773 773 query24_1787 759 774 759 query25 456 317 266 266 query26 1239238 135 135 query27 2784424 261 261 query28 4634152615131513 query29 950 458 370 370 query30 252 155 133 133 query31 752 401 334 334 query32 123 81 73 73 query33 470 232 187 187 query34 981 877 481 481 query35 403 392 344 344 query36 555 581 568 568 query37 116 89 69 69 query38 1005839 811 811 query39 525 488 469 469 query39_1490 455 481 455 query40 202 90 85 85 query41 52 52 52 52 query42 75 71 71 71 query43 240 239 211 211 query44 992 554 559 554 query45 108 105 95 95 query46 819 803 529 529 query47 755 758 706 706 query48 309 307 218 218 query49 550 239 193 193 query50 708 258 204 204 query51 8295800281708002 query52 72 69 67 67 query53 195 190 151 151 query54 232 197 182 182 query55 61 60 54 54 query56 215 249 164 164 query57 682 660 643 643 query58 197 202 161 161 query59 1210123311011101 query60 242 211 184 184 query61 122 124 115 115 query62 349 205 172 172 query63 178 142 144 142 query64 2616691 671 671 query65 1677160515891589 query66 1830257 203 203 query67 9669977297289728 query68 29251204696 696 query69 334 207 198 198 query70 663 558 587 558 query71 237 183 170 170 query72 2302175115701570 query73 642 605 346 346 query74 1581122411331133 query75 11741125950 950 query76 2288728 550 550 query77 266 257 227 227 query78 3843359831913191 query79 2921826 592 592 query80 1612340 290 290 query81 477 154 129 129 query82 615 131 95 95 query83 236 169 147 147 query84 291 117 94 94 query85 848 384 309 309 query86 392 178 176 176 query87 1014984 885 885 query88 2859215021292129 query89 285 206 176 176 query90 1960128 127 127 query91 145 148 102 102 query92 75 71 67 67 query93 20161081686 686 query94 662 264 217 217 query95 528 314 228 228 query96 814 581 253 253 query97 10241066983 983 query98 143 119 115 115 query99 408 356 316 316 Total cold run time: 177367 ms Total hot run time: 82426 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
hello-stephen commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5508571885 TPC-H: Total hot run time: 16856 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit a77d52120da471f8d24ab2ca83ed3aaaf936c236, data reload: false -- Round 1 -- q1 17586 303830483038 q2 2117265 225 225 q3 10205 893 535 535 q4 4673245 205 205 q5 7679574 387 387 q6 139 116 94 94 q7 533 488 379 379 q8 9241942 1020942 q9 3483237724022377 q10 6467666 530 530 q11 391 201 177 177 q12 613 259 205 205 q13 18125 151811461146 q14 164 154 139 139 q15 q16 437 397 366 366 q17 1394888 819 819 q18 3072225622592256 q19 1259916 772 772 q20 366 279 203 203 q21 5579182618741826 q22 336 269 235 235 Total cold run time: 93859 ms Total hot run time: 16856 ms - Round 2, with runtime_filter_mode=off - q1 3419333833503338 q2 502 416 387 387 q3 2236227421632163 q4 11771161887 887 q5 2168209721002097 q6 168 119 88 88 q7 1038907 862 862 q8 1603139814101398 q9 3138308131093081 q10 1631153314131413 q11 359 275 256 256 q12 449 424 343 343 q13 1466150811581158 q14 183 179 164 164 q15 q16 392 388 361 361 q17 3556328832343234 q18 4784439147344391 q19 889 793 846 793 q20 1069985 828 828 q21 3842312033483120 q22 390 348 316 316 Total cold run time: 34459 ms Total hot run time: 30678 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
feiniaofeiafei commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5506566704 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
hello-stephen commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5439733890 ClickBench: Total hot run time: 14.58 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit ec54b397fb34801f8b26ac4053c2c6a3cde01305, data reload: false query1 0.010.010.00 query2 0.080.040.04 query3 0.240.110.11 query4 1.590.100.09 query5 0.160.160.16 query6 1.260.670.68 query7 0.030.010.01 query8 0.040.030.03 query9 0.290.210.21 query10 0.340.350.37 query11 0.170.120.11 query12 0.150.110.12 query13 0.300.310.30 query14 0.450.450.45 query15 0.360.350.35 query16 0.240.240.23 query17 0.670.670.66 query18 0.170.150.14 query19 1.181.161.18 query20 0.010.010.02 query21 15.44 0.160.13 query22 5.060.040.04 query23 16.17 0.270.10 query24 2.990.320.27 query25 0.100.040.03 query26 0.780.160.12 query27 0.030.030.03 query28 3.660.570.27 query29 12.46 3.182.54 query30 0.250.110.12 query31 2.750.380.17 query32 3.510.310.23 query33 1.371.401.44 query34 15.39 2.161.73 query35 1.751.721.75 query36 0.440.290.27 query37 0.060.040.04 query38 0.040.030.03 query39 0.030.020.02 query40 0.120.080.08 query41 0.070.020.02 query42 0.030.020.02 query43 0.030.030.02 Total cold run time: 90.27 s Total hot run time: 14.58 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
hello-stephen commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5439682564 TPC-DS: Total hot run time: 83010 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit ec54b397fb34801f8b26ac4053c2c6a3cde01305, data reload: false query5 4278423 347 347 query6 394 154 141 141 query7 4881431 240 240 query8 292 127 118 118 query9 8729288928882888 query10 391 220 191 191 query11 5454139312621262 query12 131 78 75 75 query13 1204441 331 331 query14 6119218321052105 query14_11977196019331933 query15 173 117 109 109 query16 938 378 370 370 query17 815 474 389 389 query18 2343348 278 278 query19 174 144 115 115 query20 87 63 65 63 query21 203 106 89 89 query22 5400524852895248 query23 6259579554925492 query23_15578556156125561 query24 72821095762 762 query24_1774 771 794 771 query25 440 322 268 268 query26 1229241 134 134 query27 2783417 250 250 query28 4625152015101510 query29 953 439 365 365 query30 257 157 132 132 query31 751 435 336 336 query32 120 73 75 73 query33 451 212 175 175 query34 980 843 489 489 query35 385 397 332 332 query36 563 547 534 534 query37 119 77 68 68 query38 991 822 825 822 query39 480 470 474 470 query39_1444 463 461 461 query40 199 89 77 77 query41 53 49 51 49 query42 76 72 71 71 query43 236 239 205 205 query44 1017545 554 545 query45 109 105 104 104 query46 776 862 516 516 query47 753 747 711 711 query48 309 309 233 233 query49 533 241 190 190 query50 789 266 194 194 query51 8075819382098193 query52 69 75 57 57 query53 201 199 148 148 query54 244 192 185 185 query55 62 58 53 53 query56 188 182 184 182 query57 709 677 715 677 query58 191 174 164 164 query59 1228124011051105 query60 224 184 203 184 query61 116 124 128 124 query62 371 220 174 174 query63 173 142 142 142 query64 2673720 631 631 query65 1640162816291628 query66 1850254 226 226 query67 9744958394079407 query68 31851210778 778 query69 332 217 190 190 query70 648 581 587 581 query71 249 175 167 167 query72 2546177117841771 query73 621 561 324 324 query74 1976122111361136 query75 11891112953 953 query76 2351734 554 554 query77 279 259 220 220 query78 3864363432633263 query79 2346831 582 582 query80 1586335 284 284 query81 483 154 132 132 query82 620 119 95 95 query83 234 168 152 152 query84 291 110 92 92 query85 805 377 318 318 query86 390 172 172 172 query87 1006974 892 892 query88 2773211621122112 query89 273 197 176 176 query90 1993129 131 129 query91 132 128 104 104 query92 81 68 63 63 query93 14521074737 737 query94 629 270 226 226 query95 532 240 285 240 query96 809 614 281 281 query97 10391062984 984 query98 164 117 134 117 query99 412 340 309 309 Total cold run time: 176705 ms Total hot run time: 83010 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
hello-stephen commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5439607291 TPC-H: Total hot run time: 16782 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit ec54b397fb34801f8b26ac4053c2c6a3cde01305, data reload: false -- Round 1 -- q1 17580 315931093109 q2 2123269 233 233 q3 10205 951 535 535 q4 4683249 206 206 q5 7670567 385 385 q6 140 112 95 95 q7 529 496 383 383 q8 9235869 854 854 q9 3498241624102410 q10 6469656 497 497 q11 385 200 176 176 q12 620 271 198 198 q13 18112 153711751175 q14 158 149 141 141 q15 q16 440 398 370 370 q17 1371903 754 754 q18 30412259 q19 1110927 797 797 q20 364 283 195 195 q21 4877181918551819 q22 328 266 228 228 Total cold run time: 92938 ms Total hot run time: 16782 ms - Round 2, with runtime_filter_mode=off - q1 3485343034213421 q2 493 400 365 365 q3 227221922192 q4 12011162903 903 q5 2177211920952095 q6 169 118 87 87 q7 1014915 850 850 q8 1638143514331433 q9 3173312031123112 q10 1628154813941394 q11 360 271 248 248 q12 454 425 338 338 q13 1492154011621162 q14 175 165 161 161 q15 q16 394 404 351 351 q17 3602324132233223 q18 4800438947344389 q19 889 832 845 832 q20 1157989 820 820 q21 3846317333623173 q22 392 332 318 318 Total cold run time: 34761 ms Total hot run time: 30867 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
feiniaofeiafei commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5438017977 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
morrySnow commented on code in PR #67067: URL: https://github.com/apache/doris/pull/67067#discussion_r3861377718 ## fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java: ## @@ -3062,6 +3062,9 @@ public void checkFileCacheQueryLimitPercent(String fileCacheQueryLimitPercentStr + "Intended for one-shot / ad-hoc cold queries.") public boolean invertedIndexSniiReadNoWriteFileCache = false; [email protected](name = "disable_join_reorder_before_eager_agg", needForward = true) +public boolean disableJoinReorderBeforeEagerAgg = false; Review Comment: use enable instead of disable ## fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/joinorder/JoinOrder.java: ## @@ -0,0 +1,278 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.nereids.rules.rewrite.joinorder; + +import org.apache.doris.nereids.rules.rewrite.StatsDerive; +import org.apache.doris.nereids.rules.rewrite.StatsDerive.DeriveContext; +import org.apache.doris.nereids.trees.expressions.ExprId; +import org.apache.doris.nereids.trees.expressions.Expression; +import org.apache.doris.nereids.trees.plans.Plan; +import org.apache.doris.nereids.trees.plans.logical.LogicalJoin; + +import com.google.common.collect.Lists; +import com.google.common.collect.Maps; + +import java.util.BitSet; +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.Set; +import java.util.stream.Collectors; + +/**JoinOrder*/ +public abstract class JoinOrder { +private static final double MAXIMUM_COST = Double.MAX_VALUE / Math.pow(10, 50); +private static final double EXECUTE_COST_PENALTY = 2; +private static final double CROSS_JOIN_PENALTY = 1_000_000; + +protected int atomSize; +protected int edgeSize; +protected final List joinLevels = Lists.newArrayList(); +protected final List edges = Lists.newArrayList(); +protected final Map bitSetToGroupInfo = Maps.newHashMap(); + +/**ExpressionInfo*/ +static class ExpressionInfo { +final Plan expr; Review Comment: 这个命名很奇怪啊，得改成plan ## fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/eageraggregation/PushDownAggregation.java: ## @@ -85,6 +91,9 @@ public class PushDownAggregation extends DefaultPlanRewriter impleme @Override public Plan rewriteRoot(Plan plan, JobContext jobContext) { +if (!plan.anyMatch(node -> node instanceof Aggregate)) { Review Comment: nonMatch ## fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/joinorder/JoinOrder.java: ## @@ -0,0 +1,278 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.nereids.rules.rewrite.joinorder; + +import org.apache.doris.nereids.rules.rewrite.StatsDerive; +import org.apache.doris.nereids.rules.rewrite.StatsDerive.DeriveContext; +import org.apache.doris.nereids.trees.expressions.ExprId; +import org.apache.doris.nereids.trees.expressions.Expression; +import org.apache.doris.nereids.trees.plans.Plan; +import org.apache.doris.nereids.trees.plans.logical.LogicalJoin; + +import com.google.common.collect.Lists; +import com.google.common.collect.Maps; + +import java.util.BitSet; +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.Set; +import
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
hello-stephen commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5421047425 # FE UT Coverage Report Increment line coverage ` 71.52% (221/309)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/67067_51c98fb729a382d80e185e41bf9a91f932b1bcc3/fe_increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/67067_51c98fb729a382d80e185e41bf9a91f932b1bcc3/fe_report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
hello-stephen commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5420382410 ClickBench: Total hot run time: 14.59 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 51c98fb729a382d80e185e41bf9a91f932b1bcc3, data reload: false query1 0.010.010.00 query2 0.090.040.04 query3 0.280.110.11 query4 1.610.110.11 query5 0.170.160.15 query6 1.170.710.67 query7 0.040.010.00 query8 0.060.030.03 query9 0.300.210.23 query10 0.360.350.35 query11 0.180.110.11 query12 0.150.130.12 query13 0.290.300.30 query14 0.450.480.43 query15 0.370.340.34 query16 0.200.220.22 query17 0.680.680.62 query18 0.190.180.17 query19 1.191.091.16 query20 0.010.010.01 query21 15.52 0.170.11 query22 4.940.040.04 query23 16.19 0.250.10 query24 3.710.310.31 query25 0.150.050.05 query26 0.840.160.12 query27 0.040.020.02 query28 3.600.600.32 query29 12.45 3.172.53 query30 0.270.120.13 query31 2.770.360.17 query32 3.500.320.22 query33 1.331.391.68 query34 15.37 2.161.82 query35 1.731.711.66 query36 0.510.290.29 query37 0.080.040.04 query38 0.050.030.03 query39 0.040.030.02 query40 0.140.080.08 query41 0.090.030.02 query42 0.040.020.03 query43 0.040.030.03 Total cold run time: 91.2 s Total hot run time: 14.59 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
hello-stephen commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5420353704 TPC-DS: Total hot run time: 82631 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 51c98fb729a382d80e185e41bf9a91f932b1bcc3, data reload: false query5 4275400 334 334 query6 430 137 125 125 query7 4924408 234 234 query8 407 121 120 120 query9 8754288328942883 query10 471 222 186 186 query11 5459137712621262 query12 138 76 70 70 query13 1225472 341 341 query14 6037218720612061 query14_11960196019771960 query15 179 129 115 115 query16 988 375 349 349 query17 885 465 379 379 query18 2366328 235 235 query19 183 146 116 116 query20 64 72 69 69 query21 230 107 89 89 query22 5309535052815281 query23 6034570357365703 query23_15491568255875587 query24 74021093761 761 query24_1787 783 780 780 query25 513 312 286 286 query26 1248249 131 131 query27 2778406 261 261 query28 4708149114881488 query29 1115457 377 377 query30 282 156 133 133 query31 786 392 324 324 query32 188 76 80 76 query33 486 227 198 198 query34 1148851 492 492 query35 404 407 353 353 query36 602 560 517 517 query37 139 85 71 71 query38 1087842 834 834 query39 496 490 477 477 query39_1471 462 458 458 query40 317 94 114 94 query41 52 50 49 49 query42 75 73 77 73 query43 242 238 212 212 query44 1089536 542 536 query45 125 99 102 99 query46 756 844 522 522 query47 755 734 729 729 query48 304 303 215 215 query49 559 246 189 189 query50 720 273 194 194 query51 8027800181108001 query52 106 86 61 61 query53 195 199 146 146 query54 311 166 152 152 query55 67 58 54 54 query56 248 199 150 150 query57 691 673 653 653 query58 235 174 164 164 query59 1202124711011101 query60 304 183 180 180 query61 113 134 119 119 query62 526 216 178 178 query63 179 144 143 143 query64 2876703 658 658 query65 1604157015861570 query66 1784266 205 205 query67 9494961895709570 query68 30471239710 710 query69 376 210 192 192 query70 657 591 594 591 query71 251 179 165 165 query72 2380170015751575 query73 730 620 353 353 query74 1981124611211121 query75 11921101954 954 query76 2311734 536 536 query77 262 254 213 213 query78 3930372832473247 query79 2437831 579 579 query80 1601340 279 279 query81 511 156 135 135 query82 1158127 95 95 query83 252 162 153 153 query84 307 113 89 89 query85 987 373 313 313 query86 550 177 160 160 query87 1032975 910 910 query88 3038210421062104 query89 335 195 173 173 query90 2035127 125 125 query91 146 121 106 106 query92 82 68 69 68 query93 16461181703 703 query94 856 253 208 208 query95 600 319 230 230 query96 832 566 282 282 query97 10421022985 985 query98 134 124 120 120 query99 537 336 302 302 Total cold run time: 180181 ms Total hot run time: 82631 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
hello-stephen commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5420311660 TPC-H: Total hot run time: 16566 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 51c98fb729a382d80e185e41bf9a91f932b1bcc3, data reload: false -- Round 1 -- q1 17595 304630093009 q2 2471249 219 219 q3 9991866 519 519 q4 4679249 203 203 q5 7669568 396 396 q6 137 115 93 93 q7 545 510 385 385 q8 9249907 953 907 q9 3446241224012401 q10 6472656 525 525 q11 415 194 186 186 q12 618 258 196 196 q13 18145 155311471147 q14 160 152 134 134 q15 q16 437 391 369 369 q17 1329895 804 804 q18 3116225622132213 q19 1141930 809 809 q20 378 280 197 197 q21 4927162718711627 q22 319 269 227 227 Total cold run time: 93239 ms Total hot run time: 16566 ms - Round 2, with runtime_filter_mode=off - q1 3371334533373337 q2 509 408 376 376 q3 2267228721082108 q4 11941161895 895 q5 2168208821222088 q6 165 117 86 86 q7 1032952 834 834 q8 1611141214151412 q9 3133307330963073 q10 1636156114101410 q11 361 272 252 252 q12 450 426 339 339 q13 1493151511531153 q14 174 173 156 156 q15 q16 385 389 352 352 q17 3584328832013201 q18 4837439047064390 q19 884 816 821 816 q20 10351044836 836 q21 3819317832933178 q22 395 338 316 316 Total cold run time: 34503 ms Total hot run time: 30608 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](eager-agg) Improve eager aggregation with greedy join reorder [doris]
feiniaofeiafei commented on PR #67067: URL: https://github.com/apache/doris/pull/67067#issuecomment-5420190149 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
