Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
morrySnow merged PR #49096: URL: https://github.com/apache/doris/pull/49096 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
feiniaofeiafei commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-3072809256 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
feiniaofeiafei commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-3072809637 run cloud_p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-3068515595 ClickBench: Total hot run time: 29.62 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 7a338b50b39e706366cfb07fd323255970d52971, data reload: false query1 0.040.030.04 query2 0.080.040.05 query3 0.240.080.08 query4 1.600.110.11 query5 0.420.450.41 query6 1.160.660.65 query7 0.020.020.02 query8 0.050.040.04 query9 0.610.500.51 query10 0.570.570.57 query11 0.160.110.11 query12 0.160.110.11 query13 0.620.610.61 query14 0.840.810.80 query15 0.900.880.88 query16 0.380.390.39 query17 1.081.091.07 query18 0.230.210.20 query19 1.951.871.85 query20 0.010.010.02 query21 15.51 0.880.53 query22 0.761.300.81 query23 14.74 1.420.60 query24 6.882.050.65 query25 0.490.100.07 query26 0.670.170.14 query27 0.060.060.06 query28 9.090.940.47 query29 12.55 3.963.30 query30 0.250.080.06 query31 2.830.620.39 query32 3.230.550.47 query33 3.093.143.10 query34 16.01 5.404.78 query35 4.784.874.88 query36 0.710.510.50 query37 0.090.070.06 query38 0.050.050.04 query39 0.030.030.03 query40 0.170.140.15 query41 0.080.030.02 query42 0.030.020.03 query43 0.040.030.03 Total cold run time: 103.26 s Total hot run time: 29.62 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-3068487224 TPC-DS: Total hot run time: 187176 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 7a338b50b39e706366cfb07fd323255970d52971, data reload: false query1 992 371 383 371 query2 6512170416711671 query3 6743206 212 206 query4 27203 23570 22949 22949 query5 4345566 454 454 query6 299 213 204 204 query7 4626485 286 286 query8 281 222 210 210 query9 8643260026342600 query10 461 342 290 290 query11 15335 15160 14739 14739 query12 154 106 102 102 query13 1656501 383 383 query14 8506600460376004 query15 223 203 163 163 query16 7193432 265 265 query17 1327716 587 587 query18 1991410 310 310 query19 197 187 170 170 query20 123 121 118 118 query21 209 123 107 107 query22 4300418940204020 query23 34245 33286 33356 33286 query24 8485238923612361 query25 577 506 437 437 query26 1231264 156 156 query27 2746516 352 352 query28 4333211620882088 query29 793 634 436 436 query30 282 219 191 191 query31 910 841 750 750 query32 70 65 61 61 query33 549 328 278 278 query34 794 861 505 505 query35 587 642 559 559 query36 961 981 926 926 query37 110 104 80 80 query38 4242420340944094 query39 1496142214461422 query40 201 113 109 109 query41 62 60 50 50 query42 121 111 111 111 query43 511 503 492 492 query44 1299813 806 806 query45 171 168 162 162 query46 829 1012622 622 query47 1795179918231799 query48 372 403 308 308 query49 730 468 398 398 query50 645 679 407 407 query51 5455558155045504 query52 104 111 102 102 query53 214 248 183 183 query54 567 574 485 485 query55 79 76 84 76 query56 286 289 267 267 query57 1195120411481148 query58 256 249 261 249 query59 2603264625912591 query60 328 317 300 300 query61 123 141 122 122 query62 806 731 651 651 query63 222 193 181 181 query64 44031144926 926 query65 4269417841934178 query66 1135416 310 310 query67 15844 15766 15500 15500 query68 9072871 516 516 query69 506 304 262 262 query70 1275114811091109 query71 467 321 283 283 query72 5364471847844718 query73 724 609 353 353 query74 9276886087068706 query75 4210325127032703 query76 36381196720 720 query77 875 368 288 288 query78 10979 11080 10258 10258 query79 2036817 578 578 query80 580 510 457 457 query81 479 270 221 221 query82 474 128 95 95 query83 293 250 231 231 query84 292 97 90 90 query85 789 378 351 351 query86 382 307 285 285 query87 4401438343234323 query88 3707227822942278 query89 394 313 300 300 query90 1827205 203 203 query91 135 139 110 110 query92 76 62 61 61 query93 1574931 573 573 query94 687 314 198 198 query95 383 285 291 285 query96 492 557 280 280 query97 2689277026272627 query98 236 205 204 204 query99 1457140612821282 Total cold run time: 278124 ms Total hot run time: 187176 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-3068427281 TPC-H: Total hot run time: 33371 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 7a338b50b39e706366cfb07fd323255970d52971, data reload: false -- Round 1 -- q1 17577 516950675067 q2 1968283 204 204 q3 10253 1288713 713 q4 10223 1022533 533 q5 7515238123802380 q6 179 155 129 129 q7 902 747 603 603 q8 9313127411311131 q9 7419517650955095 q10 6905240919581958 q11 484 293 273 273 q12 348 351 210 210 q13 17761 373730753075 q14 237 226 210 210 q15 541 484 480 480 q16 424 442 370 370 q17 599 871 349 349 q18 7778724171947194 q19 2139974 572 572 q20 341 355 210 210 q21 3737258623132313 q22 356 326 302 302 Total cold run time: 106999 ms Total hot run time: 33371 ms - Round 2, with runtime_filter_mode=off - q1 5165512250655065 q2 245 320 219 219 q3 2210266622992299 q4 1439180513801380 q5 4229442745804427 q6 208 165 124 124 q7 2092197818491849 q8 2714251125322511 q9 7338724073237240 q10 3176324128642864 q11 597 509 492 492 q12 700 758 637 637 q13 3776401133993399 q14 282 307 274 274 q15 519 482 477 477 q16 482 496 442 442 q17 1221160313831383 q18 7986793475337533 q19 824 846 940 846 q20 2018198018771877 q21 4806431742324232 q22 666 579 575 575 Total cold run time: 52693 ms Total hot run time: 50145 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
hello-stephen commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-3068302430 # FE UT Coverage Report Increment line coverage ` 71.33% (423/593)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/49096_7a338b50b39e706366cfb07fd323255970d52971/fe_increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/49096_7a338b50b39e706366cfb07fd323255970d52971/fe_report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
morrySnow commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-3068092777 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
github-actions[bot] commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-3068085260 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
morrySnow commented on code in PR #49096:
URL: https://github.com/apache/doris/pull/49096#discussion_r2204001945
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSkewExpr.java:
##
@@ -0,0 +1,64 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.analysis;
+
+import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.hint.DistributeHint;
+import org.apache.doris.nereids.hint.JoinSkewInfo;
+import org.apache.doris.nereids.pattern.MatchingContext;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.util.TypeCoercionUtils;
+
+import com.google.common.collect.ImmutableList;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**bind skew hint in DistributeHint*/
+public class BindSkewExpr extends BindExpression {
+@Override
+public List buildRules() {
+return ImmutableList.of(
+RuleType.BINDING_SKEW_EXPR.build(
+logicalJoin().when(join ->
join.getDistributeHint().getSkewInfo() != null)
+.thenApply(this::bindSkewExpr))
+);
+}
+
+private LogicalJoin
bindSkewExpr(MatchingContext> ctx) {
Review Comment:
add this to class's comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
hello-stephen commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-3061835520 # FE UT Coverage Report Increment line coverage ` 71.33% (423/593)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/49096_7a338b50b39e706366cfb07fd323255970d52971/fe_increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/49096_7a338b50b39e706366cfb07fd323255970d52971/fe_report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-3061824594 ClickBench: Total hot run time: 29.75 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 7a338b50b39e706366cfb07fd323255970d52971, data reload: false query1 0.040.040.04 query2 0.080.040.04 query3 0.240.070.07 query4 1.630.110.11 query5 0.430.420.43 query6 1.170.680.66 query7 0.030.020.02 query8 0.040.040.04 query9 0.610.530.54 query10 0.580.560.57 query11 0.160.110.11 query12 0.150.120.12 query13 0.620.600.61 query14 0.810.810.82 query15 0.910.890.88 query16 0.390.370.40 query17 1.051.111.07 query18 0.230.220.22 query19 1.951.831.83 query20 0.010.010.02 query21 15.39 0.900.52 query22 0.761.180.64 query23 14.95 1.380.61 query24 7.681.220.84 query25 0.480.180.13 query26 0.660.170.14 query27 0.060.060.05 query28 9.560.880.44 query29 12.54 3.943.34 query30 0.260.090.08 query31 2.820.600.40 query32 3.230.580.47 query33 3.063.153.21 query34 16.20 5.474.79 query35 4.804.864.84 query36 0.710.500.49 query37 0.090.070.07 query38 0.060.040.04 query39 0.030.030.02 query40 0.170.140.12 query41 0.090.020.03 query42 0.040.020.03 query43 0.040.020.02 Total cold run time: 104.81 s Total hot run time: 29.75 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-3061803255 TPC-DS: Total hot run time: 187255 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 7a338b50b39e706366cfb07fd323255970d52971, data reload: false query1 988 391 378 378 query2 6495166916721669 query3 6736213 212 212 query4 26841 23456 22923 22923 query5 4348586 440 440 query6 303 208 196 196 query7 4622504 286 286 query8 271 224 225 224 query9 8621262825952595 query10 457 330 263 263 query11 15656 15067 14802 14802 query12 152 111 106 106 query13 1652516 399 399 query14 9199599261175992 query15 213 193 163 163 query16 7408454 284 284 query17 1278730 597 597 query18 2002413 311 311 query19 198 187 172 172 query20 177 114 112 112 query21 212 122 107 107 query22 4247431941124112 query23 34088 33175 33128 33128 query24 8490233724002337 query25 529 479 415 415 query26 1239267 149 149 query27 2770503 344 344 query28 4251208520672067 query29 770 569 441 441 query30 282 223 189 189 query31 936 848 776 776 query32 95 68 60 60 query33 549 328 272 272 query34 786 845 507 507 query35 596 639 550 550 query36 957 1012865 865 query37 117 98 76 76 query38 4221433841744174 query39 1486145315891453 query40 206 116 104 104 query41 55 55 52 52 query42 126 111 110 110 query43 501 503 487 487 query44 1306819 807 807 query45 176 170 165 165 query46 844 1029632 632 query47 1781181017601760 query48 384 414 305 305 query49 738 511 395 395 query50 648 691 413 413 query51 5475554854665466 query52 115 106 103 103 query53 216 254 180 180 query54 562 573 492 492 query55 79 80 83 80 query56 303 292 268 268 query57 1210118911391139 query58 269 272 270 270 query59 2538276625732573 query60 319 309 303 303 query61 123 116 118 116 query62 854 697 630 630 query63 217 181 184 181 query64 44371168818 818 query65 4282417741904177 query66 1154432 315 315 query67 15780 15601 15541 15541 query68 7955904 534 534 query69 502 321 264 264 query70 1147114410581058 query71 464 305 305 305 query72 5317484448114811 query73 666 572 339 339 query74 9177909288288828 query75 3858320827002700 query76 37041203717 717 query77 785 371 291 291 query78 10877 11081 10429 10429 query79 2211918 588 588 query80 579 520 457 457 query81 483 257 228 228 query82 479 128 96 96 query83 248 248 230 230 query84 246 105 85 85 query85 839 363 315 315 query86 383 319 265 265 query87 4495447543654365 query88 3800222822342228 query89 382 323 289 289 query90 1838209 204 204 query91 141 147 111 111 query92 70 58 57 57 query93 1867951 587 587 query94 669 315 200 200 query95 377 289 284 284 query96 496 558 280 280 query97 2726274026202620 query98 221 201 206 201 query99 1390139712611261 Total cold run time: 277328 ms Total hot run time: 187255 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-3061768723 TPC-H: Total hot run time: 33173 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 7a338b50b39e706366cfb07fd323255970d52971, data reload: false -- Round 1 -- q1 17561 517350475047 q2 1937275 182 182 q3 10314 1269713 713 q4 10240 1004525 525 q5 7834238723012301 q6 176 157 128 128 q7 902 740 586 586 q8 9305123811031103 q9 6809506951465069 q10 6895236519751975 q11 485 315 284 284 q12 339 346 215 215 q13 17780 368930763076 q14 228 221 218 218 q15 555 523 477 477 q16 426 451 381 381 q17 624 849 385 385 q18 7434712570337033 q19 1344963 554 554 q20 348 384 234 234 q21 4120324823812381 q22 359 324 306 306 Total cold run time: 106015 ms Total hot run time: 33173 ms - Round 2, with runtime_filter_mode=off - q1 5121511450655065 q2 237 326 219 219 q3 2189266422452245 q4 1388175413501350 q5 4206435544984355 q6 215 170 137 137 q7 1993194618331833 q8 2682252825522528 q9 7226730372527252 q10 3106329528582858 q11 594 502 500 500 q12 779 770 945 770 q13 3547390833463346 q14 281 281 298 281 q15 506 491 469 469 q16 476 524 458 458 q17 1201152514401440 q18 7995761576857615 q19 832 796 851 796 q20 2042201718431843 q21 4777425442594254 q22 603 582 543 543 Total cold run time: 51996 ms Total hot run time: 50157 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
feiniaofeiafei commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-3061631358 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-3060556662 ClickBench: Total hot run time: 29.73 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 00ffa3a63a28bfa528aa8cb069964b165a7a88cf, data reload: false query1 0.040.040.03 query2 0.080.040.04 query3 0.240.080.08 query4 1.620.110.11 query5 0.440.420.41 query6 1.190.660.64 query7 0.030.020.02 query8 0.050.040.04 query9 0.590.510.52 query10 0.560.560.54 query11 0.160.110.11 query12 0.160.120.12 query13 0.620.610.61 query14 0.810.840.81 query15 0.890.880.87 query16 0.390.390.41 query17 1.061.071.03 query18 0.230.220.21 query19 1.891.821.86 query20 0.010.020.02 query21 15.42 0.930.58 query22 0.761.180.64 query23 14.96 1.370.65 query24 7.280.810.91 query25 0.490.200.25 query26 0.560.160.16 query27 0.050.060.05 query28 9.890.900.44 query29 12.53 4.013.34 query30 0.260.110.08 query31 2.800.580.38 query32 3.230.550.46 query33 3.073.203.13 query34 16.10 5.384.75 query35 4.824.834.85 query36 0.690.500.49 query37 0.090.070.07 query38 0.060.050.03 query39 0.030.020.02 query40 0.170.140.14 query41 0.080.020.02 query42 0.030.020.02 query43 0.040.030.03 Total cold run time: 104.47 s Total hot run time: 29.73 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-3060539624 TPC-DS: Total hot run time: 186103 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 00ffa3a63a28bfa528aa8cb069964b165a7a88cf, data reload: false query1 996 392 391 391 query2 6520164016311631 query3 6741215 213 213 query4 26091 24032 23347 23347 query5 4615577 430 430 query6 284 204 203 203 query7 4623481 279 279 query8 258 220 206 206 query9 8618263126462631 query10 456 331 275 275 query11 15581 15026 14746 14746 query12 170 108 104 104 query13 1653511 400 400 query14 9240580858175808 query15 212 194 169 169 query16 7228602 452 452 query17 1195734 602 602 query18 1986402 318 318 query19 204 185 165 165 query20 125 126 114 114 query21 213 129 109 109 query22 4341443541534153 query23 34231 32946 32981 32946 query24 8472237623802376 query25 534 486 384 384 query26 1227267 149 149 query27 2775508 339 339 query28 4351211120912091 query29 751 550 439 439 query30 279 209 183 183 query31 935 837 763 763 query32 69 60 59 59 query33 559 375 304 304 query34 807 830 524 524 query35 777 799 732 732 query36 944 976 873 873 query37 107 91 73 73 query38 4182423540794079 query39 1501145213951395 query40 211 118 102 102 query41 55 55 52 52 query42 131 114 112 112 query43 495 491 468 468 query44 1305817 817 817 query45 175 169 161 161 query46 823 1020612 612 query47 1779182717911791 query48 373 405 301 301 query49 737 455 386 386 query50 625 684 433 433 query51 5460536355465363 query52 105 103 100 100 query53 220 252 178 178 query54 568 550 498 498 query55 88 77 79 77 query56 291 287 292 287 query57 1165121911481148 query58 274 255 266 255 query59 2523266625492549 query60 323 319 295 295 query61 127 121 123 121 query62 776 711 664 664 query63 218 180 184 180 query64 4344980 686 686 query65 4277420141774177 query66 1139447 335 335 query67 15952 15675 15527 15527 query68 7974883 531 531 query69 479 309 270 270 query70 1200112610361036 query71 450 322 323 322 query72 5766474247254725 query73 700 586 348 348 query74 8884914486758675 query75 3781316926572657 query76 36861172710 710 query77 778 365 295 295 query78 10064 10113 94219421 query79 2462815 576 576 query80 616 500 467 467 query81 469 254 216 216 query82 440 124 95 95 query83 273 245 230 230 query84 289 116 90 90 query85 867 369 313 313 query86 372 286 264 264 query87 4381441342984298 query88 3259222922472229 query89 451 327 291 291 query90 1956207 198 198 query91 134 139 105 105 query92 74 57 54 54 query93 1738961 587 587 query94 668 407 288 288 query95 375 281 283 281 query96 494 559 278 278 query97 2722277426352635 query98 235 196 202 196 query99 1446138012921292 Total cold run time: 275893 ms Total hot run time: 186103 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-3060506801 TPC-H: Total hot run time: 33186 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 00ffa3a63a28bfa528aa8cb069964b165a7a88cf, data reload: false -- Round 1 -- q1 17557 554050605060 q2 1919277 184 184 q3 10442 1324719 719 q4 10315 1023509 509 q5 8534227223802272 q6 195 156 127 127 q7 889 762 604 604 q8 9292134811441144 q9 7216510950915091 q10 6958239119431943 q11 500 297 272 272 q12 362 348 217 217 q13 17776 365230163016 q14 219 222 227 222 q15 554 486 480 480 q16 422 437 365 365 q17 619 866 378 378 q18 7576723670677067 q19 1434967 604 604 q20 354 358 223 223 q21 4111321023872387 q22 361 328 302 302 Total cold run time: 107605 ms Total hot run time: 33186 ms - Round 2, with runtime_filter_mode=off - q1 5194506750685067 q2 235 319 224 224 q3 2141272223272327 q4 1378183113571357 q5 4237454645164516 q6 210 181 130 130 q7 2023196618151815 q8 2705259925462546 q9 7391735372467246 q10 3163323029662966 q11 583 506 492 492 q12 689 779 624 624 q13 3694415432693269 q14 267 300 266 266 q15 535 476 478 476 q16 454 484 451 451 q17 1184164014021402 q18 8025780076057605 q19 785 791 892 791 q20 1929198918141814 q21 4726428241794179 q22 656 591 515 515 Total cold run time: 52204 ms Total hot run time: 50078 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
feiniaofeiafei commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-3060439119 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
feiniaofeiafei commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-3060409607 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
hello-stephen commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-3057598766 # FE UT Coverage Report Increment line coverage ` 72.11% (424/588)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/49096_275d74003d46791afade50c8f852a679b4ff249c/fe_increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/49096_275d74003d46791afade50c8f852a679b4ff249c/fe_report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-3057506424 ClickBench: Total hot run time: 30.12 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 275d74003d46791afade50c8f852a679b4ff249c, data reload: false query1 0.040.040.04 query2 0.070.040.04 query3 0.250.080.08 query4 1.610.110.11 query5 0.430.430.40 query6 1.180.640.67 query7 0.030.020.02 query8 0.040.030.04 query9 0.600.510.51 query10 0.570.560.57 query11 0.160.120.11 query12 0.150.120.12 query13 0.620.620.62 query14 0.810.810.80 query15 0.900.870.89 query16 0.400.390.39 query17 1.041.071.04 query18 0.220.210.21 query19 1.961.841.89 query20 0.020.010.02 query21 15.40 0.890.54 query22 0.751.370.79 query23 14.73 1.360.64 query24 6.801.211.62 query25 0.500.150.21 query26 0.570.150.15 query27 0.060.060.06 query28 10.37 0.890.43 query29 12.57 4.073.35 query30 0.260.090.06 query31 2.840.590.39 query32 3.230.560.49 query33 3.123.063.12 query34 16.18 5.394.80 query35 4.864.774.83 query36 0.690.500.48 query37 0.090.070.06 query38 0.050.040.04 query39 0.030.020.03 query40 0.170.150.13 query41 0.070.030.02 query42 0.040.020.03 query43 0.040.030.02 Total cold run time: 104.52 s Total hot run time: 30.12 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-3057489128 TPC-DS: Total hot run time: 186501 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 275d74003d46791afade50c8f852a679b4ff249c, data reload: false query1 1016387 385 385 query2 6531182517531753 query3 6738213 217 213 query4 26936 23297 23430 23297 query5 4329568 422 422 query6 300 208 196 196 query7 4622529 291 291 query8 270 219 220 219 query9 8632268026862680 query10 476 327 264 264 query11 16077 15063 14855 14855 query12 148 111 104 104 query13 1647530 417 417 query14 9507573557985735 query15 212 195 159 159 query16 7417614 494 494 query17 1232714 571 571 query18 1997389 288 288 query19 190 199 161 161 query20 122 116 115 115 query21 218 128 107 107 query22 3929447645984476 query23 34843 33718 33071 33071 query24 8442238223822382 query25 543 447 389 389 query26 1218267 143 143 query27 2777506 371 371 query28 4332216121432143 query29 718 555 426 426 query30 272 214 181 181 query31 932 831 746 746 query32 74 63 67 63 query33 545 365 293 293 query34 785 874 505 505 query35 749 802 761 761 query36 938 960 881 881 query37 109 100 71 71 query38 4097425640254025 query39 1487142114931421 query40 200 115 102 102 query41 52 57 50 50 query42 126 101 104 101 query43 512 522 485 485 query44 1289852 828 828 query45 176 170 161 161 query46 821 1016636 636 query47 1736178016631663 query48 390 415 304 304 query49 727 491 385 385 query50 626 697 408 408 query51 5571549354735473 query52 107 103 95 95 query53 218 242 191 191 query54 569 582 494 494 query55 80 79 78 78 query56 305 294 273 273 query57 1175117511061106 query58 267 254 248 248 query59 2688278326232623 query60 345 312 316 312 query61 127 118 161 118 query62 799 727 664 664 query63 216 183 180 180 query64 42971019675 675 query65 4283416841484148 query66 1105403 303 303 query67 15917 15534 15349 15349 query68 4526892 530 530 query69 513 304 273 273 query70 1210112210641064 query71 405 330 304 304 query72 5530473049154730 query73 663 597 348 348 query74 9062905886808680 query75 3195317026752675 query76 32441152707 707 query77 500 393 307 307 query78 993510125 92029202 query79 2084835 633 633 query80 816 520 442 442 query81 499 263 218 218 query82 390 129 96 96 query83 255 243 235 235 query84 244 109 83 83 query85 756 352 324 324 query86 385 303 266 266 query87 4435439143494349 query88 2903227122602260 query89 375 316 283 283 query90 1796204 207 204 query91 152 152 125 125 query92 70 69 58 58 query93 1407931 579 579 query94 715 435 319 319 query95 399 308 300 300 query96 497 577 281 281 query97 2638273526402640 query98 234 213 209 209 query99 1349141012501250 Total cold run time: 271313 ms Total hot run time: 186501 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-3057451794 TPC-H: Total hot run time: 33454 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 275d74003d46791afade50c8f852a679b4ff249c, data reload: false -- Round 1 -- q1 17557 518350525052 q2 1971275 179 179 q3 10461 1321743 743 q4 10257 1051542 542 q5 7930244323422342 q6 181 162 133 133 q7 915 762 622 622 q8 9323133811311131 q9 7776505951405059 q10 6941241519821982 q11 482 291 275 275 q12 357 358 215 215 q13 17813 377131353135 q14 231 231 227 227 q15 543 494 476 476 q16 432 432 380 380 q17 610 885 351 351 q18 7695727271407140 q19 1637958 548 548 q20 335 334 218 218 q21 3861317324102410 q22 368 331 294 294 Total cold run time: 107676 ms Total hot run time: 33454 ms - Round 2, with runtime_filter_mode=off - q1 5225509250815081 q2 248 313 223 223 q3 2230268323092309 q4 1405178813561356 q5 4277454745664547 q6 213 168 128 128 q7 2040195517981798 q8 2678265226312631 q9 7337716971837169 q10 3119331128552855 q11 555 509 513 509 q12 755 790 642 642 q13 3768409035503550 q14 274 295 265 265 q15 528 473 468 468 q16 449 516 493 493 q17 1183160613901390 q18 7890773876287628 q19 845 869 998 869 q20 2133212420002000 q21 5247470545424542 q22 661 608 568 568 Total cold run time: 53060 ms Total hot run time: 51021 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
feiniaofeiafei commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-3057378133 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
feiniaofeiafei commented on code in PR #49096: URL: https://github.com/apache/doris/pull/49096#discussion_r2197497714 ## fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/SaltJoin.java: ## @@ -0,0 +1,383 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.nereids.rules.rewrite; + +import org.apache.doris.nereids.hint.DistributeHint; +import org.apache.doris.nereids.hint.Hint.HintStatus; +import org.apache.doris.nereids.pattern.MatchingContext; +import org.apache.doris.nereids.rules.Rule; +import org.apache.doris.nereids.rules.RuleType; +import org.apache.doris.nereids.rules.exploration.join.JoinReorderContext; +import org.apache.doris.nereids.trees.expressions.Alias; +import org.apache.doris.nereids.trees.expressions.Cast; +import org.apache.doris.nereids.trees.expressions.ComparisonPredicate; +import org.apache.doris.nereids.trees.expressions.EqualPredicate; +import org.apache.doris.nereids.trees.expressions.EqualTo; +import org.apache.doris.nereids.trees.expressions.Expression; +import org.apache.doris.nereids.trees.expressions.InPredicate; +import org.apache.doris.nereids.trees.expressions.IsNull; +import org.apache.doris.nereids.trees.expressions.NamedExpression; +import org.apache.doris.nereids.trees.expressions.Not; +import org.apache.doris.nereids.trees.expressions.NullSafeEqual; +import org.apache.doris.nereids.trees.expressions.Or; +import org.apache.doris.nereids.trees.expressions.Slot; +import org.apache.doris.nereids.trees.expressions.SlotReference; +import org.apache.doris.nereids.trees.expressions.functions.Function; +import org.apache.doris.nereids.trees.expressions.functions.generator.ExplodeNumbers; +import org.apache.doris.nereids.trees.expressions.functions.scalar.If; +import org.apache.doris.nereids.trees.expressions.functions.scalar.Random; +import org.apache.doris.nereids.trees.expressions.literal.BigIntLiteral; +import org.apache.doris.nereids.trees.expressions.literal.IntegerLiteral; +import org.apache.doris.nereids.trees.expressions.literal.Literal; +import org.apache.doris.nereids.trees.expressions.literal.NullLiteral; +import org.apache.doris.nereids.trees.plans.DistributeType; +import org.apache.doris.nereids.trees.plans.JoinType; +import org.apache.doris.nereids.trees.plans.Plan; +import org.apache.doris.nereids.trees.plans.algebra.SetOperation.Qualifier; +import org.apache.doris.nereids.trees.plans.logical.LogicalFilter; +import org.apache.doris.nereids.trees.plans.logical.LogicalGenerate; +import org.apache.doris.nereids.trees.plans.logical.LogicalJoin; +import org.apache.doris.nereids.trees.plans.logical.LogicalProject; +import org.apache.doris.nereids.trees.plans.logical.LogicalUnion; +import org.apache.doris.nereids.types.DataType; +import org.apache.doris.nereids.types.IntegerType; +import org.apache.doris.nereids.types.TinyIntType; +import org.apache.doris.nereids.util.TypeCoercionUtils; +import org.apache.doris.nereids.util.Utils; + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableSet; +import org.jetbrains.annotations.Nullable; + +import java.util.HashSet; +import java.util.List; +import java.util.Optional; +import java.util.Set; +import java.util.stream.Collectors; + +/** + * Current capabilities and limitations of SaltJoin rewrite handling: + * - Supports single-side skew in INNER JOIN, NOT support double-side (both tables) skew + * - Supports left table skew and NOT support right table skew in LEFT JOIN + * - Supports right table skew and Not support left table skew in RIGHT JOIN + * + * INNER JOIN and LEFT JOIN use case: + * Applicable when left table is skewed and right table is too large for broadcast + * + * Here are some examples in rewrite: + * case1: + * LogicalJoin(type:inner, t1.a=t2.a, hint:skew(t1.a(null,1,2))) + * +--LogicalOlapScan(t1) + * +--LogicalOlapScan(t2) + * -> + * LogicalJoin (type:inner, t1.a=t2.a and r1=r2) + * |--LogicalProject (t1.a, if (t1.a IN (1, 2), random(0, 999), DEFAULT_SALT_VALUE) AS r1)) + * | +--LogicalFilter(t1.a is not null) + * |+--LogicalOlapScan(t1) + * +--LogicalProject (projections: t2.a, if(explodeNumber IS NULL, DEFAULT_SALT_VALUE, explodeNumbe
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
hello-stephen commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-3056985364 # FE UT Coverage Report Increment line coverage ` 72.11% (424/588)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/49096_810172bd8e4b37265aaa7236c4b8e1816ab27414/fe_increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/49096_810172bd8e4b37265aaa7236c4b8e1816ab27414/fe_report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-3056894614 ClickBench: Total hot run time: 29.02 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 810172bd8e4b37265aaa7236c4b8e1816ab27414, data reload: false query1 0.040.030.03 query2 0.080.040.04 query3 0.250.070.07 query4 1.620.110.12 query5 0.450.430.45 query6 1.160.660.67 query7 0.020.020.02 query8 0.040.040.03 query9 0.600.510.52 query10 0.580.570.57 query11 0.160.110.11 query12 0.150.120.11 query13 0.620.610.62 query14 0.790.800.81 query15 0.890.870.85 query16 0.380.380.40 query17 1.041.111.08 query18 0.220.210.21 query19 1.931.871.81 query20 0.020.010.02 query21 15.39 0.880.53 query22 0.741.220.66 query23 14.91 1.360.64 query24 8.041.250.35 query25 0.450.280.10 query26 0.630.170.14 query27 0.060.060.05 query28 9.370.870.46 query29 12.64 4.043.36 query30 0.270.100.06 query31 2.830.600.38 query32 3.230.570.49 query33 3.093.053.08 query34 16.19 5.384.68 query35 4.794.824.84 query36 0.710.500.49 query37 0.090.070.06 query38 0.050.040.04 query39 0.030.030.02 query40 0.170.140.13 query41 0.080.020.02 query42 0.030.020.02 query43 0.040.030.03 Total cold run time: 104.87 s Total hot run time: 29.02 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-3056871442 TPC-DS: Total hot run time: 187430 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 810172bd8e4b37265aaa7236c4b8e1816ab27414, data reload: false query1 989 397 396 396 query2 5992177918441779 query3 5739211 211 211 query4 25982 24011 23318 23318 query5 3644566 443 443 query6 270 219 211 211 query7 2845497 294 294 query8 248 232 215 215 query9 6032267726652665 query10 396 344 275 275 query11 15571 14992 14889 14889 query12 171 114 101 101 query13 883 520 402 402 query14 8437613160186018 query15 212 206 168 168 query16 7810440 266 266 query17 1345733 600 600 query18 2035416 318 318 query19 195 186 187 186 query20 126 119 122 119 query21 222 128 110 110 query22 4016411740224022 query23 33892 33151 33054 33054 query24 8286235023592350 query25 542 484 390 390 query26 1436263 148 148 query27 2788502 348 348 query28 4390214921292129 query29 727 543 440 440 query30 283 224 189 189 query31 964 834 767 767 query32 73 65 61 61 query33 544 331 281 281 query34 814 848 515 515 query35 598 636 558 558 query36 941 962 876 876 query37 119 99 86 86 query38 4171409241234092 query39 1484144814901448 query40 229 133 106 106 query41 60 54 51 51 query42 123 113 115 113 query43 514 525 510 510 query44 1346820 829 820 query45 171 168 167 167 query46 846 1028631 631 query47 1718176217361736 query48 375 427 331 331 query49 736 473 381 381 query50 640 688 420 420 query51 5447548155635481 query52 113 107 98 98 query53 223 254 210 210 query54 576 584 503 503 query55 81 82 82 82 query56 297 306 291 291 query57 1173116911261126 query58 275 256 256 256 query59 2703278126532653 query60 319 313 300 300 query61 122 120 122 120 query62 787 755 643 643 query63 220 190 184 184 query64 42641219875 875 query65 4258419041694169 query66 1077421 317 317 query67 15619 15643 15219 15219 query68 8705881 529 529 query69 495 314 268 268 query70 1196109810501050 query71 478 325 313 313 query72 5603474448904744 query73 742 633 348 348 query74 9211884886748674 query75 3996317126922692 query76 36441136705 705 query77 779 395 296 296 query78 11044 10901 10263 10263 query79 2786822 593 593 query80 637 510 434 434 query81 480 260 228 228 query82 436 130 99 99 query83 286 247 235 235 query84 293 106 84 84 query85 773 382 319 319 query86 393 321 289 289 query87 4586446544034403 query88 3299226222392239 query89 394 315 291 291 query90 1867207 200 200 query91 139 147 114 114 query92 72 56 55 55 query93 2171925 576 576 query94 669 313 196 196 query95 374 291 283 283 query96 488 573 278 278 query97 2754274826372637 query98 245 204 277 204 query99 1419139112601260 Total cold run time: 269978 ms Total hot run time: 187430 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-3056832314 TPC-H: Total hot run time: 33188 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 810172bd8e4b37265aaa7236c4b8e1816ab27414, data reload: false -- Round 1 -- q1 17569 513750455045 q2 1936279 193 193 q3 10296 1278720 720 q4 10232 1018517 517 q5 7473242823552355 q6 177 158 130 130 q7 901 728 604 604 q8 9312129210611061 q9 6827507350955073 q10 6885239819901990 q11 504 288 275 275 q12 349 351 210 210 q13 17763 370730453045 q14 237 226 214 214 q15 549 491 482 482 q16 439 449 377 377 q17 596 854 363 363 q18 7527729771027102 q19 1203964 552 552 q20 341 358 218 218 q21 3868319223642364 q22 371 314 298 298 Total cold run time: 105355 ms Total hot run time: 33188 ms - Round 2, with runtime_filter_mode=off - q1 5074514051165116 q2 241 321 217 217 q3 2185273822842284 q4 1409183913661366 q5 4253414241424142 q6 204 166 120 120 q7 1872187816581658 q8 2504239924572399 q9 6871677867446744 q10 2938313926962696 q11 560 511 503 503 q12 638 737 586 586 q13 3305375131083108 q14 263 278 260 260 q15 511 468 468 468 q16 429 485 418 418 q17 1118152913311331 q18 7313711470047004 q19 784 746 763 746 q20 1918198218181818 q21 4763433342534253 q22 630 587 553 553 Total cold run time: 49783 ms Total hot run time: 47790 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
feiniaofeiafei commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-3056716491 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
feiniaofeiafei commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-3056503463 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
feiniaofeiafei commented on code in PR #49096:
URL: https://github.com/apache/doris/pull/49096#discussion_r2196906248
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownAliasThroughJoin.java:
##
@@ -97,9 +98,13 @@ public Rule build() {
List newHash =
replaceJoinConjuncts(join.getHashJoinConjuncts(), replaceMap);
List newOther =
replaceJoinConjuncts(join.getOtherJoinConjuncts(), replaceMap);
List newMark =
replaceJoinConjuncts(join.getMarkJoinConjuncts(), replaceMap);
-
+DistributeHint hint = join.getDistributeHint();
Review Comment:
No, it cannot be immutable because hints are stored in the statementContext.
If it were immutable, the hint state couldn't be tracked when executing the
EXPLAIN command.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
feiniaofeiafei commented on code in PR #49096:
URL: https://github.com/apache/doris/pull/49096#discussion_r2196855906
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSkewExpr.java:
##
@@ -0,0 +1,64 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.analysis;
+
+import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.hint.DistributeHint;
+import org.apache.doris.nereids.hint.JoinSkewInfo;
+import org.apache.doris.nereids.pattern.MatchingContext;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.util.TypeCoercionUtils;
+
+import com.google.common.collect.ImmutableList;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**bind skew hint in DistributeHint*/
+public class BindSkewExpr extends BindExpression {
+@Override
+public List buildRules() {
+return ImmutableList.of(
+RuleType.BINDING_SKEW_EXPR.build(
+logicalJoin().when(join ->
join.getDistributeHint().getSkewInfo() != null)
+.thenApply(this::bindSkewExpr))
+);
+}
+
+private LogicalJoin
bindSkewExpr(MatchingContext> ctx) {
Review Comment:
No, it can not, BindSkewExpr needs rule LeadingJoin finished.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
feiniaofeiafei commented on code in PR #49096: URL: https://github.com/apache/doris/pull/49096#discussion_r2194975850 ## fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/SaltJoin.java: ## @@ -0,0 +1,383 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.nereids.rules.rewrite; + +import org.apache.doris.nereids.hint.DistributeHint; +import org.apache.doris.nereids.hint.Hint.HintStatus; +import org.apache.doris.nereids.pattern.MatchingContext; +import org.apache.doris.nereids.rules.Rule; +import org.apache.doris.nereids.rules.RuleType; +import org.apache.doris.nereids.rules.exploration.join.JoinReorderContext; +import org.apache.doris.nereids.trees.expressions.Alias; +import org.apache.doris.nereids.trees.expressions.Cast; +import org.apache.doris.nereids.trees.expressions.ComparisonPredicate; +import org.apache.doris.nereids.trees.expressions.EqualPredicate; +import org.apache.doris.nereids.trees.expressions.EqualTo; +import org.apache.doris.nereids.trees.expressions.Expression; +import org.apache.doris.nereids.trees.expressions.InPredicate; +import org.apache.doris.nereids.trees.expressions.IsNull; +import org.apache.doris.nereids.trees.expressions.NamedExpression; +import org.apache.doris.nereids.trees.expressions.Not; +import org.apache.doris.nereids.trees.expressions.NullSafeEqual; +import org.apache.doris.nereids.trees.expressions.Or; +import org.apache.doris.nereids.trees.expressions.Slot; +import org.apache.doris.nereids.trees.expressions.SlotReference; +import org.apache.doris.nereids.trees.expressions.functions.Function; +import org.apache.doris.nereids.trees.expressions.functions.generator.ExplodeNumbers; +import org.apache.doris.nereids.trees.expressions.functions.scalar.If; +import org.apache.doris.nereids.trees.expressions.functions.scalar.Random; +import org.apache.doris.nereids.trees.expressions.literal.BigIntLiteral; +import org.apache.doris.nereids.trees.expressions.literal.IntegerLiteral; +import org.apache.doris.nereids.trees.expressions.literal.Literal; +import org.apache.doris.nereids.trees.expressions.literal.NullLiteral; +import org.apache.doris.nereids.trees.plans.DistributeType; +import org.apache.doris.nereids.trees.plans.JoinType; +import org.apache.doris.nereids.trees.plans.Plan; +import org.apache.doris.nereids.trees.plans.algebra.SetOperation.Qualifier; +import org.apache.doris.nereids.trees.plans.logical.LogicalFilter; +import org.apache.doris.nereids.trees.plans.logical.LogicalGenerate; +import org.apache.doris.nereids.trees.plans.logical.LogicalJoin; +import org.apache.doris.nereids.trees.plans.logical.LogicalProject; +import org.apache.doris.nereids.trees.plans.logical.LogicalUnion; +import org.apache.doris.nereids.types.DataType; +import org.apache.doris.nereids.types.IntegerType; +import org.apache.doris.nereids.types.TinyIntType; +import org.apache.doris.nereids.util.TypeCoercionUtils; +import org.apache.doris.nereids.util.Utils; + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableSet; +import org.jetbrains.annotations.Nullable; + +import java.util.HashSet; +import java.util.List; +import java.util.Optional; +import java.util.Set; +import java.util.stream.Collectors; + +/** + * Current capabilities and limitations of SaltJoin rewrite handling: + * - Supports single-side skew in INNER JOIN, NOT support double-side (both tables) skew + * - Supports left table skew and NOT support right table skew in LEFT JOIN + * - Supports right table skew and Not support left table skew in RIGHT JOIN + * + * INNER JOIN and LEFT JOIN use case: + * Applicable when left table is skewed and right table is too large for broadcast + * + * Here are some examples in rewrite: + * case1: + * LogicalJoin(type:inner, t1.a=t2.a, hint:skew(t1.a(null,1,2))) + * +--LogicalOlapScan(t1) + * +--LogicalOlapScan(t2) + * -> + * LogicalJoin (type:inner, t1.a=t2.a and r1=r2) + * |--LogicalProject (t1.a, if (t1.a IN (1, 2), random(0, 999), DEFAULT_SALT_VALUE) AS r1)) + * | +--LogicalFilter(t1.a is not null) + * |+--LogicalOlapScan(t1) + * +--LogicalProject (projections: t2.a, if(explodeNumber IS NULL, DEFAULT_SALT_VALUE, explodeNumbe
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
morrySnow commented on code in PR #49096:
URL: https://github.com/apache/doris/pull/49096#discussion_r2194792325
##
fe/fe-core/src/main/java/org/apache/doris/nereids/properties/DistributionSpecHash.java:
##
@@ -53,6 +53,8 @@ public class DistributionSpecHash extends DistributionSpec {
private final long tableId;
private final Set partitionIds;
private final long selectedIndexId;
+// used for window skew rewrite
+private final boolean isSkew;
Review Comment:
maybe the better way is we create specific orderSpec for must local sort
order requirement
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
morrySnow commented on code in PR #49096:
URL: https://github.com/apache/doris/pull/49096#discussion_r2194612710
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSkewExpr.java:
##
@@ -0,0 +1,64 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.analysis;
+
+import org.apache.doris.nereids.CascadesContext;
+import org.apache.doris.nereids.hint.DistributeHint;
+import org.apache.doris.nereids.hint.JoinSkewInfo;
+import org.apache.doris.nereids.pattern.MatchingContext;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalJoin;
+import org.apache.doris.nereids.util.TypeCoercionUtils;
+
+import com.google.common.collect.ImmutableList;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**bind skew hint in DistributeHint*/
+public class BindSkewExpr extends BindExpression {
+@Override
+public List buildRules() {
+return ImmutableList.of(
+RuleType.BINDING_SKEW_EXPR.build(
+logicalJoin().when(join ->
join.getDistributeHint().getSkewInfo() != null)
+.thenApply(this::bindSkewExpr))
+);
+}
+
+private LogicalJoin
bindSkewExpr(MatchingContext> ctx) {
Review Comment:
could this class merged into BindExpression?
##
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/algebra/Aggregate.java:
##
@@ -106,4 +111,23 @@ default boolean isDistinct() {
return getOutputExpressions().stream().allMatch(e -> e instanceof Slot)
&& getGroupByExpressions().stream().allMatch(e -> e instanceof
Slot);
}
+
+/**canSkewRewrite*/
Review Comment:
add comment to explain could rewrite situations
##
fe/fe-core/src/main/java/org/apache/doris/nereids/properties/DistributionSpecHash.java:
##
@@ -53,6 +53,8 @@ public class DistributionSpecHash extends DistributionSpec {
private final long tableId;
private final Set partitionIds;
private final long selectedIndexId;
+// used for window skew rewrite
+private final boolean isSkew;
Review Comment:
why process window need add a skew flag here?
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownAliasThroughJoin.java:
##
@@ -97,9 +98,13 @@ public Rule build() {
List newHash =
replaceJoinConjuncts(join.getHashJoinConjuncts(), replaceMap);
List newOther =
replaceJoinConjuncts(join.getOtherJoinConjuncts(), replaceMap);
List newMark =
replaceJoinConjuncts(join.getMarkJoinConjuncts(), replaceMap);
-
+DistributeHint hint = join.getDistributeHint();
Review Comment:
it is mutable? this is dangerous, could it be immutable?
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/LogicalWindowToPhysicalWindow.java:
##
@@ -95,10 +95,12 @@ private PhysicalWindow implement(LogicalWindow
logicalWindow) {
Plan newRoot = logicalWindow.child();
for (PartitionKeyGroup partitionKeyGroup : partitionKeyGroupList) {
-for (OrderKeyGroup orderKeyGroup : partitionKeyGroup.groups) {
+boolean isSkew = partitionKeyGroup.isSkew();
+for (int i = 0; i < partitionKeyGroup.groups.size(); ++i) {
+OrderKeyGroup orderKeyGroup = partitionKeyGroup.groups.get(i);
// in OrderKeyGroup, create PhysicalWindow for each
WindowFrameGroup;
// each PhysicalWindow contains the same windowExpressions as
WindowFrameGroup.groups
-newRoot = createPhysicalPlanNodeForWindowFrameGroup(newRoot,
orderKeyGroup);
+newRoot = createPhysicalPlanNodeForWindowFrameGroup(newRoot,
orderKeyGroup, 0 == i && isSkew);
Review Comment:
why need `0 == i`? could u add some comment to explain it?
##
fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java:
##
@@ -743,6 +743,10 @@ public class SessionVaria
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
starocean999 commented on code in PR #49096:
URL: https://github.com/apache/doris/pull/49096#discussion_r2194630821
##
regression-test/suites/nereids_rules_p0/salt_join/salt_join.groovy:
##
@@ -0,0 +1,282 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+suite("salt_join") {
+//sql "set disable_nereids_rules=JOIN_SKEW_ADD_SALT"
+sql "SET ignore_shape_nodes='PhysicalDistribute,PhysicalProject'"
+sql "set disable_nereids_rules='prune_empty_partition'"
+sql "set runtime_filter_mode=OFF"
+sql "drop table if exists test_skew9;"
+sql """create table test_skew9(a int,c varchar(100), b int) distributed by
hash(a) buckets 32 properties("replication_num"="1");"""
+sql """insert into test_skew9
values(1,'abc',9),(1,'abc',1),(1,'def',2),(null,'def',2),(2,'abc',2),(3,'abc',4),(5,'abc',6),(2,'def',2),(5,'abc',null),(3,'abc',null)"""
+sql "drop table if exists test_skew10;"
+sql """create table test_skew10(a int,c varchar(100), b int) distributed
by hash(a) buckets 32 properties("replication_num"="1");"""
+sql """insert into test_skew10
values(1,'abc',9),(null,'abc1',3),(1,'def',2),(2,'def1',7),(2,'def',2),(3,'abc',4),(5,'def',8),(2,'def',2),(5,'def',null),(3,'abc',null)"""
+qt_simple """
+select * from test_skew9 tl inner join [shuffle(skew(tl.b(1,2)))]
test_skew10 tr on tl.b = tr.b order by 1,2,3,4,5,6;
+"""
+qt_left_simple """
+select * from test_skew9 tl left join [shuffle(skew(tl.b(1,2)))]
test_skew10 tr on tl.b = tr.b order by 1,2,3,4,5,6;
+"""
+qt_right_simple """
+select * from test_skew9 tl right join [shuffle(skew(tl.b(1,2)))]
test_skew10 tr on tl.b = tr.b order by 1,2,3,4,5,6;
+"""
+qt_skew_value_null """
+select * from test_skew9 tl inner join [shuffle(skew(tl.b(1,2,null)))]
test_skew10 tr on tl.b = tr.b order by 1,2,3,4,5,6;
+"""
+qt_skew_value_only_null """
+select * from test_skew9 tl inner join [shuffle(skew(tl.b(null)))]
test_skew10 tr on tl.b = tr.b order by 1,2,3,4,5,6;
+"""
+qt_leading """
+select /*+leading(tl shuffle (skew(tl.b(1,2))) tr) */ * from test_skew9 tl
join test_skew10 tr on tl.b=tr.b order by 1,2,3,4,5,6;
+"""
+qt_has_other_equal_condition """
+select /*+leading(tl shuffle (skew(tl.b(1,2))) tr) */ * from test_skew9 tl
join test_skew10 tr on tl.b=tr.b and tl.a=tr.a order by 1,2,3,4,5,6;
+"""
+qt_has_other_unequal_condition """
+select /*+leading(tl shuffle (skew(tl.b(1,2))) tr) */ * from test_skew9 tl
join test_skew10 tr on tl.b=tr.b and tl.a
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
morrySnow commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2990052526 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
hello-stephen commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2972753929 # FE UT Coverage Report Increment line coverage ` 72.51% (430/593)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/49096_535125b4876db8371f669c1ba3f6b8b22c7ea8ed/fe_increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/49096_535125b4876db8371f669c1ba3f6b8b22c7ea8ed/fe_report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
feiniaofeiafei commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2972618949 run feut -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2969738458 ClickBench: Total hot run time: 28.47 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 535125b4876db8371f669c1ba3f6b8b22c7ea8ed, data reload: false query1 0.030.030.03 query2 0.070.030.04 query3 0.240.060.07 query4 1.610.100.10 query5 0.420.430.42 query6 1.180.660.67 query7 0.020.020.02 query8 0.040.030.03 query9 0.570.520.50 query10 0.570.580.57 query11 0.160.110.11 query12 0.150.120.11 query13 0.620.600.60 query14 0.790.810.80 query15 0.880.850.85 query16 0.410.370.38 query17 1.081.051.05 query18 0.230.210.21 query19 1.931.811.80 query20 0.020.010.01 query21 15.40 0.930.56 query22 0.751.180.72 query23 14.89 1.380.65 query24 6.771.340.44 query25 0.490.130.18 query26 0.610.160.14 query27 0.060.040.04 query28 9.670.960.45 query29 12.54 3.943.30 query30 0.250.090.06 query31 2.820.640.38 query32 3.230.560.47 query33 2.993.013.10 query34 15.84 5.054.45 query35 4.554.534.53 query36 0.690.490.48 query37 0.080.060.06 query38 0.050.040.04 query39 0.030.020.02 query40 0.160.130.13 query41 0.080.030.03 query42 0.040.030.02 query43 0.040.030.03 Total cold run time: 103.05 s Total hot run time: 28.47 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2969722352 TPC-DS: Total hot run time: 192157 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 535125b4876db8371f669c1ba3f6b8b22c7ea8ed, data reload: false query1 1393104310291029 query2 6380180717591759 query3 10965 460342824282 query4 55969 25403 23284 23284 query5 5158490 464 464 query6 345 203 201 201 query7 4886500 302 302 query8 297 232 208 208 query9 5754266626612661 query10 457 362 277 277 query11 15311 15035 14797 14797 query12 169 113 108 108 query13 1037519 408 408 query14 10156 632462466246 query15 203 201 175 175 query16 7085609 473 473 query17 1065701 618 618 query18 1578406 307 307 query19 205 193 171 171 query20 121 123 117 117 query21 207 124 106 106 query22 4750469444814481 query23 33998 33405 33472 33405 query24 6695243024722430 query25 510 492 449 449 query26 680 284 161 161 query27 2137524 349 349 query28 2970217922122179 query29 622 606 454 454 query30 283 221 187 187 query31 877 859 757 757 query32 74 66 67 66 query33 494 383 347 347 query34 799 862 543 543 query35 773 821 734 734 query36 938 1008897 897 query37 117 101 87 87 query38 4296423841984198 query39 1531144714291429 query40 214 120 103 103 query41 58 61 55 55 query42 131 107 118 107 query43 500 498 464 464 query44 1364859 876 859 query45 192 176 166 166 query46 847 1056715 715 query47 1811184917741774 query48 391 429 319 319 query49 664 504 393 393 query50 707 708 433 433 query51 4185426441824182 query52 112 123 102 102 query53 227 263 188 188 query54 580 600 522 522 query55 86 82 88 82 query56 318 323 297 297 query57 1210123011361136 query58 264 271 253 253 query59 2801276626232623 query60 343 347 320 320 query61 127 126 133 126 query62 736 784 705 705 query63 225 194 193 193 query64 15221062738 738 query65 4245421641694169 query66 727 426 317 317 query67 15923 15761 15392 15392 query68 7974902 530 530 query69 539 317 267 267 query70 1219112310481048 query71 485 314 293 293 query72 5933489549454895 query73 1413698 351 351 query74 9376922886698669 query75 3711318627352735 query76 42171173799 799 query77 612 351 284 284 query78 990310243 93349334 query79 2617813 581 581 query80 685 502 456 456 query81 490 255 213 213 query82 453 123 96 96 query83 356 248 237 237 query84 287 105 87 87 query85 812 353 309 309 query86 390 295 276 276 query87 4398443143274327 query88 2929229723082297 query89 399 310 281 281 query90 1946207 212 207 query91 142 176 110 110 query92 81 59 55 55 query93 1691941 597 597 query94 690 410 278 278 query95 364 304 284 284 query96 503 581 275 275 query97 2732273926472647 query98 224 205 204 204 query99 1421141012561256 Total cold run time: 300787 ms Total hot run time: 192157 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2969674138 TPC-H: Total hot run time: 33885 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 535125b4876db8371f669c1ba3f6b8b22c7ea8ed, data reload: false -- Round 1 -- q1 17573 508349984998 q2 1934265 154 154 q3 10376 1322762 762 q4 10330 1007540 540 q5 8667235723082308 q6 206 162 132 132 q7 907 744 610 610 q8 9308126510701070 q9 6757502350555023 q10 6814232918951895 q11 479 288 285 285 q12 340 352 214 214 q13 17751 364230423042 q14 242 223 214 214 q15 564 483 487 483 q16 414 422 381 381 q17 598 857 347 347 q18 7443722672377226 q19 1784959 563 563 q20 356 340 221 221 q21 3967262524112411 q22 1047102610061006 Total cold run time: 107857 ms Total hot run time: 33885 ms - Round 2, with runtime_filter_mode=off - q1 5102504150105010 q2 232 316 216 216 q3 2182264122572257 q4 1369179014221422 q5 4215410443294104 q6 226 176 133 133 q7 2027190517621762 q8 2623261725732573 q9 7181717671757175 q10 2995318526472647 q11 590 511 505 505 q12 702 795 647 647 q13 3467386032583258 q14 309 295 279 279 q15 538 489 492 489 q16 459 490 454 454 q17 1173156713341334 q18 7692745974177417 q19 817 823 919 823 q20 1973204318711871 q21 5021448044644464 q22 1064104010141014 Total cold run time: 51957 ms Total hot run time: 49854 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
feiniaofeiafei commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2969557463 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
feiniaofeiafei commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2968792987 run cloud_p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
hello-stephen commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2965669644 # FE UT Coverage Report Increment line coverage ` 75.76% (450/594)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/49096_b7fd1c21dd18ca5e0c21520f44204f8e17ef8050/fe_increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/49096_b7fd1c21dd18ca5e0c21520f44204f8e17ef8050/fe_report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2965471658 ClickBench: Total hot run time: 29.06 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit b7fd1c21dd18ca5e0c21520f44204f8e17ef8050, data reload: false query1 0.040.040.03 query2 0.140.110.11 query3 0.260.190.19 query4 1.600.190.11 query5 0.430.420.47 query6 1.170.660.65 query7 0.020.010.01 query8 0.040.030.03 query9 0.580.510.51 query10 0.570.570.57 query11 0.150.110.11 query12 0.140.120.12 query13 0.610.610.60 query14 0.780.820.80 query15 0.880.860.86 query16 0.380.380.39 query17 1.041.031.04 query18 0.220.220.21 query19 1.871.811.82 query20 0.020.020.01 query21 15.41 0.940.56 query22 0.761.200.75 query23 14.80 1.390.65 query24 6.751.420.64 query25 0.520.180.06 query26 0.640.150.15 query27 0.050.050.05 query28 9.450.880.45 query29 12.53 3.983.31 query30 0.250.090.07 query31 2.830.580.39 query32 3.230.560.47 query33 3.093.123.08 query34 15.77 5.184.56 query35 4.564.544.53 query36 0.670.500.48 query37 0.080.060.07 query38 0.050.040.04 query39 0.030.020.02 query40 0.180.140.12 query41 0.070.020.03 query42 0.040.030.02 query43 0.040.030.02 Total cold run time: 102.74 s Total hot run time: 29.06 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2965417140 TPC-H: Total hot run time: 33970 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit b7fd1c21dd18ca5e0c21520f44204f8e17ef8050, data reload: false -- Round 1 -- q1 26326 513350945094 q2 1942281 191 191 q3 10371 1280721 721 q4 10262 1000513 513 q5 8040239223192319 q6 183 172 132 132 q7 909 757 604 604 q8 9341128910421042 q9 6830508350975083 q10 6856229919071907 q11 489 306 268 268 q12 338 348 212 212 q13 17786 365431513151 q14 231 237 217 217 q15 564 485 482 482 q16 436 437 370 370 q17 598 844 379 379 q18 7651719971417141 q19 1592980 571 571 q20 347 341 228 228 q21 3890324023592359 q22 10501011986 986 Total cold run time: 116032 ms Total hot run time: 33970 ms - Round 2, with runtime_filter_mode=off - q1 5172504950205020 q2 243 322 220 220 q3 2224263023262326 q4 1353190215521552 q5 4507436443394339 q6 218 174 127 127 q7 1958192417661766 q8 2564262425242524 q9 7234707771837077 q10 2992316128022802 q11 592 523 514 514 q12 712 778 597 597 q13 3512393932813281 q14 280 315 269 269 q15 525 471 487 471 q16 462 490 463 463 q17 1190155313761376 q18 7696745573787378 q19 788 875 986 875 q20 1993205018541854 q21 4819436043394339 q22 11141007973 973 Total cold run time: 52148 ms Total hot run time: 50143 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
feiniaofeiafei commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2965347126 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
feiniaofeiafei commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2966407149 run vault_p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
feiniaofeiafei commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2966406882 run cloud_p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2965455622 TPC-DS: Total hot run time: 184348 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit b7fd1c21dd18ca5e0c21520f44204f8e17ef8050, data reload: false query1 1013480 487 480 query2 6559179017691769 query3 6762221 214 214 query4 26252 23318 22985 22985 query5 4359643 443 443 query6 328 225 202 202 query7 4625497 306 306 query8 267 225 212 212 query9 8642263426382634 query10 489 315 264 264 query11 15515 15065 14807 14807 query12 157 112 111 111 query13 1661514 431 431 query14 8783617960936093 query15 202 197 174 174 query16 7223631 500 500 query17 1202735 577 577 query18 1958400 299 299 query19 191 189 153 153 query20 123 113 115 113 query21 218 124 112 112 query22 4043412440134013 query23 33927 33099 32878 32878 query24 8467237123502350 query25 565 453 382 382 query26 1232266 173 173 query27 2738496 330 330 query28 4342212020922092 query29 782 569 436 436 query30 281 216 185 185 query31 926 853 751 751 query32 70 65 62 62 query33 558 371 303 303 query34 799 855 529 529 query35 778 807 735 735 query36 921 981 867 867 query37 111 103 79 79 query38 4105406840884068 query39 1464141614121412 query40 222 119 109 109 query41 62 61 67 61 query42 118 106 106 106 query43 494 515 469 469 query44 1305808 796 796 query45 183 172 168 168 query46 842 1018616 616 query47 1742179116971697 query48 390 418 299 299 query49 754 485 395 395 query50 636 671 412 412 query51 4144408841144088 query52 109 108 95 95 query53 216 249 187 187 query54 581 587 492 492 query55 86 112 82 82 query56 305 310 285 285 query57 1173115011091109 query58 267 257 252 252 query59 2492259525262526 query60 343 315 300 300 query61 123 125 124 124 query62 785 710 673 673 query63 219 191 186 186 query64 44171023687 687 query65 4218420441344134 query66 1175405 306 306 query67 15702 15358 15181 15181 query68 6799878 525 525 query69 491 298 262 262 query70 1193109211091092 query71 410 308 311 308 query72 5717468948244689 query73 681 640 356 356 query74 8879921785918591 query75 3164321026892689 query76 31961180741 741 query77 477 382 295 295 query78 990810133 92899289 query79 1859782 562 562 query80 568 516 452 452 query81 486 253 218 218 query82 175 126 95 95 query83 258 254 231 231 query84 252 114 90 90 query85 762 354 312 312 query86 380 307 292 292 query87 4358430843964308 query88 3462228122812281 query89 379 316 283 283 query90 1978209 211 209 query91 143 142 118 118 query92 78 63 57 57 query93 1840917 566 566 query94 673 398 317 317 query95 373 289 287 287 query96 489 569 284 284 query97 2688277127082708 query98 235 209 204 204 query99 1323139812781278 Total cold run time: 269415 ms Total hot run time: 184348 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
github-actions[bot] commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2934666518 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
github-actions[bot] commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-293437 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
feiniaofeiafei commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2933118608 run external -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
feiniaofeiafei commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2921883766 run external -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
feiniaofeiafei commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2921779491 run feut -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
feiniaofeiafei commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2921779175 run external -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2921384286 ClickBench: Total hot run time: 29.49 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 48ce695654a0e7e924ce4ec3e0b0f588e87a782b, data reload: false query1 0.040.050.03 query2 0.120.100.11 query3 0.240.200.19 query4 1.600.190.11 query5 0.450.400.44 query6 1.160.670.67 query7 0.020.020.02 query8 0.040.030.03 query9 0.600.510.52 query10 0.570.590.56 query11 0.160.100.11 query12 0.160.120.11 query13 0.610.610.60 query14 0.780.800.82 query15 0.880.850.85 query16 0.370.370.39 query17 1.051.041.04 query18 0.230.220.21 query19 1.911.811.82 query20 0.020.010.01 query21 15.43 0.860.56 query22 0.751.270.78 query23 14.77 1.370.65 query24 7.081.431.27 query25 0.550.190.06 query26 0.560.170.15 query27 0.050.050.06 query28 10.19 0.920.46 query29 12.60 4.023.27 query30 0.260.090.08 query31 2.820.620.39 query32 3.230.550.47 query33 3.063.113.03 query34 15.75 5.144.43 query35 4.454.524.52 query36 0.650.490.48 query37 0.090.070.06 query38 0.050.040.04 query39 0.030.020.03 query40 0.160.150.13 query41 0.070.020.02 query42 0.030.020.02 query43 0.040.030.03 Total cold run time: 103.68 s Total hot run time: 29.49 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2921374671 TPC-DS: Total hot run time: 192969 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 48ce695654a0e7e924ce4ec3e0b0f588e87a782b, data reload: false query1 1423107310491049 query2 6262181718181817 query3 11030 460644924492 query4 53874 25391 23216 23216 query5 5073501 480 480 query6 342 199 222 199 query7 4939495 296 296 query8 287 228 207 207 query9 6001268026752675 query10 446 344 270 270 query11 14958 14985 15003 14985 query12 160 110 111 110 query13 1070553 410 410 query14 10264 655065166516 query15 211 206 193 193 query16 7123615 502 502 query17 1076781 624 624 query18 1578420 324 324 query19 207 206 201 201 query20 142 126 123 123 query21 209 128 115 115 query22 4418441842174217 query23 34358 33641 33668 33641 query24 7142247525072475 query25 455 464 413 413 query26 726 282 153 153 query27 2506520 343 343 query28 3023217621572157 query29 593 587 462 462 query30 274 220 196 196 query31 872 874 757 757 query32 72 63 64 63 query33 489 388 311 311 query34 784 868 538 538 query35 775 847 781 781 query36 950 1007883 883 query37 109 98 77 77 query38 4279426742424242 query39 1527145515601455 query40 234 125 109 109 query41 63 56 58 56 query42 126 126 119 119 query43 509 512 475 475 query44 1427860 856 856 query45 179 188 178 178 query46 872 1041666 666 query47 1934193418201820 query48 399 424 326 326 query49 678 511 416 416 query50 670 730 423 423 query51 4300429142494249 query52 117 109 112 109 query53 236 268 190 190 query54 582 573 514 514 query55 88 89 84 84 query56 320 308 279 279 query57 1121116210991099 query58 277 251 248 248 query59 2576262925072507 query60 328 354 299 299 query61 127 125 124 124 query62 723 734 654 654 query63 229 194 189 189 query64 17961037668 668 query65 4233412041174117 query66 726 397 306 306 query67 16111 15683 15324 15324 query68 7824918 545 545 query69 541 311 270 270 query70 1251108910701070 query71 525 310 324 310 query72 5875477047744770 query73 1220631 355 355 query74 9111883589938835 query75 3720320926902690 query76 41371190790 790 query77 602 379 291 291 query78 10166 10156 93769376 query79 2211849 581 581 query80 608 508 447 447 query81 496 256 218 218 query82 439 131 103 103 query83 247 247 234 234 query84 299 107 90 90 query85 821 370 314 314 query86 413 291 295 291 query87 4404438543624362 query88 3494239423152315 query89 411 321 295 295 query90 1754206 222 206 query91 156 141 123 123 query92 78 60 62 60 query93 1846946 591 591 query94 658 406 308 308 query95 369 292 292 292 query96 503 575 284 284 query97 2736274926642664 query98 230 208 203 203 query99 1433141813141314 Total cold run time: 299586 ms Total hot run time: 192969 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2921354367 TPC-H: Total hot run time: 33918 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 48ce695654a0e7e924ce4ec3e0b0f588e87a782b, data reload: false -- Round 1 -- q1 26310 511950695069 q2 1974272 175 175 q3 10299 1262700 700 q4 10230 1003529 529 q5 7552227123772271 q6 191 168 134 134 q7 895 758 625 625 q8 9306126111101110 q9 6766512750435043 q10 6821227918781878 q11 489 284 280 280 q12 349 359 214 214 q13 17774 367730993099 q14 217 226 220 220 q15 562 490 495 490 q16 429 437 371 371 q17 602 845 355 355 q18 7667722471947194 q19 1220943 563 563 q20 336 342 222 222 q21 3764325224142414 q22 10631003962 962 Total cold run time: 114816 ms Total hot run time: 33918 ms - Round 2, with runtime_filter_mode=off - q1 5085507550875075 q2 245 318 220 220 q3 2179266022932293 q4 1365177413241324 q5 4396438444354384 q6 208 167 128 128 q7 2025191618171817 q8 2581263525082508 q9 7277725670647064 q10 3012318127722772 q11 592 523 501 501 q12 672 732 605 605 q13 3493389733613361 q14 282 321 295 295 q15 520 489 487 487 q16 442 498 449 449 q17 1151149813991399 q18 7796766374237423 q19 822 855 890 855 q20 2007199918281828 q21 4908443745444437 q22 10521056995 995 Total cold run time: 52110 ms Total hot run time: 50220 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
feiniaofeiafei commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2921318546 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2921114689 ClickBench: Total hot run time: 28.65 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 3958723eb737392e08e9100b4e97397c8615dafe, data reload: false query1 0.040.030.04 query2 0.120.110.10 query3 0.260.190.19 query4 1.600.190.20 query5 0.450.440.44 query6 1.150.680.67 query7 0.020.020.02 query8 0.050.040.04 query9 0.590.530.53 query10 0.570.580.58 query11 0.160.110.12 query12 0.150.110.12 query13 0.620.590.61 query14 0.840.800.81 query15 0.890.860.87 query16 0.380.380.37 query17 1.081.071.04 query18 0.230.210.21 query19 1.991.831.86 query20 0.010.010.01 query21 15.39 0.910.53 query22 0.781.230.65 query23 14.87 1.420.65 query24 6.880.960.37 query25 0.500.260.10 query26 0.690.160.14 query27 0.060.050.04 query28 9.690.950.44 query29 12.52 3.993.27 query30 0.260.090.06 query31 2.810.620.38 query32 3.230.550.48 query33 3.133.053.15 query34 15.70 5.124.49 query35 4.534.604.47 query36 0.660.490.48 query37 0.080.060.06 query38 0.050.040.04 query39 0.030.020.02 query40 0.170.140.13 query41 0.080.030.03 query42 0.040.030.02 query43 0.040.040.03 Total cold run time: 103.39 s Total hot run time: 28.65 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2921108487 TPC-DS: Total hot run time: 193562 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 3958723eb737392e08e9100b4e97397c8615dafe, data reload: false query1 1442107710701070 query2 6524180818161808 query3 11143 458544974497 query4 25681 23804 23532 23532 query5 4409623 465 465 query6 336 220 208 208 query7 3987506 304 304 query8 280 230 216 216 query9 8499264026642640 query10 476 334 283 283 query11 15436 14951 14878 14878 query12 164 109 109 109 query13 1557531 416 416 query14 8977607361496073 query15 212 188 179 179 query16 7658648 485 485 query17 1195766 627 627 query18 2071429 318 318 query19 201 204 181 181 query20 127 123 126 123 query21 223 129 114 114 query22 442443054305 query23 34558 33942 33853 33853 query24 8502241324462413 query25 521 474 415 415 query26 1192269 151 151 query27 2798504 357 357 query28 4758218921642164 query29 702 561 445 445 query30 274 222 192 192 query31 905 842 796 796 query32 71 63 60 60 query33 532 361 314 314 query34 828 901 566 566 query35 823 841 766 766 query36 981 1005881 881 query37 123 97 76 76 query38 4349423342364233 query39 1566149814921492 query40 216 122 109 109 query41 61 57 55 55 query42 130 111 127 111 query43 507 499 466 466 query44 1343860 836 836 query45 182 174 161 161 query46 898 1035653 653 query47 1830192118331833 query48 397 426 329 329 query49 738 486 418 418 query50 653 687 400 400 query51 4265426642224222 query52 117 106 100 100 query53 238 260 191 191 query54 601 581 513 513 query55 89 83 85 83 query56 322 314 290 290 query57 1175121411601160 query58 277 264 267 264 query59 2694284927462746 query60 341 334 315 315 query61 127 158 125 125 query62 772 750 681 681 query63 230 187 186 186 query64 39961037703 703 query65 4298428442884284 query66 981 431 306 306 query67 16268 15795 15456 15456 query68 8556895 535 535 query69 491 311 269 269 query70 1231108810771077 query71 474 315 294 294 query72 5847468145904590 query73 675 562 375 375 query74 9328913089798979 query75 3936322626952695 query76 37541189757 757 query77 790 373 305 305 query78 10196 10293 94759475 query79 1936820 590 590 query80 610 510 446 446 query81 478 255 223 223 query82 425 124 93 93 query83 255 253 231 231 query84 246 111 89 89 query85 846 355 323 323 query86 331 322 301 301 query87 4464451443114311 query88 3307227822752275 query89 411 321 290 290 query90 1896211 214 211 query91 143 145 109 109 query92 81 60 56 56 query93 1554948 584 584 query94 674 427 311 311 query95 374 295 290 290 query96 492 578 284 284 query97 2709282827162716 query98 236 217 210 210 query99 1464143612671267 Total cold run time: 280481 ms Total hot run time: 193562 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2921094843 TPC-H: Total hot run time: 33936 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 3958723eb737392e08e9100b4e97397c8615dafe, data reload: false -- Round 1 -- q1 26070 500649784978 q2 1959261 172 172 q3 10298 1235706 706 q4 10215 987 511 511 q5 7495238323002300 q6 179 161 131 131 q7 889 759 606 606 q8 9301129510911091 q9 6854513950845084 q10 7083232019391939 q11 512 299 286 286 q12 358 358 220 220 q13 18230 382832953295 q14 238 229 224 224 q15 580 501 503 501 q16 447 448 376 376 q17 591 864 363 363 q18 7545711270477047 q19 1782960 549 549 q20 323 334 219 219 q21 3628312323622362 q22 10581036976 976 Total cold run time: 115635 ms Total hot run time: 33936 ms - Round 2, with runtime_filter_mode=off - q1 5144503650185018 q2 237 315 220 220 q3 2108264223302330 q4 1356176013211321 q5 4446444043854385 q6 224 173 126 126 q7 1980189617471747 q8 2577254624882488 q9 7266721770617061 q10 3017319727692769 q11 582 504 509 504 q12 686 747 619 619 q13 3505382032623262 q14 289 312 279 279 q15 560 491 499 491 q16 456 471 427 427 q17 1137153214011401 q18 7859763075117511 q19 810 881 990 881 q20 2026206618261826 q21 4863460244404440 q22 1091106910351035 Total cold run time: 52219 ms Total hot run time: 50141 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
feiniaofeiafei commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2921051634 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2919509538 ClickBench: Total hot run time: 29.32 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 9c3453365e558ad137ba51591d3b7edbf17474b2, data reload: false query1 0.040.040.03 query2 0.130.100.10 query3 0.240.200.19 query4 1.580.200.19 query5 0.480.460.46 query6 1.180.680.66 query7 0.020.020.02 query8 0.050.040.03 query9 0.610.510.52 query10 0.580.580.57 query11 0.160.110.10 query12 0.150.120.12 query13 0.620.600.60 query14 0.830.810.80 query15 0.880.860.86 query16 0.390.400.38 query17 1.031.011.03 query18 0.220.210.21 query19 1.921.891.86 query20 0.010.010.01 query21 15.40 0.910.56 query22 0.751.070.63 query23 15.11 1.390.62 query24 6.991.430.94 query25 0.430.290.16 query26 0.530.170.14 query27 0.050.050.05 query28 9.580.920.44 query29 12.58 3.943.27 query30 0.250.090.07 query31 2.820.600.38 query32 3.500.560.46 query33 3.123.113.16 query34 15.82 5.124.50 query35 4.534.564.50 query36 0.660.490.47 query37 0.090.070.06 query38 0.050.040.04 query39 0.030.020.03 query40 0.160.130.12 query41 0.080.020.02 query42 0.030.020.02 query43 0.040.030.03 Total cold run time: 103.72 s Total hot run time: 29.32 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2919495593 TPC-DS: Total hot run time: 191805 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 9c3453365e558ad137ba51591d3b7edbf17474b2, data reload: false query1 143710571057 query2 6356178217761776 query3 10994 465344094409 query4 54638 25607 23274 23274 query5 5081519 464 464 query6 338 196 206 196 query7 4883520 290 290 query8 282 238 213 213 query9 5559261926172617 query10 431 337 261 261 query11 15078 15096 14883 14883 query12 166 106 109 106 query13 1030518 398 398 query14 10260 643866036438 query15 221 207 179 179 query16 7095640 487 487 query17 1082755 619 619 query18 1537437 314 314 query19 212 206 173 173 query20 132 142 121 121 query21 238 127 108 108 query22 4395438942494249 query23 34433 33563 33645 33563 query24 7021241624302416 query25 464 476 428 428 query26 728 283 152 152 query27 2428520 345 345 query28 3473218021602160 query29 568 577 432 432 query30 287 217 204 204 query31 874 887 755 755 query32 70 63 60 60 query33 457 384 316 316 query34 807 873 547 547 query35 809 827 762 762 query36 943 974 908 908 query37 114 103 79 79 query38 4288437242734273 query39 1518149114811481 query40 212 123 117 117 query41 65 59 58 58 query42 129 108 118 108 query43 497 510 483 483 query44 1393851 856 851 query45 179 179 175 175 query46 912 1063665 665 query47 1839186517671767 query48 409 437 334 334 query49 661 518 397 397 query50 684 715 426 426 query51 4214430742224222 query52 111 116 109 109 query53 239 259 189 189 query54 580 585 509 509 query55 85 90 85 85 query56 312 315 311 311 query57 1139117610941094 query58 265 283 262 262 query59 2715264226192619 query60 329 329 338 329 query61 163 123 127 123 query62 760 741 682 682 query63 228 193 196 193 query64 17261033666 666 query65 4194415041274127 query66 729 410 309 309 query67 15857 15829 15448 15448 query68 7696906 520 520 query69 537 304 271 271 query70 1153108010711071 query71 465 319 309 309 query72 5784485339073907 query73 1187735 355 355 query74 9291922686728672 query75 3253319226822682 query76 38221182788 788 query77 543 478 283 283 query78 10394 10051 94389438 query79 2683809 580 580 query80 699 519 469 469 query81 483 257 223 223 query82 605 126 96 96 query83 256 257 241 241 query84 293 117 97 97 query85 789 355 362 355 query86 380 302 294 294 query87 4431448544244424 query88 3478230123002300 query89 414 311 280 280 query90 1724212 215 212 query91 141 145 111 111 query92 70 62 62 62 query93 2440971 582 582 query94 737 422 296 296 query95 370 307 287 287 query96 499 584 284 284 query97 2724275526672667 query98 250 214 210 210 query99 1649141912681268 Total cold run time: 300593 ms Total hot run time: 191805 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2919453538 TPC-H: Total hot run time: 33783 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 9c3453365e558ad137ba51591d3b7edbf17474b2, data reload: false -- Round 1 -- q1 26083 520850435043 q2 1962266 179 179 q3 10288 1227679 679 q4 10238 998 528 528 q5 7564238122892289 q6 175 164 131 131 q7 902 738 592 592 q8 9303127911161116 q9 6884503350865033 q10 6856230618931893 q11 492 290 269 269 q12 346 344 215 215 q13 17778 367331233123 q14 235 235 222 222 q15 560 473 495 473 q16 442 436 373 373 q17 574 858 372 372 q18 7597714272247142 q19 1690947 601 601 q20 332 330 209 209 q21 3825310623432343 q22 10581019958 958 Total cold run time: 115184 ms Total hot run time: 33783 ms - Round 2, with runtime_filter_mode=off - q1 5220508250445044 q2 233 310 223 223 q3 2152264622752275 q4 1415175713391339 q5 4468436343994363 q6 231 166 122 122 q7 2000195717771777 q8 2787262426162616 q9 7186717069226922 q10 2979317227772777 q11 578 523 510 510 q12 669 762 587 587 q13 3557391632543254 q14 298 295 268 268 q15 513 477 501 477 q16 449 501 435 435 q17 1134147714051405 q18 7720746573557355 q19 816 849 932 849 q20 1971205618951895 q21 4780470645054505 q22 1104104310021002 Total cold run time: 52260 ms Total hot run time: 5 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
feiniaofeiafei commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2919379997 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2918708288 ClickBench: Total hot run time: 29.23 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 0fad94aec5d227985bd3d3f82a42609c014f47e8, data reload: false query1 0.040.040.03 query2 0.120.100.12 query3 0.240.200.19 query4 1.600.190.19 query5 0.460.450.45 query6 1.390.660.67 query7 0.030.020.02 query8 0.040.040.03 query9 0.590.540.50 query10 0.570.570.59 query11 0.160.100.12 query12 0.140.120.12 query13 0.630.600.60 query14 0.800.790.83 query15 0.900.860.86 query16 0.390.400.39 query17 1.061.041.05 query18 0.230.210.21 query19 1.951.801.77 query20 0.010.010.01 query21 15.39 0.930.56 query22 0.781.240.67 query23 14.88 1.380.69 query24 6.991.590.92 query25 0.570.250.06 query26 0.590.170.14 query27 0.060.050.05 query28 9.630.910.47 query29 12.58 4.033.29 query30 0.250.100.07 query31 2.810.630.39 query32 3.220.550.46 query33 3.133.043.10 query34 15.76 5.094.48 query35 4.504.534.55 query36 0.660.500.48 query37 0.090.070.06 query38 0.050.040.03 query39 0.030.030.02 query40 0.160.130.13 query41 0.070.020.02 query42 0.040.020.02 query43 0.040.030.02 Total cold run time: 103.63 s Total hot run time: 29.23 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2918696350 TPC-DS: Total hot run time: 193590 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 0fad94aec5d227985bd3d3f82a42609c014f47e8, data reload: false query1 1381107611091076 query2 6203179617871787 query3 11007 446246624462 query4 55032 25631 23521 23521 query5 5182499 479 479 query6 372 207 192 192 query7 4967511 283 283 query8 310 225 226 225 query9 6221260926512609 query10 442 339 276 276 query11 15083 15040 14877 14877 query12 158 117 105 105 query13 1123534 412 412 query14 10250 649666336496 query15 223 206 190 190 query16 7047675 504 504 query17 1099739 633 633 query18 1600416 343 343 query19 208 201 186 186 query20 141 125 122 122 query21 209 130 106 106 query22 4228440442594259 query23 34431 33519 33602 33519 query24 6635240624642406 query25 487 473 415 415 query26 718 285 151 151 query27 2272515 350 350 query28 3083216221572157 query29 617 580 460 460 query30 272 226 191 191 query31 855 869 770 770 query32 73 60 68 60 query33 461 373 312 312 query34 802 867 556 556 query35 815 837 758 758 query36 934 983 897 897 query37 113 98 79 79 query38 4370433141584158 query39 1521147014611461 query40 204 127 112 112 query41 60 61 59 59 query42 139 114 108 108 query43 503 514 487 487 query44 1379863 849 849 query45 183 203 178 178 query46 869 1020672 672 query47 1796187117721772 query48 393 440 326 326 query49 676 498 411 411 query50 669 732 420 420 query51 4249432242664266 query52 124 115 105 105 query53 228 259 188 188 query54 576 590 523 523 query55 87 83 86 83 query56 312 316 302 302 query57 1165118411071107 query58 278 280 262 262 query59 2727278226142614 query60 330 333 333 333 query61 139 131 125 125 query62 752 728 689 689 query63 244 201 202 201 query64 18531041747 747 query65 4276417541654165 query66 729 418 300 300 query67 15994 15771 15494 15494 query68 7538844 541 541 query69 540 305 276 276 query70 1219109011121090 query71 499 324 289 289 query72 6059479550044795 query73 1332660 356 356 query74 8961889891378898 query75 3829337527342734 query76 41981192803 803 query77 632 402 296 296 query78 10278 10201 94659465 query79 2456830 585 585 query80 638 535 458 458 query81 475 258 216 216 query82 464 133 101 101 query83 256 262 239 239 query84 296 116 91 91 query85 839 363 326 326 query86 401 329 296 296 query87 4367442944614429 query88 3589228623002286 query89 408 327 285 285 query90 1749221 215 215 query91 143 144 114 114 query92 79 60 55 55 query93 2137972 590 590 query94 685 418 302 302 query95 373 302 283 283 query96 509 571 285 285 query97 2719280226812681 query98 240 211 201 201 query99 1451138812681268 Total cold run time: 301238 ms Total hot run time: 193590 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2918668634 TPC-H: Total hot run time: 33867 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 0fad94aec5d227985bd3d3f82a42609c014f47e8, data reload: false -- Round 1 -- q1 25892 504850045004 q2 1994280 187 187 q3 10249 1243682 682 q4 10221 1027519 519 q5 7532237522822282 q6 183 159 131 131 q7 904 728 608 608 q8 9296122210301030 q9 6956512551445125 q10 6854235019191919 q11 490 305 280 280 q12 340 342 214 214 q13 17776 375731043104 q14 235 219 209 209 q15 566 490 511 490 q16 428 451 379 379 q17 577 826 384 384 q18 7820725573327255 q19 1431995 574 574 q20 332 334 217 217 q21 3607255623362336 q22 1060992 938 938 Total cold run time: 114743 ms Total hot run time: 33867 ms - Round 2, with runtime_filter_mode=off - q1 5211513951095109 q2 234 321 224 224 q3 2210265322632263 q4 1374184413451345 q5 4397439143724372 q6 226 184 127 127 q7 2031195718091809 q8 2607273525982598 q9 7129715370187018 q10 3052320628342834 q11 593 511 492 492 q12 651 762 604 604 q13 3617395833113311 q14 276 305 273 273 q15 540 498 492 492 q16 447 510 473 473 q17 1128152714321432 q18 7962756974197419 q19 784 854 870 854 q20 2007208019781978 q21 4823446243764376 q22 1092108910201020 Total cold run time: 52391 ms Total hot run time: 50423 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
feiniaofeiafei commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2918582611 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2909835651 TPC-DS: Total hot run time: 186243 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit f69931ae053761b22f25bb6a0a1f520146edb003, data reload: false query1 1008484 508 484 query2 6551183118161816 query3 6745231 226 226 query4 26296 23758 23167 23167 query5 4380645 506 506 query6 326 219 215 215 query7 4652489 314 314 query8 307 255 243 243 query9 8603263326232623 query10 491 334 258 258 query11 15268 15075 14876 14876 query12 162 115 102 102 query13 1657563 406 406 query14 9378614961426142 query15 204 199 172 172 query16 7189626 476 476 query17 1169719 570 570 query18 2002398 310 310 query19 193 198 167 167 query20 123 116 130 116 query21 219 128 112 112 query22 4175418340854085 query23 34170 33095 33213 33095 query24 8485244024212421 query25 588 502 427 427 query26 1245274 155 155 query27 2744531 346 346 query28 4380211820932093 query29 819 569 468 468 query30 286 273 190 190 query31 946 869 771 771 query32 73 62 68 62 query33 554 376 335 335 query34 799 845 531 531 query35 780 815 748 748 query36 939 974 889 889 query37 113 94 78 78 query38 4131407840634063 query39 1470147714291429 query40 217 125 107 107 query41 57 58 55 55 query42 122 108 113 108 query43 518 509 478 478 query44 1353829 826 826 query45 179 179 167 167 query46 832 1021651 651 query47 1749176817261726 query48 386 441 313 313 query49 769 515 425 425 query50 671 685 406 406 query51 4148417940804080 query52 110 113 101 101 query53 231 260 187 187 query54 587 587 510 510 query55 85 81 83 81 query56 314 310 289 289 query57 1154115010961096 query58 267 254 248 248 query59 2606267225612561 query60 329 320 330 320 query61 132 125 124 124 query62 815 720 638 638 query63 234 189 196 189 query64 44271005681 681 query65 4319427042674267 query66 1143420 320 320 query67 15994 15581 15391 15391 query68 7885921 515 515 query69 505 312 273 273 query70 1208111711041104 query71 468 329 308 308 query72 5468468048584680 query73 682 609 360 360 query74 8937899387638763 query75 3845320426822682 query76 36691193768 768 query77 811 384 291 291 query78 10111 10212 92959295 query79 2572853 588 588 query80 604 539 465 465 query81 493 250 216 216 query82 455 127 101 101 query83 284 254 249 249 query84 297 111 81 81 query85 800 386 416 386 query86 395 321 303 303 query87 4421445743884388 query88 3504233623352335 query89 383 327 282 282 query90 1857223 216 216 query91 146 143 111 111 query92 137 68 62 62 query93 2046940 581 581 query94 671 401 291 291 query95 386 299 292 292 query96 506 584 284 284 query97 2730275826552655 query98 225 216 215 215 query99 1437139812711271 Total cold run time: 275306 ms Total hot run time: 186243 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2909849501 ClickBench: Total hot run time: 28.57 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit f69931ae053761b22f25bb6a0a1f520146edb003, data reload: false query1 0.040.030.04 query2 0.130.100.11 query3 0.260.200.20 query4 1.600.190.18 query5 0.450.440.45 query6 1.130.660.65 query7 0.020.020.02 query8 0.040.030.04 query9 0.610.520.51 query10 0.580.590.57 query11 0.150.110.11 query12 0.140.120.11 query13 0.600.600.59 query14 0.770.800.80 query15 0.890.850.86 query16 0.380.370.37 query17 1.021.051.01 query18 0.220.210.20 query19 1.921.841.86 query20 0.010.010.02 query21 15.42 0.890.55 query22 0.761.290.72 query23 14.74 1.360.66 query24 6.992.030.39 query25 0.320.350.06 query26 0.580.170.13 query27 0.060.050.05 query28 9.200.900.43 query29 12.59 3.973.33 query30 0.260.090.07 query31 2.820.580.38 query32 3.230.550.47 query33 3.003.103.05 query34 15.91 5.104.41 query35 4.484.544.46 query36 0.680.490.48 query37 0.090.070.06 query38 0.060.040.03 query39 0.030.030.02 query40 0.160.140.13 query41 0.080.030.03 query42 0.030.020.02 query43 0.030.030.02 Total cold run time: 102.48 s Total hot run time: 28.57 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2909801438 TPC-H: Total hot run time: 34063 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit f69931ae053761b22f25bb6a0a1f520146edb003, data reload: false -- Round 1 -- q1 26324 512851355128 q2 2057284 192 192 q3 10381 1276712 712 q4 10234 1001540 540 q5 7532247823282328 q6 182 163 135 135 q7 898 742 608 608 q8 9322126611051105 q9 6832509951225099 q10 6819234818911891 q11 494 295 271 271 q12 347 349 217 217 q13 17791 370630783078 q14 234 226 221 221 q15 540 491 489 489 q16 421 435 370 370 q17 576 858 362 362 q18 7563718571337133 q19 1225962 564 564 q20 346 340 229 229 q21 3947320824152415 q22 10461022976 976 Total cold run time: 115111 ms Total hot run time: 34063 ms - Round 2, with runtime_filter_mode=off - q1 5141515650535053 q2 236 328 231 231 q3 2145265722952295 q4 1374178715141514 q5 4607444543804380 q6 210 169 129 129 q7 1983186417511751 q8 2570251925242519 q9 7148717171767171 q10 2978318827192719 q11 566 508 491 491 q12 703 743 615 615 q13 3447384833493349 q14 279 295 278 278 q15 512 473 460 460 q16 448 503 455 455 q17 1155155314411441 q18 7586750974537453 q19 833 883 999 883 q20 1992201718951895 q21 4745431643334316 q22 10911033960 960 Total cold run time: 51749 ms Total hot run time: 50358 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
feiniaofeiafei commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2909718507 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2903171192 ClickBench: Total hot run time: 28.62 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit f5aa8b68cb7f3a9f3fc76054b11db45f35ea3029, data reload: false query1 0.040.040.03 query2 0.120.110.11 query3 0.250.200.20 query4 1.590.200.10 query5 0.420.430.42 query6 1.150.640.66 query7 0.030.010.01 query8 0.040.040.03 query9 0.570.520.52 query10 0.550.580.57 query11 0.150.110.11 query12 0.150.110.12 query13 0.610.610.60 query14 0.790.810.81 query15 0.870.860.85 query16 0.370.400.39 query17 1.081.071.03 query18 0.220.210.21 query19 1.941.771.81 query20 0.020.010.01 query21 15.40 0.910.53 query22 0.761.310.74 query23 14.75 1.420.62 query24 7.241.490.47 query25 0.520.390.07 query26 0.620.170.14 query27 0.060.050.04 query28 9.080.920.45 query29 12.52 3.963.30 query30 0.260.090.07 query31 2.820.630.39 query32 3.230.550.48 query33 2.993.073.05 query34 15.54 5.084.44 query35 4.494.604.50 query36 0.660.500.48 query37 0.090.060.06 query38 0.050.040.04 query39 0.030.030.02 query40 0.170.140.13 query41 0.080.030.03 query42 0.040.020.02 query43 0.040.040.03 Total cold run time: 102.4 s Total hot run time: 28.62 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2903165745 TPC-DS: Total hot run time: 186741 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit f5aa8b68cb7f3a9f3fc76054b11db45f35ea3029, data reload: false query1 996 477 507 477 query2 6559180617981798 query3 6751218 215 215 query4 26411 23311 23039 23039 query5 4328648 480 480 query6 318 216 208 208 query7 4631523 295 295 query8 301 259 239 239 query9 8613267626782676 query10 502 353 273 273 query11 15667 15033 14843 14843 query12 174 108 107 107 query13 1639529 405 405 query14 9221617161776171 query15 197 195 165 165 query16 7244627 486 486 query17 1178714 579 579 query18 1972391 294 294 query19 190 184 169 169 query20 120 116 116 116 query21 215 126 107 107 query22 4148433542924292 query23 33971 33114 33250 33114 query24 8636240124592401 query25 539 469 393 393 query26 1236271 151 151 query27 2753501 341 341 query28 4340213521142114 query29 793 561 444 444 query30 289 217 195 195 query31 951 834 755 755 query32 73 64 71 64 query33 554 358 308 308 query34 794 852 536 536 query35 790 820 726 726 query36 958 974 885 885 query37 115 103 82 82 query38 4136416140774077 query39 1489141214131412 query40 223 128 116 116 query41 64 62 59 59 query42 125 110 117 110 query43 508 508 457 457 query44 1321833 833 833 query45 182 176 176 176 query46 846 1023646 646 query47 1804185417401740 query48 387 435 314 314 query49 815 579 474 474 query50 692 701 403 403 query51 4051411340744074 query52 110 107 101 101 query53 222 249 188 188 query54 577 579 492 492 query55 85 83 88 83 query56 346 325 299 299 query57 1130115711081108 query58 272 268 261 261 query59 2507273825582558 query60 329 326 319 319 query61 131 129 128 128 query62 789 726 679 679 query63 231 187 190 187 query64 43911010676 676 query65 4303425842524252 query66 1131425 319 319 query67 15799 15454 15582 15454 query68 7876885 525 525 query69 461 321 284 284 query70 1202111310901090 query71 436 349 317 317 query72 5640472547544725 query73 673 631 361 361 query74 8993933190619061 query75 3623320727082708 query76 33911186779 779 query77 771 399 299 299 query78 993410146 93919391 query79 2025825 573 573 query80 604 517 448 448 query81 478 256 225 225 query82 430 137 99 99 query83 259 259 234 234 query84 251 111 88 88 query85 813 356 323 323 query86 340 323 298 298 query87 4467445443254325 query88 3903233323192319 query89 396 330 290 290 query90 1948221 212 212 query91 143 145 117 117 query92 70 64 57 57 query93 1646936 576 576 query94 695 410 300 300 query95 381 298 294 294 query96 507 572 287 287 query97 2698281026862686 query98 225 211 209 209 query99 1431142413031303 Total cold run time: 273873 ms Total hot run time: 186741 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2903153425 TPC-H: Total hot run time: 34002 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit f5aa8b68cb7f3a9f3fc76054b11db45f35ea3029, data reload: false -- Round 1 -- q1 26004 515250415041 q2 2065287 190 190 q3 10559 1268685 685 q4 10230 989 546 546 q5 7859240423762376 q6 184 160 132 132 q7 901 733 611 611 q8 9314124211171117 q9 6858513551385135 q10 6826232119161916 q11 488 287 281 281 q12 348 351 213 213 q13 17801 372030673067 q14 230 222 206 206 q15 553 501 497 497 q16 433 436 387 387 q17 616 877 376 376 q18 7751724071017101 q19 1499971 563 563 q20 330 343 230 230 q21 3836317323572357 q22 10541002975 975 Total cold run time: 115739 ms Total hot run time: 34002 ms - Round 2, with runtime_filter_mode=off - q1 5111510150805080 q2 245 332 247 247 q3 2196266823522352 q4 1358174713961396 q5 4508445843734373 q6 209 173 126 126 q7 1961189117141714 q8 2600252124562456 q9 7221713772167137 q10 3004322627332733 q11 598 515 493 493 q12 710 787 611 611 q13 3442391832693269 q14 270 291 280 280 q15 546 516 494 494 q16 427 496 472 472 q17 1160157013851385 q18 7626753974417441 q19 819 800 831 800 q20 2020206119171917 q21 4738427442584258 q22 10481013988 988 Total cold run time: 51817 ms Total hot run time: 50022 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
feiniaofeiafei commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2903132991 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2896485066 ClickBench: Total hot run time: 29.54 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 719d020e9d4cffb7612515d976a70d739eb39796, data reload: false query1 0.030.040.03 query2 0.120.110.12 query3 0.260.200.19 query4 1.590.190.19 query5 0.440.430.45 query6 1.450.670.67 query7 0.020.010.02 query8 0.040.030.04 query9 0.560.520.52 query10 0.570.580.57 query11 0.160.110.11 query12 0.150.110.11 query13 0.620.600.61 query14 0.800.810.81 query15 0.870.860.84 query16 0.390.370.38 query17 1.041.031.06 query18 0.230.220.22 query19 1.971.801.88 query20 0.010.020.01 query21 15.40 0.910.58 query22 0.761.270.73 query23 14.79 1.440.67 query24 6.851.901.08 query25 0.500.120.15 query26 0.700.160.13 query27 0.060.060.05 query28 9.600.890.48 query29 12.65 3.973.28 query30 0.250.100.07 query31 2.820.610.38 query32 3.230.560.47 query33 3.053.063.06 query34 15.73 5.184.53 query35 4.554.474.52 query36 0.650.500.48 query37 0.080.060.06 query38 0.050.040.03 query39 0.030.020.02 query40 0.170.140.13 query41 0.080.030.03 query42 0.040.020.02 query43 0.040.030.02 Total cold run time: 103.4 s Total hot run time: 29.54 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2896479714 TPC-DS: Total hot run time: 193676 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 719d020e9d4cffb7612515d976a70d739eb39796, data reload: false query1 1406110910491049 query2 6118188319091883 query3 10977 461946294619 query4 55039 24871 23351 23351 query5 5085501 450 450 query6 341 202 190 190 query7 4877504 302 302 query8 321 259 250 250 query9 5639265726622657 query10 444 339 282 282 query11 15074 14991 14829 14829 query12 158 117 104 104 query13 1051546 426 426 query14 10170 648164236423 query15 225 204 180 180 query16 7130705 537 537 query17 1112766 628 628 query18 1537417 332 332 query19 211 205 184 184 query20 131 129 123 123 query21 209 125 113 113 query22 4397447743084308 query23 34238 33559 33453 33453 query24 6623245524742455 query25 476 493 410 410 query26 708 285 153 153 query27 2233513 356 356 query28 3469218121842181 query29 581 581 453 453 query30 274 225 199 199 query31 897 841 778 778 query32 75 60 64 60 query33 467 368 324 324 query34 794 898 553 553 query35 781 823 763 763 query36 927 1016908 908 query37 117 103 76 76 query38 4225437641524152 query39 1540147214441444 query40 208 125 121 121 query41 54 52 51 51 query42 129 109 119 109 query43 505 536 505 505 query44 1377835 847 835 query45 185 173 164 164 query46 865 1036660 660 query47 1842184017571757 query48 416 440 339 339 query49 687 528 454 454 query50 683 700 428 428 query51 4248435742514251 query52 124 115 111 111 query53 237 266 191 191 query54 612 574 525 525 query55 93 90 82 82 query56 315 324 319 319 query57 1174121111481148 query58 264 264 265 264 query59 2787295528352835 query60 339 339 309 309 query61 129 124 121 121 query62 700 736 660 660 query63 224 187 190 187 query64 17161004681 681 query65 4322424242384238 query66 695 398 308 308 query67 15718 15767 15499 15499 query68 7448889 526 526 query69 550 316 271 271 query70 1172117511021102 query71 505 329 311 311 query72 5999480149164801 query73 1474703 355 355 query74 9342913190529052 query75 3784317827112711 query76 41661281796 796 query77 639 362 292 292 query78 10187 993092469246 query79 2879816 586 586 query80 763 507 436 436 query81 479 254 217 217 query82 439 124 98 98 query83 367 252 229 229 query84 295 98 84 84 query85 791 363 313 313 query86 389 288 303 288 query87 4405435043744350 query88 3543231923052305 query89 407 327 276 276 query90 1779222 206 206 query91 146 143 123 123 query92 75 57 61 57 query93 2360959 568 568 query94 657 416 298 298 query95 366 294 291 291 query96 505 578 288 288 query97 2788277526342634 query98 240 218 202 202 query99 1417139112741274 Total cold run time: 301081 ms Total hot run time: 193676 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2896466390 TPC-H: Total hot run time: 34163 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 719d020e9d4cffb7612515d976a70d739eb39796, data reload: false -- Round 1 -- q1 26131 515050575057 q2 2075275 193 193 q3 10516 1268725 725 q4 10237 982 539 539 q5 7644239423452345 q6 181 164 132 132 q7 938 766 631 631 q8 9319131211841184 q9 6729507450685068 q10 6867231719151915 q11 502 287 284 284 q12 357 368 218 218 q13 17785 376031193119 q14 233 223 211 211 q15 536 476 490 476 q16 436 434 383 383 q17 618 887 376 376 q18 7538714471547144 q19 2285968 556 556 q20 337 345 244 244 q21 4001320924032403 q22 10291011960 960 Total cold run time: 116294 ms Total hot run time: 34163 ms - Round 2, with runtime_filter_mode=off - q1 5251513551725135 q2 239 339 232 232 q3 2162270922832283 q4 1368176414031403 q5 4470445344314431 q6 222 168 128 128 q7 2028193917651765 q8 2591259325632563 q9 7203717669516951 q10 2981318027722772 q11 574 516 507 507 q12 694 763 639 639 q13 3573389033573357 q14 301 309 288 288 q15 520 485 464 464 q16 451 466 443 443 q17 1155158713681368 q18 7751763275587558 q19 858 826 915 826 q20 1981195318761876 q21 4881451944094409 q22 1141105710301030 Total cold run time: 52395 ms Total hot run time: 50428 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
feiniaofeiafei commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2896283981 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2891185522 ClickBench: Total hot run time: 28.84 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit a532b9b187a59537a2319fb9a19800e2f87e9f8c, data reload: false query1 0.040.030.03 query2 0.120.100.11 query3 0.250.190.19 query4 1.590.180.18 query5 0.470.450.44 query6 1.160.660.65 query7 0.020.010.02 query8 0.040.030.03 query9 0.600.520.50 query10 0.570.580.57 query11 0.160.110.11 query12 0.150.120.12 query13 0.620.600.60 query14 0.780.790.81 query15 0.880.870.85 query16 0.370.380.37 query17 1.041.031.02 query18 0.220.210.21 query19 1.981.841.80 query20 0.010.010.02 query21 15.42 0.910.54 query22 0.751.150.67 query23 15.00 1.360.62 query24 7.241.330.59 query25 0.540.300.10 query26 0.660.170.14 query27 0.050.050.05 query28 9.460.930.44 query29 12.55 3.993.36 query30 0.270.090.06 query31 2.820.590.38 query32 3.230.550.47 query33 3.043.093.05 query34 15.68 5.064.50 query35 4.514.574.49 query36 0.670.490.48 query37 0.090.060.07 query38 0.060.050.03 query39 0.030.020.02 query40 0.170.130.14 query41 0.080.030.03 query42 0.040.020.03 query43 0.030.030.03 Total cold run time: 103.46 s Total hot run time: 28.84 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2891128876 TPC-H: Total hot run time: 33839 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit a532b9b187a59537a2319fb9a19800e2f87e9f8c, data reload: false -- Round 1 -- q1 25975 504550125012 q2 2065286 192 192 q3 10376 1253707 707 q4 10216 1026495 495 q5 7483234723352335 q6 184 167 131 131 q7 913 742 632 632 q8 9318128111231123 q9 6901507950385038 q10 6853234018891889 q11 513 274 269 269 q12 346 350 217 217 q13 17773 364630863086 q14 228 224 210 210 q15 538 490 484 484 q16 436 436 374 374 q17 611 846 384 384 q18 7913739671287128 q19 1431975 563 563 q20 341 352 230 230 q21 4125314723692369 q22 10031007971 971 Total cold run time: 115542 ms Total hot run time: 33839 ms - Round 2, with runtime_filter_mode=off - q1 5140509551265095 q2 237 327 227 227 q3 2163263722972297 q4 1317180813611361 q5 4440440243554355 q6 218 165 130 130 q7 2011197017811781 q8 2576261825502550 q9 7208718069556955 q10 3022318027942794 q11 595 520 482 482 q12 664 790 609 609 q13 3556392032783278 q14 284 308 300 300 q15 530 480 473 473 q16 440 473 449 449 q17 1137157113981398 q18 7711760474387438 q19 838 875 984 875 q20 2012205018481848 q21 4731451144244424 q22 1103104210081008 Total cold run time: 51933 ms Total hot run time: 50127 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2891168774 TPC-DS: Total hot run time: 193292 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit a532b9b187a59537a2319fb9a19800e2f87e9f8c, data reload: false query1 1397108110801080 query2 6318188719201887 query3 10992 444243574357 query4 53229 25399 23139 23139 query5 5106589 468 468 query6 352 212 207 207 query7 4923508 315 315 query8 340 266 242 242 query9 5767265526622655 query10 446 342 271 271 query11 15032 14987 14862 14862 query12 164 114 103 103 query13 1066526 428 428 query14 10170 645363156315 query15 207 193 180 180 query16 7056636 488 488 query17 1072724 580 580 query18 1546407 324 324 query19 202 191 164 164 query20 135 128 121 121 query21 211 137 110 110 query22 4321450043244324 query23 34118 33614 33480 33480 query24 6551243924702439 query25 468 464 414 414 query26 686 278 157 157 query27 2263508 359 359 query28 3208218321482148 query29 600 572 436 436 query30 273 223 192 192 query31 848 863 800 800 query32 78 65 60 60 query33 446 364 317 317 query34 789 869 554 554 query35 832 805 740 740 query36 937 1063892 892 query37 109 103 78 78 query38 4277420141204120 query39 1521156114721472 query40 216 133 120 120 query41 61 57 61 57 query42 137 118 118 118 query43 514 543 496 496 query44 1344856 900 856 query45 184 185 170 170 query46 871 1041660 660 query47 1843186018551855 query48 402 433 332 332 query49 705 530 442 442 query50 689 698 414 414 query51 4178424642444244 query52 121 118 106 106 query53 238 270 190 190 query54 603 614 517 517 query55 95 92 87 87 query56 311 326 324 324 query57 1173119411651165 query58 275 268 278 268 query59 2729291128252825 query60 343 336 324 324 query61 131 137 123 123 query62 712 744 691 691 query63 229 192 209 192 query64 14791042734 734 query65 4288421542004200 query66 719 405 303 303 query67 15849 15439 15542 15439 query68 5663913 543 543 query69 558 313 279 279 query70 1198112911401129 query71 464 335 324 324 query72 5285481949564819 query73 1260655 366 366 query74 8935916489278927 query75 3225317027152715 query76 38551190750 750 query77 559 403 289 289 query78 10060 10047 93179317 query79 2384825 597 597 query80 643 519 439 439 query81 469 270 218 218 query82 288 128 92 92 query83 248 247 232 232 query84 298 97 86 86 query85 750 350 308 308 query86 354 314 296 296 query87 4327443243464346 query88 3293237323692369 query89 398 322 279 279 query90 1675205 204 204 query91 138 145 113 113 query92 77 57 56 56 query93 2499933 583 583 query94 691 419 321 321 query95 386 304 293 293 query96 515 587 293 293 query97 2716277126692669 query98 247 209 208 208 query99 1341139713021302 Total cold run time: 293294 ms Total hot run time: 193292 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
feiniaofeiafei commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2890999117 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
feiniaofeiafei commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2879134026 run complie -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
924060929 commented on code in PR #49096:
URL: https://github.com/apache/doris/pull/49096#discussion_r2086315730
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/implementation/AggregateStrategies.java:
##
@@ -2088,4 +2107,130 @@ private boolean couldConvertToMulti(LogicalAggregate aggregate)
}
return true;
}
+
+/**
+ * LogicalAggregate(groupByExpr=[a], outputExpr=[a,count(distinct b)])
+ * ->
+ * +--PhysicalHashAggregate(groupByExpr=[a], outputExpr=[a,
count(partial_count(m))]
+ * +--PhysicalDistribute(shuffleColumn=[a])
+ * +--PhysicalHashAggregate(groupByExpr=[a], outputExpr=[a,
partial_count(m)]
+ * +--PhysicalHashAggregate(groupByExpr=[a, saltExpr],
outputExpr=[a, multi_distinct_count(b) as m])
+ * +--PhysicalDistribute(shuffleColumn=[a, saltExpr])
+ * +--PhysicalProject(projects=[a, b, xxhash_32(b)%512 as
saltExpr])
+ * +--PhysicalHashAggregate(groupByExpr=[a, b], outputExpr=[a,
b])
+ * */
+private PhysicalHashAggregate
countDistinctSkewRewrite(LogicalAggregate logicalAgg,
+CascadesContext cascadesContext) {
+if (!logicalAgg.canSkewRewrite()) {
+return null;
+}
+
+// 1.local agg
+ImmutableList.Builder localAggGroupByBuilder =
ImmutableList.builderWithExpectedSize(
+logicalAgg.getGroupByExpressions().size() + 1);
+localAggGroupByBuilder.addAll(logicalAgg.getGroupByExpressions());
+Count count = (Count)
logicalAgg.getAggregateFunctions().iterator().next();
+if (!(count.child(0) instanceof Slot)) {
+return null;
+}
+localAggGroupByBuilder.add(count.child(0));
+List localAggGroupBy = localAggGroupByBuilder.build();
+List localAggOutput =
Utils.fastToImmutableList((List) localAggGroupBy);
+RequireProperties requireAny =
RequireProperties.of(PhysicalProperties.ANY);
+boolean maybeUsingStreamAgg =
maybeUsingStreamAgg(cascadesContext.getConnectContext(),
+localAggGroupBy);
+boolean couldBanned = false;
+AggregateParam localParam = new AggregateParam(AggPhase.LOCAL,
AggMode.INPUT_TO_BUFFER, couldBanned);
+PhysicalHashAggregate localAgg = new
PhysicalHashAggregate<>(localAggGroupBy, localAggOutput,
+Optional.empty(), localParam, maybeUsingStreamAgg,
Optional.empty(), null,
+requireAny, logicalAgg.child());
+// add shuffle expr in project
+ImmutableList.Builder projections =
ImmutableList.builderWithExpectedSize(
+localAgg.getOutputs().size() + 1);
+projections.addAll(localAgg.getOutputs());
+Alias modAlias = getShuffleExpr(count, cascadesContext);
+projections.add(modAlias);
+PhysicalProject physicalProject = new
PhysicalProject<>(projections.build(), null, localAgg);
+
+// 2.second phase agg: multi_distinct_count(b) group by a,h
+ImmutableList.Builder secondPhaseAggGroupByBuilder =
ImmutableList.builderWithExpectedSize(
+logicalAgg.getGroupByExpressions().size() + 1);
+
secondPhaseAggGroupByBuilder.addAll(logicalAgg.getGroupByExpressions());
+secondPhaseAggGroupByBuilder.add(modAlias.toSlot());
+List secondPhaseAggGroupBy =
secondPhaseAggGroupByBuilder.build();
+ImmutableList.Builder secondPhaseAggOutput =
ImmutableList.builderWithExpectedSize(
+secondPhaseAggGroupBy.size() + 1);
+secondPhaseAggOutput.addAll((List) secondPhaseAggGroupBy);
+Alias aliasTarget = new Alias(new TinyIntLiteral((byte) 0));
+for (NamedExpression ne : logicalAgg.getOutputExpressions()) {
+if (ne instanceof Alias) {
+if (((Alias) ne).child().equals(count)) {
+aliasTarget = (Alias) ne;
+}
+}
+}
+AggregateParam secondParam = new AggregateParam(AggPhase.GLOBAL,
AggMode.INPUT_TO_RESULT, couldBanned);
+AggregateFunction multiDistinct = count.convertToMultiDistinct();
+Alias multiDistinctAlias = new Alias(new
AggregateExpression(multiDistinct, secondParam));
+secondPhaseAggOutput.add(multiDistinctAlias);
+List shuffleIds = new ArrayList<>();
+for (Expression expr : secondPhaseAggGroupBy) {
+if (expr instanceof Slot) {
+shuffleIds.add(((Slot) expr).getExprId());
+}
+}
+RequireProperties secondRequireProperties = RequireProperties.of(
+PhysicalProperties.createHash(shuffleIds,
ShuffleType.REQUIRE));
+PhysicalHashAggregate secondPhaseAgg = new
PhysicalHashAggregate<>(
+secondPhaseAggGroupBy, secondPhaseAggOutput.build(),
+Optional.empty(), secondParam, false, Optional.empty(), null,
+secondRequireProperties, physicalProject);
+
+/
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2875050680 ClickBench: Total hot run time: 29.38 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 1bd1200f83df5c588dfca21ef9272dce6cb569ab, data reload: false query1 0.040.040.03 query2 0.130.100.12 query3 0.250.190.19 query4 1.590.200.20 query5 0.590.580.59 query6 1.200.710.71 query7 0.030.010.01 query8 0.040.040.04 query9 0.590.510.50 query10 0.560.580.56 query11 0.160.120.11 query12 0.150.120.12 query13 0.620.600.61 query14 0.780.800.80 query15 0.890.860.88 query16 0.370.400.38 query17 1.021.031.03 query18 0.220.200.21 query19 2.001.851.85 query20 0.020.010.01 query21 15.40 0.900.54 query22 0.761.160.84 query23 14.75 1.390.60 query24 7.102.470.76 query25 0.460.270.08 query26 0.570.160.13 query27 0.050.050.05 query28 8.890.820.44 query29 12.58 3.923.34 query30 0.260.100.06 query31 2.830.600.38 query32 3.240.540.46 query33 3.032.993.08 query34 15.86 5.084.51 query35 4.544.554.52 query36 0.660.490.48 query37 0.090.060.06 query38 0.050.030.04 query39 0.040.020.03 query40 0.190.140.13 query41 0.080.030.03 query42 0.030.020.03 query43 0.040.030.03 Total cold run time: 102.75 s Total hot run time: 29.38 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2875044416 TPC-DS: Total hot run time: 190818 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 1bd1200f83df5c588dfca21ef9272dce6cb569ab, data reload: false query1 1407108210531053 query2 6413187018181818 query3 11006 436044954360 query4 54875 25311 23200 23200 query5 5055565 464 464 query6 337 216 203 203 query7 4881499 302 302 query8 327 260 250 250 query9 5328267426502650 query10 439 316 275 275 query11 15089 14979 14837 14837 query12 167 115 106 106 query13 1021540 416 416 query14 10284 650463456345 query15 231 205 186 186 query16 7195700 493 493 query17 1105768 612 612 query18 1628421 323 323 query19 204 203 171 171 query20 123 129 121 121 query21 203 127 113 113 query22 4427435243364336 query23 34489 33472 33511 33472 query24 6770242124422421 query25 462 479 398 398 query26 727 272 159 159 query27 2414516 350 350 query28 3090215421742154 query29 579 574 441 441 query30 278 221 196 196 query31 873 868 777 777 query32 71 67 60 60 query33 453 358 320 320 query34 796 867 542 542 query35 809 844 769 769 query36 952 1005888 888 query37 117 111 80 80 query38 4396429242444244 query39 1515146814661466 query40 218 121 111 111 query41 57 58 57 57 query42 134 113 119 113 query43 525 521 475 475 query44 1345847 829 829 query45 183 180 177 177 query46 874 1042667 667 query47 1834187617881788 query48 406 449 343 343 query49 702 544 432 432 query50 694 716 408 408 query51 4311428042054205 query52 114 111 103 103 query53 235 271 193 193 query54 611 595 532 532 query55 87 91 84 84 query56 335 316 301 301 query57 1144121611641164 query58 278 264 266 264 query59 2775279727042704 query60 340 338 312 312 query61 133 128 127 127 query62 718 750 687 687 query63 243 192 202 192 query64 21361141692 692 query65 4333423842554238 query66 728 409 308 308 query67 15804 15754 15249 15249 query68 7310906 530 530 query69 551 307 272 272 query70 1233113611231123 query71 516 317 306 306 query72 5584242051132420 query73 1341760 355 355 query74 9037919788708870 query75 4060321827322732 query76 42391182757 757 query77 650 374 292 292 query78 10121 10062 93829382 query79 5642806 568 568 query80 680 514 431 431 query81 475 256 219 219 query82 635 129 98 98 query83 343 252 231 231 query84 290 106 91 91 query85 774 371 313 313 query86 352 309 262 262 query87 4515442644054405 query88 3675229022802280 query89 450 313 296 296 query90 1925208 209 208 query91 149 154 116 116 query92 75 63 57 57 query93 3337931 578 578 query94 670 414 299 299 query95 375 291 286 286 query96 506 574 290 290 query97 3168321431233123 query98 232 224 213 213 query99 1432140612891289 Total cold run time: 306175 ms Total hot run time: 190818 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
doris-robot commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2875030054 TPC-H: Total hot run time: 34014 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 1bd1200f83df5c588dfca21ef9272dce6cb569ab, data reload: false -- Round 1 -- q1 26569 510950625062 q2 2092285 206 206 q3 10354 1225696 696 q4 10226 992 521 521 q5 7534231523842315 q6 179 166 141 141 q7 922 733 626 626 q8 9310124811311131 q9 6821503051325030 q10 6874229719161916 q11 488 276 271 271 q12 341 352 209 209 q13 17770 374530703070 q14 227 233 223 223 q15 543 477 493 477 q16 413 431 369 369 q17 603 862 373 373 q18 7493716471617161 q19 1402949 567 567 q20 335 354 222 222 q21 3998264424682468 q22 10141005960 960 Total cold run time: 115508 ms Total hot run time: 34014 ms - Round 2, with runtime_filter_mode=off - q1 5107506950755069 q2 242 332 232 232 q3 2156263622752275 q4 1369178213511351 q5 4414439344064393 q6 223 176 136 136 q7 1996193217651765 q8 2582252825292528 q9 7189722269326932 q10 3097317127952795 q11 591 524 515 515 q12 698 763 613 613 q13 3506391632603260 q14 294 303 294 294 q15 526 487 472 472 q16 425 492 439 439 q17 1145156613891389 q18 7745760973507350 q19 814 847 850 847 q20 2059200818401840 q21 5035460647924606 q22 1063103210191019 Total cold run time: 52276 ms Total hot run time: 50120 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhancement](nereids) add transform rule SaltJoin, countDistinctSkewRewrite, and window skew rewrite [doris]
feiniaofeiafei commented on PR #49096: URL: https://github.com/apache/doris/pull/49096#issuecomment-2875008850 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
