Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]

2025-01-07 Thread via GitHub


doris-robot commented on PR #44690:
URL: https://github.com/apache/doris/pull/44690#issuecomment-2575557933

   
   
   ClickBench: Total hot run time: 30.91 s
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 80ed26cb37bcf73b5dbf16b729139492be258d10, 
data reload: false
   
   query1   0.040.030.03
   query2   0.080.030.03
   query3   0.240.060.07
   query4   1.620.100.10
   query5   0.430.430.41
   query6   1.160.640.64
   query7   0.020.020.02
   query8   0.040.040.03
   query9   0.610.540.54
   query10  0.600.610.60
   query11  0.140.100.10
   query12  0.140.110.10
   query13  0.590.570.58
   query14  2.832.782.82
   query15  0.910.840.83
   query16  0.380.390.36
   query17  1.071.041.04
   query18  0.210.200.20
   query19  1.922.011.84
   query20  0.020.000.01
   query21  15.37   0.910.54
   query22  0.751.150.63
   query23  15.00   1.410.60
   query24  6.682.030.67
   query25  0.470.110.16
   query26  0.710.180.14
   query27  0.050.060.05
   query28  9.241.330.90
   query29  12.56   3.993.38
   query30  0.250.100.05
   query31  2.820.610.37
   query32  3.230.550.45
   query33  3.063.183.17
   query34  15.84   4.924.41
   query35  4.384.424.37
   query36  0.700.490.47
   query37  0.090.070.06
   query38  0.040.040.04
   query39  0.030.020.02
   query40  0.180.150.13
   query41  0.080.020.02
   query42  0.040.030.02
   query43  0.030.040.03
   Total cold run time: 104.65 s
   Total hot run time: 30.91 s
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]

2025-01-07 Thread via GitHub


doris-robot commented on PR #44690:
URL: https://github.com/apache/doris/pull/44690#issuecomment-2575546115

   
   
   TPC-DS: Total hot run time: 190753 ms
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 80ed26cb37bcf73b5dbf16b729139492be258d10, 
data reload: false
   
   query1   983 379 375 375
   query2   6531239024392390
   query3   6708209 212 209
   query4   27095   23812   23826   23812
   query5   4304673 492 492
   query6   301 199 187 187
   query7   4638504 301 301
   query8   304 252 242 242
   query9   8604266026532653
   query10  472 296 259 259
   query11  15733   15327   15245   15245
   query12  168 109 107 107
   query13  1675533 398 398
   query14  10625   694372206943
   query15  248 192 190 190
   query16  8019629 416 416
   query17  1599728 559 559
   query18  2007398 299 299
   query19  284 179 156 156
   query20  124 114 115 114
   query21  204 118 101 101
   query22  4417440643504350
   query23  34434   33407   33285   33285
   query24  7292241624492416
   query25  503 450 371 371
   query26  1193273 155 155
   query27  2032493 334 334
   query28  3896241124382411
   query29  711 565 407 407
   query30  228 187 153 153
   query31  997 915 834 834
   query32  74  67  65  65
   query33  554 355 305 305
   query34  817 843 507 507
   query35  798 827 725 725
   query36  10011019947 947
   query37  128 122 80  80
   query38  4156432741364136
   query39  1530143714561437
   query40  211 125 106 106
   query41  46  47  44  44
   query42  122 104 103 103
   query43  527 527 519 519
   query44  1338798 799 798
   query45  187 169 173 169
   query46  912 1055681 681
   query47  1943197719111911
   query48  398 416 313 313
   query49  776 516 431 431
   query50  698 752 417 417
   query51  7043697569466946
   query52  104 110 92  92
   query53  233 262 187 187
   query54  519 487 406 406
   query55  80  76  83  76
   query56  272 284 259 259
   query57  1202120311661166
   query58  245 231 240 231
   query59  2989309628482848
   query60  285 288 286 286
   query61  114 119 108 108
   query62  918 800 751 751
   query63  238 202 202 202
   query64  4222986 695 695
   query65  3276318732183187
   query66  1049419 318 318
   query67  16054   15797   15417   15417
   query68  8854715 500 500
   query69  453 302 263 263
   query70  1238119711391139
   query71  454 298 282 282
   query72  5980357138003571
   query73  802 768 355 355
   query74  9257960791169116
   query75  4110315427122712
   query76  34781185752 752
   query77  768 383 284 284
   query78  10093   10307   94349434
   query79  2694807 594 594
   query80  632 519 435 435
   query81  469 274 229 229
   query82  678 163 120 120
   query83  161 175 159 159
   query84  228 90  78  78
   query85  791 357 309 309
   query86  369 303 297 297
   query87  4557459244124412
   query88  3541222922312229
   query89  411 328 291 291
   query90  1933189 225 189
   query91  141 133 107 107
   query92  76  61  59  59
   query93  1592995 571 571
   query94  654 396 282 282
   query95  358 271 249 249
   query96  481 554 266 266
   query97  2907299727822782
   query98  240 212 204 204
   query99  1651161514501450
   Total cold run time: 281439 ms
   Total hot run time: 190753 ms
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to

Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]

2025-01-07 Thread via GitHub


doris-robot commented on PR #44690:
URL: https://github.com/apache/doris/pull/44690#issuecomment-2575543559

   TeamCity be ut coverage result:
Function Coverage: 38.85% (10129/26069) 
Line Coverage: 29.91% (85721/286586)
Region Coverage: 29.00% (43726/150762)
Branch Coverage: 25.55% (22322/87362)
Coverage Report: 
http://coverage.selectdb-in.cc/coverage/80ed26cb37bcf73b5dbf16b729139492be258d10_80ed26cb37bcf73b5dbf16b729139492be258d10/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]

2025-01-07 Thread via GitHub


doris-robot commented on PR #44690:
URL: https://github.com/apache/doris/pull/44690#issuecomment-2575519135

   
   
   TPC-H: Total hot run time: 32621 ms
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 80ed26cb37bcf73b5dbf16b729139492be258d10, 
data reload: false
   
   -- Round 1 --
   q1   17808   659364076407
   q2   2051292 165 165
   q3   10564   1269722 722
   q4   10217   936 450 450
   q5   7610241621602160
   q6   201 178 147 147
   q7   891 728 618 618
   q8   9294131210981098
   q9   483644274427
   q10  6758232018831883
   q11  457 283 252 252
   q12  348 369 224 224
   q13  17786   363730333033
   q14  242 246 223 223
   q15  575 517 505 505
   q16  620 613 589 589
   q17  571 853 340 340
   q18  7292643964346434
   q19  1269955 575 575
   q20  301 314 181 181
   q21  2775213318901890
   q22  358 330 298 298
   Total cold run time: 102824 ms
   Total hot run time: 32621 ms
   
   - Round 2, with runtime_filter_mode=off -
   q1   6645654865296529
   q2   235 333 235 235
   q3   2191273623552355
   q4   1482186114201420
   q5   4601443846544438
   q6   232 187 144 144
   q7   2047202719061906
   q8   2692287227102710
   q9   7322739074947390
   q10  3063324328212821
   q11  573 525 500 500
   q12  761 797 654 654
   q13  3487386332573257
   q14  293 314 310 310
   q15  551 511 493 493
   q16  657 686 677 677
   q17  1238166813711371
   q18  8039748372057205
   q19  801 835 871 835
   q20  1882197718691869
   q21  5581493247254725
   q22  604 628 575 575
   Total cold run time: 54977 ms
   Total hot run time: 52419 ms
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]

2025-01-07 Thread via GitHub


zhiqiang- commented on PR #44690:
URL: https://github.com/apache/doris/pull/44690#issuecomment-2575363274

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]

2025-01-07 Thread via GitHub


zhiqiang- commented on PR #44690:
URL: https://github.com/apache/doris/pull/44690#issuecomment-2575165169

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]

2025-01-06 Thread via GitHub


zhiqiang- commented on PR #44690:
URL: https://github.com/apache/doris/pull/44690#issuecomment-2574471235

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]

2024-12-23 Thread via GitHub


doris-robot commented on PR #44690:
URL: https://github.com/apache/doris/pull/44690#issuecomment-2559885474

   TeamCity be ut coverage result:
Function Coverage: 38.78% (10092/26025) 
Line Coverage: 29.77% (85126/285920)
Region Coverage: 28.88% (43451/150431)
Branch Coverage: 25.42% (22148/87118)
Coverage Report: 
http://coverage.selectdb-in.cc/coverage/01a41c777acef54100899009ae1a290bfced0459_01a41c777acef54100899009ae1a290bfced0459/report/index.html


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]

2024-12-23 Thread via GitHub


doris-robot commented on PR #44690:
URL: https://github.com/apache/doris/pull/44690#issuecomment-2559886558

   
   
   TPC-DS: Total hot run time: 197062 ms
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 01a41c777acef54100899009ae1a290bfced0459, 
data reload: false
   
   query1   13271010978 978
   query2   6291242123002300
   query3   10959   479747874787
   query4   26772   23323   23310   23310
   query5   5289511 485 485
   query6   309 194 200 194
   query7   4020300 308 300
   query8   303 240 238 238
   query9   8499273827242724
   query10  521 251 261 251
   query11  15686   15247   15177   15177
   query12  167 101 101 101
   query13  1562423 394 394
   query14  10191   662369716623
   query15  239 177 183 177
   query16  7867458 507 458
   query17  1526624 596 596
   query18  2043345 339 339
   query19  325 178 197 178
   query20  131 125 125 125
   query21  211 109 108 108
   query22  4894482147444744
   query23  34810   33663   33631   33631
   query24  8652272026582658
   query25  597 413 399 399
   query26  722 155 159 155
   query27  2618347 355 347
   query28  5252250625052505
   query29  832 459 447 447
   query30  232 153 154 153
   query31  1039838 828 828
   query32  93  61  57  57
   query33  704 295 317 295
   query34  951 565 543 543
   query35  916 750 746 746
   query36  943 937 937
   query37  134 76  73  73
   query38  4288418042064180
   query39  1512147314831473
   query40  209 102 100 100
   query41  54  45  46  45
   query42  112 106 105 105
   query43  505 514 483 483
   query44  1200806 820 806
   query45  186 168 171 168
   query46  1163755 769 755
   query47  2059199319801980
   query48  472 316 327 316
   query49  981 430 423 423
   query50  914 419 429 419
   query51  7433719872267198
   query52  112 95  102 95
   query53  261 190 198 190
   query54  650 419 450 419
   query55  83  83  86  83
   query56  294 265 241 241
   query57  1313118911921189
   query58  254 231 221 221
   query59  3266307730093009
   query60  297 257 261 257
   query61  107 106 104 104
   query62  870 685 726 685
   query63  227 193 194 193
   query64  3763682 663 663
   query65  3234319732053197
   query66  780 293 297 293
   query67  15972   15536   15473   15473
   query68  5067561 622 561
   query69  484 259 250 250
   query70  1126113610771077
   query71  491 272 269 269
   query72  7071395939033903
   query73  811 356 364 356
   query74  9145880587768776
   query75  3749265826922658
   query76  3680115611791156
   query77  565 301 295 295
   query78  10438   950694999499
   query79  1147626 634 626
   query80  756 443 446 443
   query81  515 233 221 221
   query82  430 122 119 119
   query83  201 150 150 150
   query84  242 77  69  69
   query85  1055310 306 306
   query86  334 293 294 293
   query87  4595441844694418
   query88  2770227922562256
   query89  421 289 287 287
   query90  2020192 193 192
   query91  146 107 107 107
   query92  63  51  55  51
   query93  1474594 588 588
   query94  828 283 268 268
   query95  366 332 255 255
   query96  598 278 277 277
   query97  2869274726982698
   query98  218 194 201 194
   query99  1536132013261320
   Total cold run time: 285577 ms
   Total hot run time: 197062 ms
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go 

Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]

2024-12-23 Thread via GitHub


doris-robot commented on PR #44690:
URL: https://github.com/apache/doris/pull/44690#issuecomment-2559866651

   
   
   TPC-H: Total hot run time: 40342 ms
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 01a41c777acef54100899009ae1a290bfced0459, 
data reload: false
   
   -- Round 1 --
   q1   17650   806977007700
   q2   2073178 187 178
   q3   10518   113610871087
   q4   10217   804 769 769
   q5   7584286728052805
   q6   238 149 143 143
   q7   1039636 620 620
   q8   9428198020091980
   q9   6538601760546017
   q10  6965232123062306
   q11  453 261 257 257
   q12  432 225 231 225
   q13  17781   300029482948
   q14  254 218 220 218
   q15  552 513 501 501
   q16  678 606 587 587
   q17  989 581 575 575
   q18  7541676067146714
   q19  1347116511581158
   q20  470 188 191 188
   q21  3555309930573057
   q22  361 328 309 309
   Total cold run time: 106663 ms
   Total hot run time: 40342 ms
   
   - Round 2, with runtime_filter_mode=off -
   q1   7709767276347634
   q2   332 226 232 226
   q3   2924304630183018
   q4   2098198919751975
   q5   5842586358675863
   q6   234 147 146 146
   q7   2273181518361815
   q8   3460356935693569
   q9   8989908990619061
   q10  3546355835543554
   q11  606 527 500 500
   q12  833 614 643 614
   q13  12238   306030983060
   q14  310 278 301 278
   q15  541 509 504 504
   q16  657 660 652 652
   q17  1903172116821682
   q18  8683804378987898
   q19  1704161216161612
   q20  2111189218941892
   q21  6203596159205920
   q22  649 616 572 572
   Total cold run time: 73845 ms
   Total hot run time: 62045 ms
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]

2024-12-23 Thread via GitHub


zhiqiang- commented on PR #44690:
URL: https://github.com/apache/doris/pull/44690#issuecomment-2559697999

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]

2024-12-22 Thread via GitHub


zhiqiang- commented on PR #44690:
URL: https://github.com/apache/doris/pull/44690#issuecomment-2559023226

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]

2024-12-22 Thread via GitHub


doris-robot commented on PR #44690:
URL: https://github.com/apache/doris/pull/44690#issuecomment-2558911647

   
   
   ClickBench: Total hot run time: 33.68 s
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 504645872e4216e94f69fc2d4dc418efefbf2a89, 
data reload: false
   
   query1   0.030.030.03
   query2   0.110.040.04
   query3   0.280.060.06
   query4   1.590.080.08
   query5   0.420.410.40
   query6   1.180.660.65
   query7   0.020.010.02
   query8   0.050.040.04
   query9   0.630.550.55
   query10  0.610.630.60
   query11  0.250.130.13
   query12  0.250.130.13
   query13  0.620.610.60
   query14  2.682.712.80
   query15  1.010.880.88
   query16  0.380.380.39
   query17  1.051.041.05
   query18  0.160.160.16
   query19  2.011.842.04
   query20  0.020.040.03
   query21  15.35   0.680.67
   query22  3.886.902.22
   query23  16.58   1.961.31
   query24  5.470.310.28
   query25  0.180.090.09
   query26  0.570.180.18
   query27  0.090.090.09
   query28  11.12   1.091.06
   query29  12.58   3.463.45
   query30  0.270.050.05
   query31  2.840.430.42
   query32  3.240.510.50
   query33  3.123.063.09
   query34  16.21   4.384.38
   query35  4.534.414.45
   query36  0.680.500.50
   query37  0.200.170.17
   query38  0.160.150.16
   query39  0.050.040.04
   query40  0.200.140.15
   query41  0.100.060.05
   query42  0.060.050.05
   query43  0.060.050.05
   Total cold run time: 110.89 s
   Total hot run time: 33.68 s
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]

2024-12-22 Thread via GitHub


doris-robot commented on PR #44690:
URL: https://github.com/apache/doris/pull/44690#issuecomment-2558907020

   
   
   TPC-DS: Total hot run time: 199031 ms
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 504645872e4216e94f69fc2d4dc418efefbf2a89, 
data reload: false
   
   query1   1316952 944 944
   query2   6241249525672495
   query3   10974   472847744728
   query4   26483   23958   23885   23885
   query5   4369525 507 507
   query6   301 197 197 197
   query7   3995300 307 300
   query8   317 247 223 223
   query9   8433266726632663
   query10  475 266 261 261
   query11  16024   15419   15255   15255
   query12  165 107 121 107
   query13  1564418 396 396
   query14  9833748373377337
   query15  240 181 186 181
   query16  7723473 485 473
   query17  1525656 625 625
   query18  2026345 374 345
   query19  208 173 177 173
   query20  127 119 124 119
   query21  206 107 107 107
   query22  4881471945924592
   query23  34735   33775   33728   33728
   query24  9674279728712797
   query25  619 400 418 400
   query26  1331162 160 160
   query27  2457347 357 347
   query28  6292244824612448
   query29  922 426 436 426
   query30  242 157 157 157
   query31  1024871 855 855
   query32  97  58  64  58
   query33  736 311 309 309
   query34  1199540 538 538
   query35  907 783 765 765
   query36  1128963 982 963
   query37  148 84  81  81
   query38  4344423243774232
   query39  1546145614601456
   query40  205 107 104 104
   query41  53  51  54  51
   query42  123 107 112 107
   query43  566 534 531 531
   query44  1294823 824 823
   query45  196 177 178 177
   query46  1227780 807 780
   query47  2043198620021986
   query48  486 322 305 305
   query49  1024409 411 409
   query50  933 435 434 434
   query51  7247714172107141
   query52  111 94  95  94
   query53  275 201 211 201
   query54  772 426 439 426
   query55  88  82  86  82
   query56  272 260 254 254
   query57  1340120212231202
   query58  253 239 235 235
   query59  3359315129202920
   query60  270 254 247 247
   query61  115 110 120 110
   query62  887 711 698 698
   query63  227 234 193 193
   query64  4053685 664 664
   query65  3341321332243213
   query66  806 319 313 313
   query67  16326   15618   15573   15573
   query68  5740598 580 580
   query69  514 264 270 264
   query70  1158115710801080
   query71  523 249 249 249
   query72  6823380540773805
   query73  907 353 358 353
   query74  9729897388578857
   query75  3776268427042684
   query76  3801133914541339
   query77  578 286 274 274
   query78  10424   957795539553
   query79  2005617 621 617
   query80  720 445 457 445
   query81  517 233 237 233
   query82  679 124 121 121
   query83  186 156 146 146
   query84  236 69  67  67
   query85  1354416 301 301
   query86  389 299 290 290
   query87  4574441545064415
   query88  3444220721972197
   query89  462 308 291 291
   query90  2081194 192 192
   query91  147 104 107 104
   query92  67  55  61  55
   query93  1528627 612 612
   query94  850 301 299 299
   query95  371 246 255 246
   query96  648 271 263 263
   query97  2889266826922668
   query98  224 198 201 198
   query99  1609132713421327
   Total cold run time: 291672 ms
   Total hot run time: 199031 ms
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go 

Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]

2024-12-22 Thread via GitHub


doris-robot commented on PR #44690:
URL: https://github.com/apache/doris/pull/44690#issuecomment-2558896782

   
   
   TPC-H: Total hot run time: 40800 ms
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 504645872e4216e94f69fc2d4dc418efefbf2a89, 
data reload: false
   
   -- Round 1 --
   q1   17870   807078417841
   q2   2055185 175 175
   q3   10698   103111801031
   q4   10418   781 751 751
   q5   7601283928682839
   q6   241 150 149 149
   q7   1028629 612 612
   q8   9355201520102010
   q9   6518614762286147
   q10  6961232923502329
   q11  469 280 256 256
   q12  434 239 224 224
   q13  17774   295929802959
   q14  242 216 212 212
   q15  548 504 491 491
   q16  677 592 601 592
   q17  1021651 616 616
   q18  7429684968376837
   q19  1342118011531153
   q20  480 181 189 181
   q21  3684308831063088
   q22  372 319 307 307
   Total cold run time: 107217 ms
   Total hot run time: 40800 ms
   
   - Round 2, with runtime_filter_mode=off -
   q1   7748846977497749
   q2   326 235 233 233
   q3   3072310030303030
   q4   2163203919591959
   q5   5912592759025902
   q6   233 150 152 150
   q7   2306190518921892
   q8   3561366336113611
   q9   9014916691809166
   q10  3573361935963596
   q11  603 505 528 505
   q12  815 657 660 657
   q13  15392   309831603098
   q14  314 301 304 301
   q15  568 505 510 505
   q16  721 676 636 636
   q17  1921174616981698
   q18  8782812377317731
   q19  1843168116971681
   q20  2128192619731926
   q21  6035584258285828
   q22  653 607 586 586
   Total cold run time: 77683 ms
   Total hot run time: 62440 ms
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]

2024-12-22 Thread via GitHub


zhiqiang- commented on PR #44690:
URL: https://github.com/apache/doris/pull/44690#issuecomment-2558790029

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]

2024-12-15 Thread via GitHub


github-actions[bot] commented on code in PR #44690:
URL: https://github.com/apache/doris/pull/44690#discussion_r1886310686


##
be/src/vec/exec/scan/scanner_context.cpp:
##
@@ -541,4 +465,148 @@ void ScannerContext::update_peak_running_scanner(int num) 
{
 _local_state->_peak_running_scanner->add(num);
 }
 
+int32_t ScannerContext::_get_margin(std::unique_lock& 
transfer_lock) {
+// margin_1 is used to ensure each scan operator could have at least 
_min_scan_concurrency scan tasks.
+int32_t margin_1 = _min_concurrency - (_tasks_queue.size() + 
_num_scheduled_scanners);
+
+// margin_2 is used to ensure the scan scheduler could have at least 
_min_scan_concurrency_of_scan_scheduler scan tasks.
+int32_t margin_2 =
+_min_concurrency_of_scan_scheduler -
+(_scanner_scheduler->get_active_threads() + 
_scanner_scheduler->get_queue_size());
+
+if (margin_1 <= 0 && margin_2 <= 0) {
+return 0;
+}
+
+int32_t margin = std::max(margin_1, margin_2);
+margin = std::min(margin, _basic_margin);
+// // We dont want any scan operator to occupy all the resources.
+// if (margin_1 >= margin_2) {
+// margin = margin_1;
+// } else {
+// // 到这里说明 _scan_scheduler 的资源使用没有饱和,我们可以让当前的 scan operator 多提交一些 
scan task
+// // 但是不希望 _scan_schedulers 被某一个 scan operator 占用完,所以我们限制 margin 的大小
+// // 默认参数下,一个 scan node 会有 C/2 个 scan operator,那么每个 scan operator
+// // 提交 8 个 scan task 的话,一个 scan node 会有 (C/2) * 8 = 4C 个 task
+// // 恰好是一个 scan scheduler 的最小并发数
+// margin = _basic_margin;
+// // margin = 8;
+// }
+
+VLOG_DEBUG << fmt::format(
+"[{}|{}] schedule scan task, margin_1: {} = {} - ({} + {}), 
margin_2: {} = {} - "
+"({} + {}), margin: {}",
+print_id(_query_id), ctx_id, margin_1, _min_concurrency, 
_tasks_queue.size(),
+_num_scheduled_scanners, margin_2, 
_min_concurrency_of_scan_scheduler,
+_scanner_scheduler->get_active_threads(), 
_scanner_scheduler->get_queue_size(), margin);
+
+return margin;
+}
+
+Status ScannerContext::schedule_scan_task(std::shared_ptr 
current_scan_task,
+  std::unique_lock& 
transfer_lock) {
+std::list> tasks_to_submit;
+
+// TODO: REFINE ME
+_scanner_scheduler->write_lock();
+Defer defer1([&]() { _scanner_scheduler->write_unlock(); });
+
+int32_t margin = _get_margin(transfer_lock);
+
+// margin is less than zero. Means this scan operator could not submit any 
scan task for now.
+if (margin <= 0) {
+// Be careful with current scan task.
+// We need to add it back to task queue to make sure it could be 
resubmitted.
+if (current_scan_task && current_scan_task->cached_blocks.empty() &&
+!current_scan_task->is_eos()) {
+// This usually happens when we should downgrade the concurrency.
+_pending_scanners.push(current_scan_task->scanner);
+VLOG_DEBUG << fmt::format(
+"{} push back scanner to task queue, because diff <= 0, 
task_queue size "
+"{}, _num_scheduled_scanners {}",
+ctx_id, _tasks_queue.size(), _num_scheduled_scanners);
+}
+return Status::OK();
+}
+
+bool first_pull = true;
+
+while (margin-- > 0) {
+std::shared_ptr task_to_run;
+const int32_t current_concurrency =
+_tasks_queue.size() + _num_scheduled_scanners + 
tasks_to_submit.size();
+VLOG_DEBUG << fmt::format("{} currenct concurrency: {} = {} + {} + 
{}", ctx_id,
+  current_concurrency, _tasks_queue.size(), 
_num_scheduled_scanners,
+  tasks_to_submit.size());
+if (first_pull) {
+task_to_run = pull_next_scan_task(current_scan_task, 
current_concurrency);
+if (task_to_run == nullptr) {
+// In two situations we will get nullptr.
+// 1. current_concurrency already reached _max_concurrency.
+// 2. all scanners are finished.
+if (current_scan_task && 
current_scan_task->cached_blocks.empty() &&
+!current_scan_task->is_eos()) {
+// Current scan task is not eos, but we can not resubmit 
it.
+// Add current_scan_task back to task queue, so that we 
have chance to resubmit it in the future.
+_pending_scanners.push(current_scan_task->scanner);
+}
+}
+first_pull = false;
+} else {
+task_to_run = pull_next_scan_task(nullptr, current_concurrency);
+}
+
+if (task_to_run) {
+tasks_to_submit.push_back(task_to_run);
+} else {
+break;
+}
+}
+
+if (tasks_to_submit.empty()) {
+return Status::OK();
+}
+
+

Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]

2024-12-10 Thread via GitHub


github-actions[bot] commented on code in PR #44690:
URL: https://github.com/apache/doris/pull/44690#discussion_r1877987008


##
be/src/util/interval_histogram.h:
##
@@ -0,0 +1,75 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include 
+#include 
+#include 
+#include 
+#include 
+
+namespace doris {
+
+template 
+class IntervalHistogramStat {
+public:
+IntervalHistogramStat(size_t N) : window(N) {}
+
+void add(T value) {
+std::lock_guard lock(mutex);
+
+if (window.full()) {
+window.pop_front();
+}
+window.push_back(value);
+}
+
+T mean() {
+std::lock_guard lock(mutex);
+if (window.empty()) return T();

Review Comment:
   warning: statement should be inside braces 
[readability-braces-around-statements]
   
   ```suggestion
   if (window.empty()) { return T();
   }
   ```
   



##
be/src/util/interval_histogram.h:
##
@@ -0,0 +1,75 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include 
+#include 

Review Comment:
   warning: 'boost/circular_buffer.hpp' file not found [clang-diagnostic-error]
   ```cpp
   #include 
^
   ```
   



##
be/src/vec/exec/scan/scanner_context.h:
##
@@ -163,6 +170,16 @@ class ScannerContext : public 
std::enable_shared_from_this,
 
 bool _should_reset_thread_name = true;
 
+void decrease_scanner_scheduled() {
+std::lock_guard l(_transfer_lock);
+_num_scheduled_scanners--;
+}
+
+int32_t num_scheduled_scanners() {

Review Comment:
   warning: method 'num_scheduled_scanners' can be made const 
[readability-make-member-function-const]
   
   ```suggestion
   int32_t num_scheduled_scanners() const {
   ```
   



##
be/src/vec/exec/scan/scanner_context.cpp:
##
@@ -541,4 +448,145 @@ void ScannerContext::update_peak_running_scanner(int num) 
{
 _local_state->_peak_running_scanner->add(num);
 }
 
+int32_t ScannerContext::_get_margin(std::unique_lock& 
transfer_lock) {
+// margin_1 is used to ensure each scan operator could have at least 
_min_scan_concurrency.
+int32_t margin_1 = _min_concurrency - (_tasks_queue.size() + 
_num_scheduled_scanners);
+
+// margin_2 is used to ensure the scan scheduler could have at least 
_min_scan_concurrency_of_scan_scheduler scan tasks.
+int32_t margin_2 =
+_min_concurrency_of_scan_scheduler -
+(_scanner_scheduler->get_active_threads() + 
_scanner_scheduler->get_queue_size());
+
+int32_t margin = 0;
+// We dont want any scan operator to occupy all the resources.
+if (margin_1 >= margin_2) {
+margin = margin_1;
+} else {
+// 到这里说明 _scan_scheduler 的资源使用没有饱和,我们可以让当前的 scan operator 多提交一些 scan 
task
+// 但是不希望 _scan_schedulers 被某一个 scan operator 占用完,所以我们限制 margin 的大小
+// 默认参数下,一个 scan node 会有 C/2 个 scan operator,那么每个 scan operator
+// 提交 8 个 scan task 的话,一个 scan node 会有 (C/2) * 8 = 4C 个 task
+// 恰好是一个 scan scheduler 的最小并发数
+margin = _basic_margin;
+// margin = 8;
+}
+
+VLOG_DEBUG << fmt::format(
+"[{}|{}] schedule scan task, margin_1: {} = {} - ({} + {}), 
margin_2: {} = {} - "
+"({} + {}), margin: {}",
+print_id(_query_id), ctx_id, margin_1, _min_concurrency, 
_tasks_queue.size(),
+_num_scheduled_scanners, margin_2, 
_min_concurrency_o

Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]

2024-12-05 Thread via GitHub


zhiqiang- commented on PR #44690:
URL: https://github.com/apache/doris/pull/44690#issuecomment-2522410986

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]

2024-12-04 Thread via GitHub


zhiqiang- commented on PR #44690:
URL: https://github.com/apache/doris/pull/44690#issuecomment-2519494896

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]

2024-12-03 Thread via GitHub


zhiqiang- commented on PR #44690:
URL: https://github.com/apache/doris/pull/44690#issuecomment-2516429793

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]

2024-12-03 Thread via GitHub


zhiqiang- commented on PR #44690:
URL: https://github.com/apache/doris/pull/44690#issuecomment-2516444210

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]

2024-12-03 Thread via GitHub


zhiqiang- commented on PR #44690:
URL: https://github.com/apache/doris/pull/44690#issuecomment-2514688927

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]

2024-12-02 Thread via GitHub


zhiqiang- commented on PR #44690:
URL: https://github.com/apache/doris/pull/44690#issuecomment-2513347531

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]

2024-12-02 Thread via GitHub


zhiqiang- commented on PR #44690:
URL: https://github.com/apache/doris/pull/44690#issuecomment-2511250561

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org



Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]

2024-11-27 Thread via GitHub


doris-robot commented on PR #44690:
URL: https://github.com/apache/doris/pull/44690#issuecomment-2503817218

   
   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your 
PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
   
   Please clearly describe your PR:
   1. What problem was fixed (it's best to include specific error reporting 
information). How it was fixed.
   2. Which behaviors were modified. What was the previous behavior, what is it 
now, why was it modified, and what possible impacts might there be.
   3. What features were added. Why was this function added?
   4. Which code was refactored and why was this part of the code refactored?
   5. Which functions were optimized and what is the difference before and 
after the optimization?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org