Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]
doris-robot commented on PR #44690: URL: https://github.com/apache/doris/pull/44690#issuecomment-2575557933 ClickBench: Total hot run time: 30.91 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 80ed26cb37bcf73b5dbf16b729139492be258d10, data reload: false query1 0.040.030.03 query2 0.080.030.03 query3 0.240.060.07 query4 1.620.100.10 query5 0.430.430.41 query6 1.160.640.64 query7 0.020.020.02 query8 0.040.040.03 query9 0.610.540.54 query10 0.600.610.60 query11 0.140.100.10 query12 0.140.110.10 query13 0.590.570.58 query14 2.832.782.82 query15 0.910.840.83 query16 0.380.390.36 query17 1.071.041.04 query18 0.210.200.20 query19 1.922.011.84 query20 0.020.000.01 query21 15.37 0.910.54 query22 0.751.150.63 query23 15.00 1.410.60 query24 6.682.030.67 query25 0.470.110.16 query26 0.710.180.14 query27 0.050.060.05 query28 9.241.330.90 query29 12.56 3.993.38 query30 0.250.100.05 query31 2.820.610.37 query32 3.230.550.45 query33 3.063.183.17 query34 15.84 4.924.41 query35 4.384.424.37 query36 0.700.490.47 query37 0.090.070.06 query38 0.040.040.04 query39 0.030.020.02 query40 0.180.150.13 query41 0.080.020.02 query42 0.040.030.02 query43 0.030.040.03 Total cold run time: 104.65 s Total hot run time: 30.91 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]
doris-robot commented on PR #44690: URL: https://github.com/apache/doris/pull/44690#issuecomment-2575546115 TPC-DS: Total hot run time: 190753 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 80ed26cb37bcf73b5dbf16b729139492be258d10, data reload: false query1 983 379 375 375 query2 6531239024392390 query3 6708209 212 209 query4 27095 23812 23826 23812 query5 4304673 492 492 query6 301 199 187 187 query7 4638504 301 301 query8 304 252 242 242 query9 8604266026532653 query10 472 296 259 259 query11 15733 15327 15245 15245 query12 168 109 107 107 query13 1675533 398 398 query14 10625 694372206943 query15 248 192 190 190 query16 8019629 416 416 query17 1599728 559 559 query18 2007398 299 299 query19 284 179 156 156 query20 124 114 115 114 query21 204 118 101 101 query22 4417440643504350 query23 34434 33407 33285 33285 query24 7292241624492416 query25 503 450 371 371 query26 1193273 155 155 query27 2032493 334 334 query28 3896241124382411 query29 711 565 407 407 query30 228 187 153 153 query31 997 915 834 834 query32 74 67 65 65 query33 554 355 305 305 query34 817 843 507 507 query35 798 827 725 725 query36 10011019947 947 query37 128 122 80 80 query38 4156432741364136 query39 1530143714561437 query40 211 125 106 106 query41 46 47 44 44 query42 122 104 103 103 query43 527 527 519 519 query44 1338798 799 798 query45 187 169 173 169 query46 912 1055681 681 query47 1943197719111911 query48 398 416 313 313 query49 776 516 431 431 query50 698 752 417 417 query51 7043697569466946 query52 104 110 92 92 query53 233 262 187 187 query54 519 487 406 406 query55 80 76 83 76 query56 272 284 259 259 query57 1202120311661166 query58 245 231 240 231 query59 2989309628482848 query60 285 288 286 286 query61 114 119 108 108 query62 918 800 751 751 query63 238 202 202 202 query64 4222986 695 695 query65 3276318732183187 query66 1049419 318 318 query67 16054 15797 15417 15417 query68 8854715 500 500 query69 453 302 263 263 query70 1238119711391139 query71 454 298 282 282 query72 5980357138003571 query73 802 768 355 355 query74 9257960791169116 query75 4110315427122712 query76 34781185752 752 query77 768 383 284 284 query78 10093 10307 94349434 query79 2694807 594 594 query80 632 519 435 435 query81 469 274 229 229 query82 678 163 120 120 query83 161 175 159 159 query84 228 90 78 78 query85 791 357 309 309 query86 369 303 297 297 query87 4557459244124412 query88 3541222922312229 query89 411 328 291 291 query90 1933189 225 189 query91 141 133 107 107 query92 76 61 59 59 query93 1592995 571 571 query94 654 396 282 282 query95 358 271 249 249 query96 481 554 266 266 query97 2907299727822782 query98 240 212 204 204 query99 1651161514501450 Total cold run time: 281439 ms Total hot run time: 190753 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]
doris-robot commented on PR #44690: URL: https://github.com/apache/doris/pull/44690#issuecomment-2575543559 TeamCity be ut coverage result: Function Coverage: 38.85% (10129/26069) Line Coverage: 29.91% (85721/286586) Region Coverage: 29.00% (43726/150762) Branch Coverage: 25.55% (22322/87362) Coverage Report: http://coverage.selectdb-in.cc/coverage/80ed26cb37bcf73b5dbf16b729139492be258d10_80ed26cb37bcf73b5dbf16b729139492be258d10/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]
doris-robot commented on PR #44690: URL: https://github.com/apache/doris/pull/44690#issuecomment-2575519135 TPC-H: Total hot run time: 32621 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 80ed26cb37bcf73b5dbf16b729139492be258d10, data reload: false -- Round 1 -- q1 17808 659364076407 q2 2051292 165 165 q3 10564 1269722 722 q4 10217 936 450 450 q5 7610241621602160 q6 201 178 147 147 q7 891 728 618 618 q8 9294131210981098 q9 483644274427 q10 6758232018831883 q11 457 283 252 252 q12 348 369 224 224 q13 17786 363730333033 q14 242 246 223 223 q15 575 517 505 505 q16 620 613 589 589 q17 571 853 340 340 q18 7292643964346434 q19 1269955 575 575 q20 301 314 181 181 q21 2775213318901890 q22 358 330 298 298 Total cold run time: 102824 ms Total hot run time: 32621 ms - Round 2, with runtime_filter_mode=off - q1 6645654865296529 q2 235 333 235 235 q3 2191273623552355 q4 1482186114201420 q5 4601443846544438 q6 232 187 144 144 q7 2047202719061906 q8 2692287227102710 q9 7322739074947390 q10 3063324328212821 q11 573 525 500 500 q12 761 797 654 654 q13 3487386332573257 q14 293 314 310 310 q15 551 511 493 493 q16 657 686 677 677 q17 1238166813711371 q18 8039748372057205 q19 801 835 871 835 q20 1882197718691869 q21 5581493247254725 q22 604 628 575 575 Total cold run time: 54977 ms Total hot run time: 52419 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]
zhiqiang- commented on PR #44690: URL: https://github.com/apache/doris/pull/44690#issuecomment-2575363274 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]
zhiqiang- commented on PR #44690: URL: https://github.com/apache/doris/pull/44690#issuecomment-2575165169 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]
zhiqiang- commented on PR #44690: URL: https://github.com/apache/doris/pull/44690#issuecomment-2574471235 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]
doris-robot commented on PR #44690: URL: https://github.com/apache/doris/pull/44690#issuecomment-2559885474 TeamCity be ut coverage result: Function Coverage: 38.78% (10092/26025) Line Coverage: 29.77% (85126/285920) Region Coverage: 28.88% (43451/150431) Branch Coverage: 25.42% (22148/87118) Coverage Report: http://coverage.selectdb-in.cc/coverage/01a41c777acef54100899009ae1a290bfced0459_01a41c777acef54100899009ae1a290bfced0459/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]
doris-robot commented on PR #44690: URL: https://github.com/apache/doris/pull/44690#issuecomment-2559886558 TPC-DS: Total hot run time: 197062 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 01a41c777acef54100899009ae1a290bfced0459, data reload: false query1 13271010978 978 query2 6291242123002300 query3 10959 479747874787 query4 26772 23323 23310 23310 query5 5289511 485 485 query6 309 194 200 194 query7 4020300 308 300 query8 303 240 238 238 query9 8499273827242724 query10 521 251 261 251 query11 15686 15247 15177 15177 query12 167 101 101 101 query13 1562423 394 394 query14 10191 662369716623 query15 239 177 183 177 query16 7867458 507 458 query17 1526624 596 596 query18 2043345 339 339 query19 325 178 197 178 query20 131 125 125 125 query21 211 109 108 108 query22 4894482147444744 query23 34810 33663 33631 33631 query24 8652272026582658 query25 597 413 399 399 query26 722 155 159 155 query27 2618347 355 347 query28 5252250625052505 query29 832 459 447 447 query30 232 153 154 153 query31 1039838 828 828 query32 93 61 57 57 query33 704 295 317 295 query34 951 565 543 543 query35 916 750 746 746 query36 943 937 937 query37 134 76 73 73 query38 4288418042064180 query39 1512147314831473 query40 209 102 100 100 query41 54 45 46 45 query42 112 106 105 105 query43 505 514 483 483 query44 1200806 820 806 query45 186 168 171 168 query46 1163755 769 755 query47 2059199319801980 query48 472 316 327 316 query49 981 430 423 423 query50 914 419 429 419 query51 7433719872267198 query52 112 95 102 95 query53 261 190 198 190 query54 650 419 450 419 query55 83 83 86 83 query56 294 265 241 241 query57 1313118911921189 query58 254 231 221 221 query59 3266307730093009 query60 297 257 261 257 query61 107 106 104 104 query62 870 685 726 685 query63 227 193 194 193 query64 3763682 663 663 query65 3234319732053197 query66 780 293 297 293 query67 15972 15536 15473 15473 query68 5067561 622 561 query69 484 259 250 250 query70 1126113610771077 query71 491 272 269 269 query72 7071395939033903 query73 811 356 364 356 query74 9145880587768776 query75 3749265826922658 query76 3680115611791156 query77 565 301 295 295 query78 10438 950694999499 query79 1147626 634 626 query80 756 443 446 443 query81 515 233 221 221 query82 430 122 119 119 query83 201 150 150 150 query84 242 77 69 69 query85 1055310 306 306 query86 334 293 294 293 query87 4595441844694418 query88 2770227922562256 query89 421 289 287 287 query90 2020192 193 192 query91 146 107 107 107 query92 63 51 55 51 query93 1474594 588 588 query94 828 283 268 268 query95 366 332 255 255 query96 598 278 277 277 query97 2869274726982698 query98 218 194 201 194 query99 1536132013261320 Total cold run time: 285577 ms Total hot run time: 197062 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go
Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]
doris-robot commented on PR #44690: URL: https://github.com/apache/doris/pull/44690#issuecomment-2559866651 TPC-H: Total hot run time: 40342 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 01a41c777acef54100899009ae1a290bfced0459, data reload: false -- Round 1 -- q1 17650 806977007700 q2 2073178 187 178 q3 10518 113610871087 q4 10217 804 769 769 q5 7584286728052805 q6 238 149 143 143 q7 1039636 620 620 q8 9428198020091980 q9 6538601760546017 q10 6965232123062306 q11 453 261 257 257 q12 432 225 231 225 q13 17781 300029482948 q14 254 218 220 218 q15 552 513 501 501 q16 678 606 587 587 q17 989 581 575 575 q18 7541676067146714 q19 1347116511581158 q20 470 188 191 188 q21 3555309930573057 q22 361 328 309 309 Total cold run time: 106663 ms Total hot run time: 40342 ms - Round 2, with runtime_filter_mode=off - q1 7709767276347634 q2 332 226 232 226 q3 2924304630183018 q4 2098198919751975 q5 5842586358675863 q6 234 147 146 146 q7 2273181518361815 q8 3460356935693569 q9 8989908990619061 q10 3546355835543554 q11 606 527 500 500 q12 833 614 643 614 q13 12238 306030983060 q14 310 278 301 278 q15 541 509 504 504 q16 657 660 652 652 q17 1903172116821682 q18 8683804378987898 q19 1704161216161612 q20 2111189218941892 q21 6203596159205920 q22 649 616 572 572 Total cold run time: 73845 ms Total hot run time: 62045 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]
zhiqiang- commented on PR #44690: URL: https://github.com/apache/doris/pull/44690#issuecomment-2559697999 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]
zhiqiang- commented on PR #44690: URL: https://github.com/apache/doris/pull/44690#issuecomment-2559023226 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]
doris-robot commented on PR #44690: URL: https://github.com/apache/doris/pull/44690#issuecomment-2558911647 ClickBench: Total hot run time: 33.68 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 504645872e4216e94f69fc2d4dc418efefbf2a89, data reload: false query1 0.030.030.03 query2 0.110.040.04 query3 0.280.060.06 query4 1.590.080.08 query5 0.420.410.40 query6 1.180.660.65 query7 0.020.010.02 query8 0.050.040.04 query9 0.630.550.55 query10 0.610.630.60 query11 0.250.130.13 query12 0.250.130.13 query13 0.620.610.60 query14 2.682.712.80 query15 1.010.880.88 query16 0.380.380.39 query17 1.051.041.05 query18 0.160.160.16 query19 2.011.842.04 query20 0.020.040.03 query21 15.35 0.680.67 query22 3.886.902.22 query23 16.58 1.961.31 query24 5.470.310.28 query25 0.180.090.09 query26 0.570.180.18 query27 0.090.090.09 query28 11.12 1.091.06 query29 12.58 3.463.45 query30 0.270.050.05 query31 2.840.430.42 query32 3.240.510.50 query33 3.123.063.09 query34 16.21 4.384.38 query35 4.534.414.45 query36 0.680.500.50 query37 0.200.170.17 query38 0.160.150.16 query39 0.050.040.04 query40 0.200.140.15 query41 0.100.060.05 query42 0.060.050.05 query43 0.060.050.05 Total cold run time: 110.89 s Total hot run time: 33.68 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]
doris-robot commented on PR #44690: URL: https://github.com/apache/doris/pull/44690#issuecomment-2558907020 TPC-DS: Total hot run time: 199031 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 504645872e4216e94f69fc2d4dc418efefbf2a89, data reload: false query1 1316952 944 944 query2 6241249525672495 query3 10974 472847744728 query4 26483 23958 23885 23885 query5 4369525 507 507 query6 301 197 197 197 query7 3995300 307 300 query8 317 247 223 223 query9 8433266726632663 query10 475 266 261 261 query11 16024 15419 15255 15255 query12 165 107 121 107 query13 1564418 396 396 query14 9833748373377337 query15 240 181 186 181 query16 7723473 485 473 query17 1525656 625 625 query18 2026345 374 345 query19 208 173 177 173 query20 127 119 124 119 query21 206 107 107 107 query22 4881471945924592 query23 34735 33775 33728 33728 query24 9674279728712797 query25 619 400 418 400 query26 1331162 160 160 query27 2457347 357 347 query28 6292244824612448 query29 922 426 436 426 query30 242 157 157 157 query31 1024871 855 855 query32 97 58 64 58 query33 736 311 309 309 query34 1199540 538 538 query35 907 783 765 765 query36 1128963 982 963 query37 148 84 81 81 query38 4344423243774232 query39 1546145614601456 query40 205 107 104 104 query41 53 51 54 51 query42 123 107 112 107 query43 566 534 531 531 query44 1294823 824 823 query45 196 177 178 177 query46 1227780 807 780 query47 2043198620021986 query48 486 322 305 305 query49 1024409 411 409 query50 933 435 434 434 query51 7247714172107141 query52 111 94 95 94 query53 275 201 211 201 query54 772 426 439 426 query55 88 82 86 82 query56 272 260 254 254 query57 1340120212231202 query58 253 239 235 235 query59 3359315129202920 query60 270 254 247 247 query61 115 110 120 110 query62 887 711 698 698 query63 227 234 193 193 query64 4053685 664 664 query65 3341321332243213 query66 806 319 313 313 query67 16326 15618 15573 15573 query68 5740598 580 580 query69 514 264 270 264 query70 1158115710801080 query71 523 249 249 249 query72 6823380540773805 query73 907 353 358 353 query74 9729897388578857 query75 3776268427042684 query76 3801133914541339 query77 578 286 274 274 query78 10424 957795539553 query79 2005617 621 617 query80 720 445 457 445 query81 517 233 237 233 query82 679 124 121 121 query83 186 156 146 146 query84 236 69 67 67 query85 1354416 301 301 query86 389 299 290 290 query87 4574441545064415 query88 3444220721972197 query89 462 308 291 291 query90 2081194 192 192 query91 147 104 107 104 query92 67 55 61 55 query93 1528627 612 612 query94 850 301 299 299 query95 371 246 255 246 query96 648 271 263 263 query97 2889266826922668 query98 224 198 201 198 query99 1609132713421327 Total cold run time: 291672 ms Total hot run time: 199031 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go
Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]
doris-robot commented on PR #44690: URL: https://github.com/apache/doris/pull/44690#issuecomment-2558896782 TPC-H: Total hot run time: 40800 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 504645872e4216e94f69fc2d4dc418efefbf2a89, data reload: false -- Round 1 -- q1 17870 807078417841 q2 2055185 175 175 q3 10698 103111801031 q4 10418 781 751 751 q5 7601283928682839 q6 241 150 149 149 q7 1028629 612 612 q8 9355201520102010 q9 6518614762286147 q10 6961232923502329 q11 469 280 256 256 q12 434 239 224 224 q13 17774 295929802959 q14 242 216 212 212 q15 548 504 491 491 q16 677 592 601 592 q17 1021651 616 616 q18 7429684968376837 q19 1342118011531153 q20 480 181 189 181 q21 3684308831063088 q22 372 319 307 307 Total cold run time: 107217 ms Total hot run time: 40800 ms - Round 2, with runtime_filter_mode=off - q1 7748846977497749 q2 326 235 233 233 q3 3072310030303030 q4 2163203919591959 q5 5912592759025902 q6 233 150 152 150 q7 2306190518921892 q8 3561366336113611 q9 9014916691809166 q10 3573361935963596 q11 603 505 528 505 q12 815 657 660 657 q13 15392 309831603098 q14 314 301 304 301 q15 568 505 510 505 q16 721 676 636 636 q17 1921174616981698 q18 8782812377317731 q19 1843168116971681 q20 2128192619731926 q21 6035584258285828 q22 653 607 586 586 Total cold run time: 77683 ms Total hot run time: 62440 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]
zhiqiang- commented on PR #44690: URL: https://github.com/apache/doris/pull/44690#issuecomment-2558790029 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]
github-actions[bot] commented on code in PR #44690: URL: https://github.com/apache/doris/pull/44690#discussion_r1886310686 ## be/src/vec/exec/scan/scanner_context.cpp: ## @@ -541,4 +465,148 @@ void ScannerContext::update_peak_running_scanner(int num) { _local_state->_peak_running_scanner->add(num); } +int32_t ScannerContext::_get_margin(std::unique_lock& transfer_lock) { +// margin_1 is used to ensure each scan operator could have at least _min_scan_concurrency scan tasks. +int32_t margin_1 = _min_concurrency - (_tasks_queue.size() + _num_scheduled_scanners); + +// margin_2 is used to ensure the scan scheduler could have at least _min_scan_concurrency_of_scan_scheduler scan tasks. +int32_t margin_2 = +_min_concurrency_of_scan_scheduler - +(_scanner_scheduler->get_active_threads() + _scanner_scheduler->get_queue_size()); + +if (margin_1 <= 0 && margin_2 <= 0) { +return 0; +} + +int32_t margin = std::max(margin_1, margin_2); +margin = std::min(margin, _basic_margin); +// // We dont want any scan operator to occupy all the resources. +// if (margin_1 >= margin_2) { +// margin = margin_1; +// } else { +// // 到这里说明 _scan_scheduler 的资源使用没有饱和,我们可以让当前的 scan operator 多提交一些 scan task +// // 但是不希望 _scan_schedulers 被某一个 scan operator 占用完,所以我们限制 margin 的大小 +// // 默认参数下,一个 scan node 会有 C/2 个 scan operator,那么每个 scan operator +// // 提交 8 个 scan task 的话,一个 scan node 会有 (C/2) * 8 = 4C 个 task +// // 恰好是一个 scan scheduler 的最小并发数 +// margin = _basic_margin; +// // margin = 8; +// } + +VLOG_DEBUG << fmt::format( +"[{}|{}] schedule scan task, margin_1: {} = {} - ({} + {}), margin_2: {} = {} - " +"({} + {}), margin: {}", +print_id(_query_id), ctx_id, margin_1, _min_concurrency, _tasks_queue.size(), +_num_scheduled_scanners, margin_2, _min_concurrency_of_scan_scheduler, +_scanner_scheduler->get_active_threads(), _scanner_scheduler->get_queue_size(), margin); + +return margin; +} + +Status ScannerContext::schedule_scan_task(std::shared_ptr current_scan_task, + std::unique_lock& transfer_lock) { +std::list> tasks_to_submit; + +// TODO: REFINE ME +_scanner_scheduler->write_lock(); +Defer defer1([&]() { _scanner_scheduler->write_unlock(); }); + +int32_t margin = _get_margin(transfer_lock); + +// margin is less than zero. Means this scan operator could not submit any scan task for now. +if (margin <= 0) { +// Be careful with current scan task. +// We need to add it back to task queue to make sure it could be resubmitted. +if (current_scan_task && current_scan_task->cached_blocks.empty() && +!current_scan_task->is_eos()) { +// This usually happens when we should downgrade the concurrency. +_pending_scanners.push(current_scan_task->scanner); +VLOG_DEBUG << fmt::format( +"{} push back scanner to task queue, because diff <= 0, task_queue size " +"{}, _num_scheduled_scanners {}", +ctx_id, _tasks_queue.size(), _num_scheduled_scanners); +} +return Status::OK(); +} + +bool first_pull = true; + +while (margin-- > 0) { +std::shared_ptr task_to_run; +const int32_t current_concurrency = +_tasks_queue.size() + _num_scheduled_scanners + tasks_to_submit.size(); +VLOG_DEBUG << fmt::format("{} currenct concurrency: {} = {} + {} + {}", ctx_id, + current_concurrency, _tasks_queue.size(), _num_scheduled_scanners, + tasks_to_submit.size()); +if (first_pull) { +task_to_run = pull_next_scan_task(current_scan_task, current_concurrency); +if (task_to_run == nullptr) { +// In two situations we will get nullptr. +// 1. current_concurrency already reached _max_concurrency. +// 2. all scanners are finished. +if (current_scan_task && current_scan_task->cached_blocks.empty() && +!current_scan_task->is_eos()) { +// Current scan task is not eos, but we can not resubmit it. +// Add current_scan_task back to task queue, so that we have chance to resubmit it in the future. +_pending_scanners.push(current_scan_task->scanner); +} +} +first_pull = false; +} else { +task_to_run = pull_next_scan_task(nullptr, current_concurrency); +} + +if (task_to_run) { +tasks_to_submit.push_back(task_to_run); +} else { +break; +} +} + +if (tasks_to_submit.empty()) { +return Status::OK(); +} + +
Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]
github-actions[bot] commented on code in PR #44690: URL: https://github.com/apache/doris/pull/44690#discussion_r1877987008 ## be/src/util/interval_histogram.h: ## @@ -0,0 +1,75 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#pragma once + +#include +#include +#include +#include +#include + +namespace doris { + +template +class IntervalHistogramStat { +public: +IntervalHistogramStat(size_t N) : window(N) {} + +void add(T value) { +std::lock_guard lock(mutex); + +if (window.full()) { +window.pop_front(); +} +window.push_back(value); +} + +T mean() { +std::lock_guard lock(mutex); +if (window.empty()) return T(); Review Comment: warning: statement should be inside braces [readability-braces-around-statements] ```suggestion if (window.empty()) { return T(); } ``` ## be/src/util/interval_histogram.h: ## @@ -0,0 +1,75 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +#pragma once + +#include +#include Review Comment: warning: 'boost/circular_buffer.hpp' file not found [clang-diagnostic-error] ```cpp #include ^ ``` ## be/src/vec/exec/scan/scanner_context.h: ## @@ -163,6 +170,16 @@ class ScannerContext : public std::enable_shared_from_this, bool _should_reset_thread_name = true; +void decrease_scanner_scheduled() { +std::lock_guard l(_transfer_lock); +_num_scheduled_scanners--; +} + +int32_t num_scheduled_scanners() { Review Comment: warning: method 'num_scheduled_scanners' can be made const [readability-make-member-function-const] ```suggestion int32_t num_scheduled_scanners() const { ``` ## be/src/vec/exec/scan/scanner_context.cpp: ## @@ -541,4 +448,145 @@ void ScannerContext::update_peak_running_scanner(int num) { _local_state->_peak_running_scanner->add(num); } +int32_t ScannerContext::_get_margin(std::unique_lock& transfer_lock) { +// margin_1 is used to ensure each scan operator could have at least _min_scan_concurrency. +int32_t margin_1 = _min_concurrency - (_tasks_queue.size() + _num_scheduled_scanners); + +// margin_2 is used to ensure the scan scheduler could have at least _min_scan_concurrency_of_scan_scheduler scan tasks. +int32_t margin_2 = +_min_concurrency_of_scan_scheduler - +(_scanner_scheduler->get_active_threads() + _scanner_scheduler->get_queue_size()); + +int32_t margin = 0; +// We dont want any scan operator to occupy all the resources. +if (margin_1 >= margin_2) { +margin = margin_1; +} else { +// 到这里说明 _scan_scheduler 的资源使用没有饱和,我们可以让当前的 scan operator 多提交一些 scan task +// 但是不希望 _scan_schedulers 被某一个 scan operator 占用完,所以我们限制 margin 的大小 +// 默认参数下,一个 scan node 会有 C/2 个 scan operator,那么每个 scan operator +// 提交 8 个 scan task 的话,一个 scan node 会有 (C/2) * 8 = 4C 个 task +// 恰好是一个 scan scheduler 的最小并发数 +margin = _basic_margin; +// margin = 8; +} + +VLOG_DEBUG << fmt::format( +"[{}|{}] schedule scan task, margin_1: {} = {} - ({} + {}), margin_2: {} = {} - " +"({} + {}), margin: {}", +print_id(_query_id), ctx_id, margin_1, _min_concurrency, _tasks_queue.size(), +_num_scheduled_scanners, margin_2, _min_concurrency_o
Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]
zhiqiang- commented on PR #44690: URL: https://github.com/apache/doris/pull/44690#issuecomment-2522410986 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]
zhiqiang- commented on PR #44690: URL: https://github.com/apache/doris/pull/44690#issuecomment-2519494896 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]
zhiqiang- commented on PR #44690: URL: https://github.com/apache/doris/pull/44690#issuecomment-2516429793 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]
zhiqiang- commented on PR #44690: URL: https://github.com/apache/doris/pull/44690#issuecomment-2516444210 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]
zhiqiang- commented on PR #44690: URL: https://github.com/apache/doris/pull/44690#issuecomment-2514688927 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]
zhiqiang- commented on PR #44690: URL: https://github.com/apache/doris/pull/44690#issuecomment-2513347531 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]
zhiqiang- commented on PR #44690: URL: https://github.com/apache/doris/pull/44690#issuecomment-2511250561 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [WIP](scan) Adaptive scan concurrency [doris]
doris-robot commented on PR #44690: URL: https://github.com/apache/doris/pull/44690#issuecomment-2503817218 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org