Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
github-actions[bot] commented on PR #54243: URL: https://github.com/apache/doris/pull/54243#issuecomment-3173410641 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
yiguolei merged PR #54243: URL: https://github.com/apache/doris/pull/54243 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
github-actions[bot] commented on PR #54243: URL: https://github.com/apache/doris/pull/54243#issuecomment-3173410547 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
hello-stephen commented on PR #54243: URL: https://github.com/apache/doris/pull/54243#issuecomment-3173143130 # FE UT Coverage Report Increment line coverage ` 60.18% (133/221)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/54243_e1d519b7520a0dddb83e7c0d9e59b8d0304eb92a/fe_increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/54243_e1d519b7520a0dddb83e7c0d9e59b8d0304eb92a/fe_report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
hello-stephen commented on PR #54243: URL: https://github.com/apache/doris/pull/54243#issuecomment-3172548201 # BE Regression && UT Coverage Report Increment line coverage `77.97% (92/118)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/54243_e1d519b7520a0dddb83e7c0d9e59b8d0304eb92a_merge/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/54243_e1d519b7520a0dddb83e7c0d9e59b8d0304eb92a_merge/report/index.html) | Category | Coverage | |---|| | Function Coverage | 81.38% (22494/27642) | | Line Coverage | 74.06% (232970/314576) | | Region Coverage | 61.20% (193165/315604) | | Branch Coverage | 65.46% (83657/127807) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
hello-stephen commented on PR #54243: URL: https://github.com/apache/doris/pull/54243#issuecomment-3172536949 # BE Regression && UT Coverage Report Increment line coverage `77.97% (92/118)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/54243_e1d519b7520a0dddb83e7c0d9e59b8d0304eb92a_merge/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/54243_e1d519b7520a0dddb83e7c0d9e59b8d0304eb92a_merge/report/index.html) | Category | Coverage | |---|| | Function Coverage | 81.37% (22493/27642) | | Line Coverage | 74.05% (232953/314576) | | Region Coverage | 61.17% (193070/315604) | | Branch Coverage | 65.45% (83654/127807) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
hello-stephen commented on PR #54243: URL: https://github.com/apache/doris/pull/54243#issuecomment-3172493981 # BE Regression && UT Coverage Report Increment line coverage `77.97% (92/118)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/54243_e1d519b7520a0dddb83e7c0d9e59b8d0304eb92a_merge/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/54243_e1d519b7520a0dddb83e7c0d9e59b8d0304eb92a_merge/report/index.html) | Category | Coverage | |---|| | Function Coverage | 81.37% (22493/27642) | | Line Coverage | 74.05% (232950/314576) | | Region Coverage | 61.21% (193193/315604) | | Branch Coverage | 65.46% (83658/127807) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
hello-stephen commented on PR #54243: URL: https://github.com/apache/doris/pull/54243#issuecomment-3172490539 # BE UT Coverage Report Increment line coverage `49.15% (58/118)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/e1d519b7520a0dddb83e7c0d9e59b8d0304eb92a_e1d519b7520a0dddb83e7c0d9e59b8d0304eb92a/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/e1d519b7520a0dddb83e7c0d9e59b8d0304eb92a_e1d519b7520a0dddb83e7c0d9e59b8d0304eb92a/report/index.html) | Category | Coverage | |---|| | Function Coverage | 58.88% (16583/28165) | | Line Coverage | 47.77% (150279/314561) | | Region Coverage | 36.64% (112613/307333) | | Branch Coverage | 39.62% (50024/126257) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
doris-robot commented on PR #54243: URL: https://github.com/apache/doris/pull/54243#issuecomment-3172487215 ClickBench: Total hot run time: 34.99 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit e1d519b7520a0dddb83e7c0d9e59b8d0304eb92a, data reload: false query1 0.040.030.03 query2 0.100.050.06 query3 0.300.070.07 query4 1.590.090.08 query5 0.420.410.42 query6 1.170.670.67 query7 0.050.010.02 query8 0.070.060.06 query9 0.600.480.48 query10 0.540.530.55 query11 0.260.130.13 query12 0.250.140.13 query13 0.680.670.69 query14 0.921.121.09 query15 1.070.920.93 query16 0.380.400.38 query17 1.091.111.12 query18 0.250.230.24 query19 2.021.811.95 query20 0.020.020.02 query21 15.38 0.980.72 query22 0.971.070.93 query23 14.68 1.510.88 query24 4.860.630.39 query25 0.180.120.11 query26 0.560.230.20 query27 0.100.100.11 query28 11.09 1.160.64 query29 12.66 4.013.49 query30 3.153.003.00 query31 2.850.630.46 query32 3.240.650.57 query33 3.083.273.26 query34 16.63 5.444.93 query35 4.865.045.03 query36 0.690.550.53 query37 0.260.230.23 query38 0.240.230.23 query39 0.060.050.05 query40 0.200.180.16 query41 0.120.070.07 query42 0.080.070.07 query43 0.070.070.06 Total cold run time: 107.83 s Total hot run time: 34.99 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
doris-robot commented on PR #54243: URL: https://github.com/apache/doris/pull/54243#issuecomment-3172483876 TPC-DS: Total hot run time: 169837 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit e1d519b7520a0dddb83e7c0d9e59b8d0304eb92a, data reload: false query1 998 406 419 406 query2 6532183116921692 query3 6743217 219 217 query4 26859 23798 22930 22930 query5 4361634 522 522 query6 333 248 238 238 query7 4633521 289 289 query8 299 231 231 231 query9 8612293429272927 query10 477 313 282 282 query11 15838 14946 14709 14709 query12 180 134 130 130 query13 1649543 414 414 query14 8540589457955795 query15 256 190 177 177 query16 8046634 480 480 query17 1640778 656 656 query18 2097451 325 325 query19 283 214 193 193 query20 156 138 135 135 query21 228 131 114 114 query22 4149393739883937 query23 34627 34157 34103 34103 query24 8135239024212390 query25 588 547 498 498 query26 885 286 160 160 query27 2648493 371 371 query28 4250232023162316 query29 693 589 490 490 query30 292 247 203 203 query31 897 772 708 708 query32 87 76 80 76 query33 556 416 371 371 query34 861 870 509 509 query35 829 837 773 773 query36 10211031947 947 query37 140 107 90 90 query38 3888401339363936 query39 1434139413791379 query40 233 141 133 133 query41 59 58 54 54 query42 141 130 126 126 query43 502 527 472 472 query44 1421856 856 856 query45 192 193 183 183 query46 941 1052671 671 query47 1779183717321732 query48 410 420 308 308 query49 676 508 409 409 query50 652 692 408 408 query51 4198413842554138 query52 126 131 117 117 query53 259 291 207 207 query54 665 632 554 554 query55 88 89 87 87 query56 346 366 352 352 query57 1221118611191119 query58 351 324 330 324 query59 2642271925282528 query60 406 384 374 374 query61 125 130 147 130 query62 826 727 631 631 query63 250 206 208 206 query64 35291107842 842 query65 4220414341454143 query66 899 474 335 335 query67 query68 17001 867 1049867 query69 1039269 293 269 query70 1358115211161116 query71 724 315 317 315 query72 9159232422192219 query73 3299686 354 354 query74 9115892187538753 query75 7686314126622662 query76 88601216787 787 query77 1161415 337 337 query78 954210726 94799479 query79 15051 619 593 593 query80 1883561 508 508 query81 529 261 226 226 query82 511 152 114 114 query83 367 286 276 276 query84 310 101 86 86 query85 862 381 342 342 query86 363 310 319 310 query87 4260412740554055 query88 5531225422652254 query89 486 363 322 322 query90 2459234 240 234 query91 151 143 111 111 query92 93 73 68 68 query93 6404984 651 651 query94 1059395 282 282 query95 434 335 387 335 query96 515 601 282 282 query97 2702269625672567 query98 284 240 222 222 query99 1491142312831283 Total cold run time: 306586 ms Total hot run time: 169837 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
doris-robot commented on PR #54243: URL: https://github.com/apache/doris/pull/54243#issuecomment-3172477971 TPC-H: Total hot run time: 33618 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit e1d519b7520a0dddb83e7c0d9e59b8d0304eb92a, data reload: false -- Round 1 -- q1 16408 544953095309 q2 1912258 189 189 q3 98411227696 696 q4 9684988 505 505 q5 7490232221992199 q6 182 166 131 131 q7 914 746 630 630 q8 9112125910151015 q9 6880504550765045 q10 6901233119731973 q11 475 283 276 276 q12 348 371 216 216 q13 17371 344229852985 q14 235 227 220 220 q15 510 473 457 457 q16 432 414 365 365 q17 565 800 367 367 q18 7169696670556966 q19 905 962 523 523 q20 315 293 204 204 q21 2699296423182318 q22 998 102910361029 Total cold run time: 101346 ms Total hot run time: 33618 ms - Round 2, with runtime_filter_mode=off - q1 5319535552505250 q2 229 298 220 220 q3 2100259921582158 q4 1312169112981298 q5 4120404541954045 q6 217 183 137 137 q7 2027186719681867 q8 2547243324692433 q9 7136725471787178 q10 3137342129402940 q11 548 524 489 489 q12 736 816 641 641 q13 3515376033903390 q14 273 286 288 286 q15 486 458 436 436 q16 450 484 452 452 q17 1119129712831283 q18 7831759976317599 q19 1632911 925 911 q20 1998206219351935 q21 4823426342094209 q22 1023103110331031 Total cold run time: 52578 ms Total hot run time: 50188 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
doris-robot commented on PR #54243: URL: https://github.com/apache/doris/pull/54243#issuecomment-3172467260 # Cloud UT Coverage Report Increment line coverage ` ` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/e1d519b7520a0dddb83e7c0d9e59b8d0304eb92a_e1d519b7520a0dddb83e7c0d9e59b8d0304eb92a_cloud/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/e1d519b7520a0dddb83e7c0d9e59b8d0304eb92a_e1d519b7520a0dddb83e7c0d9e59b8d0304eb92a_cloud/report/index.html) | Category | Coverage | |---|| | Function Coverage | 81.91% (1413/1725) | | Line Coverage | 66.17% (24233/36622) | | Region Coverage | 67.36% (12058/17901) | | Branch Coverage | 57.12% (6321/11066) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
yiguolei commented on PR #54243: URL: https://github.com/apache/doris/pull/54243#issuecomment-3172451476 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
yiguolei commented on PR #54243: URL: https://github.com/apache/doris/pull/54243#issuecomment-3172322521 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
yiguolei commented on PR #54243: URL: https://github.com/apache/doris/pull/54243#issuecomment-3172309341 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
yiguolei commented on PR #54243: URL: https://github.com/apache/doris/pull/54243#issuecomment-3172256955 run p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
hello-stephen commented on PR #54243: URL: https://github.com/apache/doris/pull/54243#issuecomment-3171990886 # BE Regression && UT Coverage Report Increment line coverage `77.97% (92/118)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/54243_302da54a34adece062ba1b161f74b148b95a3b2c_merge/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/54243_302da54a34adece062ba1b161f74b148b95a3b2c_merge/report/index.html) | Category | Coverage | |---|| | Function Coverage | 81.35% (22486/27642) | | Line Coverage | 74.05% (232945/314576) | | Region Coverage | 61.52% (194145/315604) | | Branch Coverage | 65.50% (83713/127807) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
doris-robot commented on PR #54243: URL: https://github.com/apache/doris/pull/54243#issuecomment-3171877229 # BE UT Coverage Report Increment line coverage `49.15% (58/118)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/302da54a34adece062ba1b161f74b148b95a3b2c_302da54a34adece062ba1b161f74b148b95a3b2c/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/302da54a34adece062ba1b161f74b148b95a3b2c_302da54a34adece062ba1b161f74b148b95a3b2c/report/index.html) | Category | Coverage | |---|| | Function Coverage | 58.89% (16585/28165) | | Line Coverage | 47.78% (150306/314561) | | Region Coverage | 36.63% (112590/307333) | | Branch Coverage | 39.62% (50027/126257) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
doris-robot commented on PR #54243: URL: https://github.com/apache/doris/pull/54243#issuecomment-3171861107 ClickBench: Total hot run time: 35.05 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 302da54a34adece062ba1b161f74b148b95a3b2c, data reload: false query1 0.040.030.04 query2 0.110.060.06 query3 0.290.060.07 query4 1.610.080.08 query5 0.420.420.44 query6 1.170.700.68 query7 0.040.020.02 query8 0.070.050.06 query9 0.610.480.50 query10 0.550.550.56 query11 0.250.120.12 query12 0.250.130.13 query13 0.680.690.68 query14 0.921.101.12 query15 1.050.940.93 query16 0.380.390.39 query17 1.111.121.10 query18 0.250.230.24 query19 2.171.891.98 query20 0.020.010.01 query21 15.34 0.970.71 query22 0.971.160.92 query23 14.72 1.500.87 query24 4.910.630.40 query25 0.180.120.11 query26 0.570.230.20 query27 0.110.110.11 query28 11.14 1.150.65 query29 12.63 3.993.48 query30 3.113.003.06 query31 2.820.620.46 query32 3.240.660.57 query33 3.123.263.28 query34 16.42 5.514.95 query35 4.894.965.08 query36 0.670.540.51 query37 0.250.230.22 query38 0.240.240.23 query39 0.070.060.06 query40 0.200.180.17 query41 0.120.070.08 query42 0.080.070.08 query43 0.070.060.06 Total cold run time: 107.86 s Total hot run time: 35.05 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
doris-robot commented on PR #54243: URL: https://github.com/apache/doris/pull/54243#issuecomment-3171825630 TPC-DS: Total hot run time: 170582 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 302da54a34adece062ba1b161f74b148b95a3b2c, data reload: false query1 996 407 425 407 query2 6511191817371737 query3 6740223 223 223 query4 26949 23860 22793 22793 query5 4340636 519 519 query6 342 251 228 228 query7 4641516 302 302 query8 278 236 222 222 query9 8570293429772934 query10 510 377 321 321 query11 15623 15120 14764 14764 query12 183 138 142 138 query13 1684549 420 420 query14 8684595858465846 query15 208 187 184 184 query16 7712684 494 494 query17 1267747 646 646 query18 2009451 313 313 query19 223 207 187 187 query20 150 179 142 142 query21 220 122 102 102 query22 4087409540834083 query23 34498 34398 34403 34398 query24 5983242924632429 query25 486 494 431 431 query26 709 306 158 158 query27 2336509 345 345 query28 2997233823422338 query29 637 613 478 478 query30 288 231 204 204 query31 871 784 712 712 query32 90 80 82 80 query33 542 408 362 362 query34 838 865 517 517 query35 795 835 785 785 query36 10111041917 917 query37 128 115 92 92 query38 4012392339613923 query39 1459137113931371 query40 242 143 129 129 query41 63 56 57 56 query42 153 127 127 127 query43 504 530 489 489 query44 1410874 860 860 query45 198 192 185 185 query46 956 1086673 673 query47 1805180817391739 query48 410 446 308 308 query49 674 500 410 410 query50 660 690 425 425 query51 4173420843414208 query52 134 132 121 121 query53 259 289 206 206 query54 644 646 568 568 query55 96 91 87 87 query56 360 357 334 334 query57 1186121111661166 query58 341 355 342 342 query59 2618268625832583 query60 413 394 387 387 query61 129 121 127 121 query62 756 749 668 668 query63 246 213 217 213 query64 23781167776 776 query65 4238414041614140 query66 916 450 332 332 query67 query68 18173 994 909 909 query69 1059277 285 277 query70 1420112211581122 query71 717 338 326 326 query72 9142227223552272 query73 3706635 360 360 query74 9341892887418741 query75 7600309126842684 query76 89431225863 863 query77 1147418 331 331 query78 9510971292349234 query79 13841 609 584 584 query80 942 554 541 541 query81 545 259 227 227 query82 360 151 119 119 query83 302 292 280 280 query84 298 100 82 82 query85 817 371 342 342 query86 354 321 300 300 query87 4201413241004100 query88 3780224322522243 query89 475 363 316 316 query90 2636228 232 228 query91 158 146 118 118 query92 93 73 77 73 query93 4403974 669 669 query94 921 416 276 276 query95 408 327 329 327 query96 513 640 286 286 query97 2686273226172617 query98 262 239 220 220 query99 1424145513171317 Total cold run time: 295746 ms Total hot run time: 170582 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
doris-robot commented on PR #54243: URL: https://github.com/apache/doris/pull/54243#issuecomment-3171681909 TPC-H: Total hot run time: 33850 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 302da54a34adece062ba1b161f74b148b95a3b2c, data reload: false -- Round 1 -- q1 17575 534452755275 q2 1920291 197 197 q3 10302 1294694 694 q4 10215 1009553 553 q5 7700224723722247 q6 238 166 131 131 q7 913 758 628 628 q8 9307127810951095 q9 6892510351225103 q10 6975237920092009 q11 455 279 268 268 q12 356 356 234 234 q13 17781 346430093009 q14 227 251 223 223 q15 543 456 469 456 q16 419 426 376 376 q17 582 833 366 366 q18 7234695670106956 q19 1243993 554 554 q20 329 318 215 215 q21 3429246222722272 q22 10671041989 989 Total cold run time: 105702 ms Total hot run time: 33850 ms - Round 2, with runtime_filter_mode=off - q1 5386537652715271 q2 238 308 225 225 q3 2076255622172217 q4 1304170312941294 q5 4133422445284224 q6 222 175 137 137 q7 1930198318771877 q8 2572254725342534 q9 7353714973017149 q10 3276329729102910 q11 563 534 586 534 q12 775 763 626 626 q13 3454370533673367 q14 298 306 303 303 q15 568 454 451 451 q16 584 485 465 465 q17 1178149313811381 q18 8114765376727653 q19 11765 983 1093983 q20 1927199617631763 q21 14739 433542284228 q22 1052102310031003 Total cold run time: 73507 ms Total hot run time: 50595 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
hello-stephen commented on PR #54243: URL: https://github.com/apache/doris/pull/54243#issuecomment-3171252132 # FE UT Coverage Report Increment line coverage ` 60.18% (133/221)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/54243_302da54a34adece062ba1b161f74b148b95a3b2c/fe_increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/54243_302da54a34adece062ba1b161f74b148b95a3b2c/fe_report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
hello-stephen commented on PR #54243: URL: https://github.com/apache/doris/pull/54243#issuecomment-3170819570 # Cloud UT Coverage Report Increment line coverage ` ` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/302da54a34adece062ba1b161f74b148b95a3b2c_302da54a34adece062ba1b161f74b148b95a3b2c_cloud/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/302da54a34adece062ba1b161f74b148b95a3b2c_302da54a34adece062ba1b161f74b148b95a3b2c_cloud/report/index.html) | Category | Coverage | |---|| | Function Coverage | 81.91% (1413/1725) | | Line Coverage | 66.19% (24239/36622) | | Region Coverage | 67.42% (12069/17901) | | Branch Coverage | 57.18% (6328/11066) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
yiguolei commented on PR #54243: URL: https://github.com/apache/doris/pull/54243#issuecomment-3170722318 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
yiguolei commented on code in PR #54243:
URL: https://github.com/apache/doris/pull/54243#discussion_r2262083316
##
fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroup.java:
##
@@ -187,32 +185,49 @@ private WorkloadGroup(long id, String name, Map properties, long
this.properties.put(SLOT_MEMORY_POLICY,
SLOT_MEMORY_POLICY_DEFAULT_VALUE);
}
-if (properties.containsKey(ENABLE_MEMORY_OVERCOMMIT)) {
-properties.put(ENABLE_MEMORY_OVERCOMMIT,
properties.get(ENABLE_MEMORY_OVERCOMMIT).toLowerCase());
+if (properties.containsKey(MAX_CPU_PERCENT)) {
+String cpuHardLimitStr = properties.get(MAX_CPU_PERCENT);
+if (cpuHardLimitStr.endsWith("%")) {
+cpuHardLimitStr = cpuHardLimitStr.substring(0,
cpuHardLimitStr.length() - 1);
+}
+this.properties.put(MAX_CPU_PERCENT, cpuHardLimitStr);
} else {
-properties.put(ENABLE_MEMORY_OVERCOMMIT, "false");
+this.properties.put(MAX_CPU_PERCENT, MAX_CPU_PERCENT_DEFAULT_VALUE
+ "");
}
-if (properties.containsKey(CPU_HARD_LIMIT)) {
-String cpuHardLimitStr = properties.get(CPU_HARD_LIMIT);
+if (properties.containsKey(MIN_CPU_PERCENT)) {
+String cpuHardLimitStr = properties.get(MIN_CPU_PERCENT);
if (cpuHardLimitStr.endsWith("%")) {
cpuHardLimitStr = cpuHardLimitStr.substring(0,
cpuHardLimitStr.length() - 1);
}
-this.properties.put(CPU_HARD_LIMIT, cpuHardLimitStr);
+this.properties.put(MIN_CPU_PERCENT, cpuHardLimitStr);
} else {
-this.properties.put(CPU_HARD_LIMIT, CPU_HARD_LIMIT_DEFAULT_VALUE +
"");
+this.properties.put(MIN_CPU_PERCENT, "0");
}
-if (properties.containsKey(MEMORY_LIMIT)) {
-String memHardLimitStr = properties.get(MEMORY_LIMIT);
+if (properties.containsKey(MAX_MEMORY_PERCENT)) {
+String memHardLimitStr = properties.get(MAX_MEMORY_PERCENT);
// If the input is -1, it means use all memory, in this version we
could change it to 100% now
// since sum of all group's memory could be larger than 100%
if (memHardLimitStr.equals("-1")) {
Review Comment:
fixed
##
fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroupMgr.java:
##
@@ -219,27 +228,48 @@ public void createWorkloadGroup(String computeGroup,
WorkloadGroup workloadGroup
// when create/alter workload group with same tag.
// when oldWg is not null it means caller is an alter stmt.
private void checkGlobalUnlock(WorkloadGroup newWg, WorkloadGroup oldWg)
throws DdlException {
+// trim all properties
+newWg.trimProperties();
String newWgCg = newWg.getComputeGroup();
int wgNumOfCurrentCg = 0;
-boolean isAlterStmt = oldWg != null;
-boolean isCreateStmt = !isAlterStmt;
+boolean isCreateStmt = oldWg == null;
+int sumOfMinCpuPercent = 0;
+int sumOfMinMemoryPercent = 0;
+++ wgNumOfCurrentCg;
+sumOfMinCpuPercent += newWg.getMinCpuPercent();
+sumOfMinMemoryPercent += newWg.getMinMemoryPercent();
+
// 1 get sum value of all wg which has same tag without current wg
for (Map.Entry entry :
idToWorkloadGroup.entrySet()) {
WorkloadGroup wg = entry.getValue();
-String curWgCg = wg.getComputeGroup();
-
-if (newWgCg.equals(entry.getValue().getComputeGroup())) {
-wgNumOfCurrentCg++;
-}
-
-if (isAlterStmt && entry.getKey() == oldWg.getId()) {
+if (!newWgCg.equalsIgnoreCase(wg.getComputeGroup())) {
continue;
}
-if (!newWgCg.equals(curWgCg)) {
-continue;
+if (newWg.getMinCpuPercent() > newWg.getMaxCpuPercent()) {
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
doris-robot commented on PR #54243: URL: https://github.com/apache/doris/pull/54243#issuecomment-3166536359 # BE UT Coverage Report Increment line coverage `49.15% (58/118)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/1ec4e969804dfbe02f9f3d9e9c4a0a853db96b5b_1ec4e969804dfbe02f9f3d9e9c4a0a853db96b5b/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/1ec4e969804dfbe02f9f3d9e9c4a0a853db96b5b_1ec4e969804dfbe02f9f3d9e9c4a0a853db96b5b/report/index.html) | Category | Coverage | |---|| | Function Coverage | 58.91% (16597/28174) | | Line Coverage | 47.83% (150594/314842) | | Region Coverage | 36.69% (112799/307452) | | Branch Coverage | 39.68% (50183/126468) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
wangbo commented on code in PR #54243:
URL: https://github.com/apache/doris/pull/54243#discussion_r2261871351
##
fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroup.java:
##
@@ -187,32 +185,49 @@ private WorkloadGroup(long id, String name, Map properties, long
this.properties.put(SLOT_MEMORY_POLICY,
SLOT_MEMORY_POLICY_DEFAULT_VALUE);
}
-if (properties.containsKey(ENABLE_MEMORY_OVERCOMMIT)) {
-properties.put(ENABLE_MEMORY_OVERCOMMIT,
properties.get(ENABLE_MEMORY_OVERCOMMIT).toLowerCase());
+if (properties.containsKey(MAX_CPU_PERCENT)) {
+String cpuHardLimitStr = properties.get(MAX_CPU_PERCENT);
+if (cpuHardLimitStr.endsWith("%")) {
+cpuHardLimitStr = cpuHardLimitStr.substring(0,
cpuHardLimitStr.length() - 1);
+}
+this.properties.put(MAX_CPU_PERCENT, cpuHardLimitStr);
} else {
-properties.put(ENABLE_MEMORY_OVERCOMMIT, "false");
+this.properties.put(MAX_CPU_PERCENT, MAX_CPU_PERCENT_DEFAULT_VALUE
+ "");
}
-if (properties.containsKey(CPU_HARD_LIMIT)) {
-String cpuHardLimitStr = properties.get(CPU_HARD_LIMIT);
+if (properties.containsKey(MIN_CPU_PERCENT)) {
+String cpuHardLimitStr = properties.get(MIN_CPU_PERCENT);
if (cpuHardLimitStr.endsWith("%")) {
cpuHardLimitStr = cpuHardLimitStr.substring(0,
cpuHardLimitStr.length() - 1);
}
-this.properties.put(CPU_HARD_LIMIT, cpuHardLimitStr);
+this.properties.put(MIN_CPU_PERCENT, cpuHardLimitStr);
} else {
-this.properties.put(CPU_HARD_LIMIT, CPU_HARD_LIMIT_DEFAULT_VALUE +
"");
+this.properties.put(MIN_CPU_PERCENT, "0");
}
-if (properties.containsKey(MEMORY_LIMIT)) {
-String memHardLimitStr = properties.get(MEMORY_LIMIT);
+if (properties.containsKey(MAX_MEMORY_PERCENT)) {
+String memHardLimitStr = properties.get(MAX_MEMORY_PERCENT);
// If the input is -1, it means use all memory, in this version we
could change it to 100% now
// since sum of all group's memory could be larger than 100%
if (memHardLimitStr.equals("-1")) {
Review Comment:
max_memory_percent could not be -1 here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
wangbo commented on code in PR #54243:
URL: https://github.com/apache/doris/pull/54243#discussion_r2261858069
##
fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroupMgr.java:
##
@@ -219,27 +228,48 @@ public void createWorkloadGroup(String computeGroup,
WorkloadGroup workloadGroup
// when create/alter workload group with same tag.
// when oldWg is not null it means caller is an alter stmt.
private void checkGlobalUnlock(WorkloadGroup newWg, WorkloadGroup oldWg)
throws DdlException {
+// trim all properties
+newWg.trimProperties();
String newWgCg = newWg.getComputeGroup();
int wgNumOfCurrentCg = 0;
-boolean isAlterStmt = oldWg != null;
-boolean isCreateStmt = !isAlterStmt;
+boolean isCreateStmt = oldWg == null;
+int sumOfMinCpuPercent = 0;
+int sumOfMinMemoryPercent = 0;
+++ wgNumOfCurrentCg;
+sumOfMinCpuPercent += newWg.getMinCpuPercent();
+sumOfMinMemoryPercent += newWg.getMinMemoryPercent();
+
// 1 get sum value of all wg which has same tag without current wg
for (Map.Entry entry :
idToWorkloadGroup.entrySet()) {
WorkloadGroup wg = entry.getValue();
-String curWgCg = wg.getComputeGroup();
-
-if (newWgCg.equals(entry.getValue().getComputeGroup())) {
-wgNumOfCurrentCg++;
-}
-
-if (isAlterStmt && entry.getKey() == oldWg.getId()) {
+if (!newWgCg.equalsIgnoreCase(wg.getComputeGroup())) {
continue;
}
-if (!newWgCg.equals(curWgCg)) {
-continue;
+if (newWg.getMinCpuPercent() > newWg.getMaxCpuPercent()) {
Review Comment:
remove newWg's check outout for loop for cpu/memory
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
wangbo commented on code in PR #54243:
URL: https://github.com/apache/doris/pull/54243#discussion_r2261858069
##
fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroupMgr.java:
##
@@ -219,27 +228,48 @@ public void createWorkloadGroup(String computeGroup,
WorkloadGroup workloadGroup
// when create/alter workload group with same tag.
// when oldWg is not null it means caller is an alter stmt.
private void checkGlobalUnlock(WorkloadGroup newWg, WorkloadGroup oldWg)
throws DdlException {
+// trim all properties
+newWg.trimProperties();
String newWgCg = newWg.getComputeGroup();
int wgNumOfCurrentCg = 0;
-boolean isAlterStmt = oldWg != null;
-boolean isCreateStmt = !isAlterStmt;
+boolean isCreateStmt = oldWg == null;
+int sumOfMinCpuPercent = 0;
+int sumOfMinMemoryPercent = 0;
+++ wgNumOfCurrentCg;
+sumOfMinCpuPercent += newWg.getMinCpuPercent();
+sumOfMinMemoryPercent += newWg.getMinMemoryPercent();
+
// 1 get sum value of all wg which has same tag without current wg
for (Map.Entry entry :
idToWorkloadGroup.entrySet()) {
WorkloadGroup wg = entry.getValue();
-String curWgCg = wg.getComputeGroup();
-
-if (newWgCg.equals(entry.getValue().getComputeGroup())) {
-wgNumOfCurrentCg++;
-}
-
-if (isAlterStmt && entry.getKey() == oldWg.getId()) {
+if (!newWgCg.equalsIgnoreCase(wg.getComputeGroup())) {
continue;
}
-if (!newWgCg.equals(curWgCg)) {
-continue;
+if (newWg.getMinCpuPercent() > newWg.getMaxCpuPercent()) {
Review Comment:
remove newWg's check outout for loop
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
yiguolei commented on PR #54243: URL: https://github.com/apache/doris/pull/54243#issuecomment-3166325334 run feut -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
yiguolei commented on PR #54243: URL: https://github.com/apache/doris/pull/54243#issuecomment-3166325451 run beut -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
yiguolei commented on PR #54243: URL: https://github.com/apache/doris/pull/54243#issuecomment-3163380580 run feut -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
yiguolei commented on PR #54243: URL: https://github.com/apache/doris/pull/54243#issuecomment-3163380866 run beut -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
yiguolei commented on PR #54243: URL: https://github.com/apache/doris/pull/54243#issuecomment-3163040380 run feut -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
doris-robot commented on PR #54243: URL: https://github.com/apache/doris/pull/54243#issuecomment-3158367575 # BE UT Coverage Report Increment line coverage `48.33% (58/120)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/3532709f1ea972e709637404f0250fc442bbb1a7_3532709f1ea972e709637404f0250fc442bbb1a7/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/3532709f1ea972e709637404f0250fc442bbb1a7_3532709f1ea972e709637404f0250fc442bbb1a7/report/index.html) | Category | Coverage | |---|| | Function Coverage | 58.30% (16380/28097) | | Line Coverage | 47.20% (148092/313737) | | Region Coverage | 36.16% (110860/306572) | | Branch Coverage | 39.04% (49211/126039) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
doris-robot commented on PR #54243: URL: https://github.com/apache/doris/pull/54243#issuecomment-3158126149 ClickBench: Total hot run time: 35.23 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 3532709f1ea972e709637404f0250fc442bbb1a7, data reload: false query1 0.030.030.04 query2 0.110.050.06 query3 0.300.060.06 query4 1.600.080.09 query5 0.410.410.41 query6 1.170.670.65 query7 0.030.020.02 query8 0.080.050.05 query9 0.590.470.48 query10 0.530.530.53 query11 0.260.130.13 query12 0.250.140.13 query13 0.670.700.69 query14 0.931.111.30 query15 1.010.930.93 query16 0.390.400.39 query17 1.071.101.09 query18 0.260.230.24 query19 1.941.901.96 query20 0.010.010.01 query21 15.35 1.000.71 query22 0.991.130.97 query23 14.67 1.530.89 query24 4.950.620.38 query25 0.190.110.11 query26 0.560.230.20 query27 0.120.120.12 query28 11.08 1.150.66 query29 12.60 4.073.59 query30 3.083.013.02 query31 2.830.620.47 query32 3.260.650.56 query33 3.083.223.31 query34 16.77 5.514.88 query35 4.875.075.08 query36 0.670.550.52 query37 0.250.240.23 query38 0.230.240.24 query39 0.070.070.06 query40 0.210.180.17 query41 0.120.080.07 query42 0.080.070.07 query43 0.070.070.07 Total cold run time: 107.74 s Total hot run time: 35.23 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
doris-robot commented on PR #54243: URL: https://github.com/apache/doris/pull/54243#issuecomment-3158081164 TPC-DS: Total hot run time: 170687 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 3532709f1ea972e709637404f0250fc442bbb1a7, data reload: false query1 1004428 416 416 query2 6536186117111711 query3 6738226 221 221 query4 27079 23906 22882 22882 query5 4356633 529 529 query6 343 242 237 237 query7 4637519 303 303 query8 278 232 226 226 query9 8563299529422942 query10 478 337 289 289 query11 15636 14934 14802 14802 query12 184 140 130 130 query13 1675547 417 417 query14 9489590859545908 query15 211 184 167 167 query16 7815668 452 452 query17 1632783 654 654 query18 2122431 311 311 query19 311 205 180 180 query20 147 145 131 131 query21 218 164 107 107 query22 3987404939153915 query23 34761 34311 34354 34311 query24 7405239624032396 query25 536 523 435 435 query26 912 286 171 171 query27 2531485 356 356 query28 4317235623552355 query29 736 601 491 491 query30 295 233 200 200 query31 898 788 708 708 query32 94 74 74 74 query33 600 421 361 361 query34 829 830 532 532 query35 793 852 744 744 query36 10421047908 908 query37 132 109 95 95 query38 3967390239583902 query39 1463137413661366 query40 247 140 132 132 query41 62 58 54 54 query42 145 127 136 127 query43 496 517 495 495 query44 1393880 881 880 query45 198 184 177 177 query46 967 1067697 697 query47 1800185317501750 query48 415 430 313 313 query49 676 512 412 412 query50 674 695 423 423 query51 4211422142234221 query52 130 146 118 118 query53 257 288 226 226 query54 650 646 552 552 query55 91 86 92 86 query56 353 351 359 351 query57 1209122111341134 query58 338 334 331 331 query59 2659271525912591 query60 400 391 386 386 query61 127 121 146 121 query62 808 735 692 692 query63 252 216 219 216 query64 35571113779 779 query65 4266418741864186 query66 920 461 334 334 query67 query68 17053 878 857 857 query69 1101268 284 268 query70 1314114611211121 query71 711 310 327 310 query72 9208229421772177 query73 3176688 360 360 query74 9079911489138913 query75 7463310727102710 query76 84251216804 804 query77 1167406 354 354 query78 977310961 95519551 query79 14330 615 574 574 query80 1740607 470 470 query81 575 262 224 224 query82 624 152 118 118 query83 378 291 264 264 query84 299 98 84 84 query85 870 403 337 337 query86 365 345 296 296 query87 4205415841104110 query88 5449234922492249 query89 488 370 310 310 query90 2599234 238 234 query91 145 139 113 113 query92 90 72 67 67 query93 61711008658 658 query94 1001401 279 279 query95 399 337 320 320 query96 510 580 282 282 query97 2705272526372637 query98 252 247 222 222 query99 1470146512951295 Total cold run time: 305186 ms Total hot run time: 170687 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abo
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
doris-robot commented on PR #54243: URL: https://github.com/apache/doris/pull/54243#issuecomment-3157986436 TPC-H: Total hot run time: 34160 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 3532709f1ea972e709637404f0250fc442bbb1a7, data reload: false -- Round 1 -- q1 17660 539353305330 q2 1913288 178 178 q3 10682 1379762 762 q4 10324 1073550 550 q5 10173 229923262299 q6 237 168 134 134 q7 894 759 639 639 q8 9296129811121112 q9 6831509450965094 q10 6957238219831983 q11 476 285 265 265 q12 352 362 223 223 q13 17812 340729962996 q14 239 251 224 224 q15 546 469 473 469 q16 445 419 375 375 q17 584 826 357 357 q18 7393702870537028 q19 2096976 542 542 q20 318 307 211 211 q21 3417308023912391 q22 11051133998 998 Total cold run time: 109750 ms Total hot run time: 34160 ms - Round 2, with runtime_filter_mode=off - q1 5542523552655235 q2 236 311 225 225 q3 2086255721532153 q4 1297172813731373 q5 4473440545264405 q6 222 185 133 133 q7 1874194519471945 q8 2532243825982438 q9 7268731671887188 q10 3148344228802880 q11 648 509 490 490 q12 669 811 623 623 q13 3356369932043204 q14 296 290 302 290 q15 492 466 472 466 q16 460 500 467 467 q17 1181151313421342 q18 7980765578047655 q19 15114 895 981 895 q20 2815190818101810 q21 13878 428243124282 q22 1087103810361036 Total cold run time: 76654 ms Total hot run time: 50535 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
wangbo commented on code in PR #54243:
URL: https://github.com/apache/doris/pull/54243#discussion_r2256046074
##
be/src/runtime/workload_group/workload_group.h:
##
@@ -293,16 +283,18 @@ using WorkloadGroupPtr = std::shared_ptr;
struct WorkloadGroupInfo {
const uint64_t id = 0;
const std::string name = "";
-const uint64_t cpu_share = 0;
-const int64_t memory_limit = 0;
-const bool enable_memory_overcommit = false;
const int64_t version = 0;
-const int cpu_hard_limit = 0;
+const int min_cpu_percent = 0;
+const int max_cpu_percent = 100;
+const bool enable_memory_overcommit = false;
+const int64_t memory_limit = 1 << 30; // Default to 1GB
Review Comment:
这个memory limit的作用是啥
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
yiguolei commented on code in PR #54243:
URL: https://github.com/apache/doris/pull/54243#discussion_r2256026842
##
fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroup.java:
##
@@ -96,34 +105,34 @@ public class WorkloadGroup implements Writable,
GsonPostProcessable {
// default value is as followed
// cpu_share=1024, memory_limit=0%(0 means not limit),
enable_memory_overcommit=true
private static final ImmutableSet ALL_PROPERTIES_NAME = new
ImmutableSet.Builder()
-
.add(CPU_SHARE).add(MEMORY_LIMIT).add(ENABLE_MEMORY_OVERCOMMIT).add(MAX_CONCURRENCY)
-
.add(MAX_QUEUE_SIZE).add(QUEUE_TIMEOUT).add(CPU_HARD_LIMIT).add(SCAN_THREAD_NUM)
-.add(MAX_REMOTE_SCAN_THREAD_NUM).add(MIN_REMOTE_SCAN_THREAD_NUM)
-.add(MEMORY_LOW_WATERMARK).add(MEMORY_HIGH_WATERMARK)
+.add(MAX_CPU_PERCENT).add(MIN_CPU_PERCENT)
+
.add(MAX_MEMORY_PERCENT).add(MIN_MEMORY_PERCENT).add(ENABLE_MEMORY_OVERCOMMIT)
Review Comment:
还不太行,现在老版本升级过来的normal的内存只有40%,之前是软限,所以没问题。 我们现在都改成硬限了,会导致query
被cancel。所以还得让默认的enable memory overcommit 发挥作用,让他是true
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
wangbo commented on code in PR #54243:
URL: https://github.com/apache/doris/pull/54243#discussion_r2256025428
##
fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroup.java:
##
@@ -94,36 +103,35 @@ public class WorkloadGroup implements Writable,
GsonPostProcessable {
// NOTE(wb): all property is not required, some properties default value
is set in be
// default value is as followed
-// cpu_share=1024, memory_limit=0%(0 means not limit),
enable_memory_overcommit=true
private static final ImmutableSet ALL_PROPERTIES_NAME = new
ImmutableSet.Builder()
-
.add(CPU_SHARE).add(MEMORY_LIMIT).add(ENABLE_MEMORY_OVERCOMMIT).add(MAX_CONCURRENCY)
-
.add(MAX_QUEUE_SIZE).add(QUEUE_TIMEOUT).add(CPU_HARD_LIMIT).add(SCAN_THREAD_NUM)
-.add(MAX_REMOTE_SCAN_THREAD_NUM).add(MIN_REMOTE_SCAN_THREAD_NUM)
-.add(MEMORY_LOW_WATERMARK).add(MEMORY_HIGH_WATERMARK)
+.add(MAX_CPU_PERCENT).add(MIN_CPU_PERCENT)
+
.add(MAX_MEMORY_PERCENT).add(MIN_MEMORY_PERCENT).add(ENABLE_MEMORY_OVERCOMMIT)
+.add(MAX_CONCURRENCY).add(MAX_QUEUE_SIZE).add(QUEUE_TIMEOUT)
+
.add(SCAN_THREAD_NUM).add(MAX_REMOTE_SCAN_THREAD_NUM).add(MIN_REMOTE_SCAN_THREAD_NUM)
+
.add(MEMORY_LOW_WATERMARK).add(MEMORY_HIGH_WATERMARK).add(WRITE_BUFFER_RATIO)
.add(COMPUTE_GROUP).add(READ_BYTES_PER_SECOND).add(REMOTE_READ_BYTES_PER_SECOND)
-.add(WRITE_BUFFER_RATIO).add(SLOT_MEMORY_POLICY).build();
+.add(SLOT_MEMORY_POLICY).build();
private static final ImmutableMap
DEPRECATED_PROPERTIES_NAME =
new ImmutableMap.Builder()
.put(SPILL_THRESHOLD_LOW_WATERMARK, MEMORY_LOW_WATERMARK)
-.put(SPILL_THRESHOLD_HIGH_WATERMARK,
MEMORY_HIGH_WATERMARK).build();
+.put(SPILL_THRESHOLD_HIGH_WATERMARK, MEMORY_HIGH_WATERMARK)
+.put(MEMORY_LIMIT, MAX_MEMORY_PERCENT)
+.put(CPU_HARD_LIMIT, MAX_CPU_PERCENT).build();
-public static final int CPU_HARD_LIMIT_DEFAULT_VALUE = 100;
+public static final int MAX_CPU_PERCENT_DEFAULT_VALUE = 100;
// Memory limit is a string value ending with % in BE, so it is different
from other limits
// other limit is a number.
-public static final String MEMORY_LIMIT_DEFAULT_VALUE = "100%";
+public static final String MAX_MEMORY_PERCENT_DEFAULT_VALUE = "100";
public static final int MEMORY_LOW_WATERMARK_DEFAULT_VALUE = 75;
public static final int MEMORY_HIGH_WATERMARK_DEFAULT_VALUE = 85;
private static final Map ALL_PROPERTIES_DEFAULT_VALUE_MAP
= Maps.newHashMap();
static {
-ALL_PROPERTIES_DEFAULT_VALUE_MAP.put(CPU_SHARE, "-1");
-ALL_PROPERTIES_DEFAULT_VALUE_MAP.put(CPU_HARD_LIMIT, "100");
-ALL_PROPERTIES_DEFAULT_VALUE_MAP.put(MEMORY_LIMIT, "100%");
-ALL_PROPERTIES_DEFAULT_VALUE_MAP.put(ENABLE_MEMORY_OVERCOMMIT,
"false");
+ALL_PROPERTIES_DEFAULT_VALUE_MAP.put(ENABLE_MEMORY_OVERCOMMIT, "true");
Review Comment:
还有这里的ENABLE_MEMORY_OVERCOMMIT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
yiguolei commented on code in PR #54243:
URL: https://github.com/apache/doris/pull/54243#discussion_r2256024874
##
fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroup.java:
##
@@ -553,6 +614,52 @@ public long getVersion() {
return version;
}
+public int getMinCpuPercent() {
+String minCpuPercentStr = properties.get(MIN_CPU_PERCENT);
+if (minCpuPercentStr.endsWith("%")) {
+return Integer.valueOf(minCpuPercentStr.substring(0,
minCpuPercentStr.length() - 1));
+}
+return Integer.valueOf(minCpuPercentStr);
+}
+
+public int getMaxCpuPercent() {
+String maxCpuPercentStr = properties.get(MAX_CPU_PERCENT);
+if (maxCpuPercentStr.endsWith("%")) {
+return Integer.valueOf(maxCpuPercentStr.substring(0,
maxCpuPercentStr.length() - 1));
+}
+return Integer.valueOf(maxCpuPercentStr);
+}
+
+public int getMinMemoryPercent() {
+String minMemPercentStr = properties.get(MIN_MEMORY_PERCENT);
+if (minMemPercentStr.endsWith("%")) {
+return Integer.valueOf(minMemPercentStr.substring(0,
+minMemPercentStr.length() - 1));
+}
+return Integer.valueOf(minMemPercentStr);
+}
+
+public int getMaxMemoryPercent() {
+String maxMemPercentStr = properties.get(MAX_MEMORY_PERCENT);
Review Comment:
嗯,是的。
我后来是写迷糊了,不知道什么时候用的%结尾,所以直接都判断一下是否%
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
wangbo commented on code in PR #54243:
URL: https://github.com/apache/doris/pull/54243#discussion_r2256022790
##
fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroup.java:
##
@@ -96,34 +105,34 @@ public class WorkloadGroup implements Writable,
GsonPostProcessable {
// default value is as followed
// cpu_share=1024, memory_limit=0%(0 means not limit),
enable_memory_overcommit=true
Review Comment:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
wangbo commented on code in PR #54243:
URL: https://github.com/apache/doris/pull/54243#discussion_r2256021426
##
fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroup.java:
##
@@ -96,34 +105,34 @@ public class WorkloadGroup implements Writable,
GsonPostProcessable {
// default value is as followed
// cpu_share=1024, memory_limit=0%(0 means not limit),
enable_memory_overcommit=true
private static final ImmutableSet ALL_PROPERTIES_NAME = new
ImmutableSet.Builder()
-
.add(CPU_SHARE).add(MEMORY_LIMIT).add(ENABLE_MEMORY_OVERCOMMIT).add(MAX_CONCURRENCY)
-
.add(MAX_QUEUE_SIZE).add(QUEUE_TIMEOUT).add(CPU_HARD_LIMIT).add(SCAN_THREAD_NUM)
-.add(MAX_REMOTE_SCAN_THREAD_NUM).add(MIN_REMOTE_SCAN_THREAD_NUM)
-.add(MEMORY_LOW_WATERMARK).add(MEMORY_HIGH_WATERMARK)
+.add(MAX_CPU_PERCENT).add(MIN_CPU_PERCENT)
+
.add(MAX_MEMORY_PERCENT).add(MIN_MEMORY_PERCENT).add(ENABLE_MEMORY_OVERCOMMIT)
Review Comment:
ENABLE_MEMORY_OVERCOMMIT这个是不是可以删了
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
wangbo commented on code in PR #54243:
URL: https://github.com/apache/doris/pull/54243#discussion_r2256016366
##
fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroup.java:
##
@@ -553,6 +614,52 @@ public long getVersion() {
return version;
}
+public int getMinCpuPercent() {
+String minCpuPercentStr = properties.get(MIN_CPU_PERCENT);
+if (minCpuPercentStr.endsWith("%")) {
+return Integer.valueOf(minCpuPercentStr.substring(0,
minCpuPercentStr.length() - 1));
+}
+return Integer.valueOf(minCpuPercentStr);
+}
+
+public int getMaxCpuPercent() {
+String maxCpuPercentStr = properties.get(MAX_CPU_PERCENT);
+if (maxCpuPercentStr.endsWith("%")) {
+return Integer.valueOf(maxCpuPercentStr.substring(0,
maxCpuPercentStr.length() - 1));
+}
+return Integer.valueOf(maxCpuPercentStr);
+}
+
+public int getMinMemoryPercent() {
+String minMemPercentStr = properties.get(MIN_MEMORY_PERCENT);
+if (minMemPercentStr.endsWith("%")) {
+return Integer.valueOf(minMemPercentStr.substring(0,
+minMemPercentStr.length() - 1));
+}
+return Integer.valueOf(minMemPercentStr);
+}
+
+public int getMaxMemoryPercent() {
+String maxMemPercentStr = properties.get(MAX_MEMORY_PERCENT);
Review Comment:
这种写法可读性稍微差点,容易引起误解。实际只有创建或者alter的时候会以%结尾
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
hello-stephen commented on PR #54243: URL: https://github.com/apache/doris/pull/54243#issuecomment-3157565636 # Cloud UT Coverage Report Increment line coverage ` ` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/3532709f1ea972e709637404f0250fc442bbb1a7_3532709f1ea972e709637404f0250fc442bbb1a7_cloud/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/3532709f1ea972e709637404f0250fc442bbb1a7_3532709f1ea972e709637404f0250fc442bbb1a7_cloud/report/index.html) | Category | Coverage | |---|| | Function Coverage | 81.60% (1401/1717) | | Line Coverage | 65.98% (23856/36158) | | Region Coverage | 67.24% (11844/17615) | | Branch Coverage | 56.81% (6193/10902) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
yiguolei commented on code in PR #54243:
URL: https://github.com/apache/doris/pull/54243#discussion_r2255951001
##
be/src/runtime/workload_group/workload_group.cpp:
##
@@ -148,40 +153,42 @@ std::string WorkloadGroup::_memory_debug_string() const {
std::string WorkloadGroup::memory_debug_string() const {
return fmt::format(
-"WorkloadGroup[id = {}, name = {}, version = {}, "
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
yiguolei commented on PR #54243: URL: https://github.com/apache/doris/pull/54243#issuecomment-3157518334 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
yiguolei commented on code in PR #54243:
URL: https://github.com/apache/doris/pull/54243#discussion_r2255777348
##
be/src/runtime/workload_group/workload_group.cpp:
##
@@ -148,40 +153,42 @@ std::string WorkloadGroup::_memory_debug_string() const {
std::string WorkloadGroup::memory_debug_string() const {
return fmt::format(
-"WorkloadGroup[id = {}, name = {}, version = {}, "
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
yiguolei commented on code in PR #54243:
URL: https://github.com/apache/doris/pull/54243#discussion_r2255775489
##
be/src/exec/schema_scanner/schema_workload_groups_scanner.cpp:
##
@@ -31,13 +31,13 @@ namespace doris {
std::vector
SchemaWorkloadGroupsScanner::_s_tbls_columns = {
{"ID", TYPE_BIGINT, sizeof(int64_t), true},
{"NAME", TYPE_VARCHAR, sizeof(StringRef), true},
-{"CPU_SHARE", TYPE_BIGINT, sizeof(int64_t), true},
-{"MEMORY_LIMIT", TYPE_VARCHAR, sizeof(StringRef), true},
-{"ENABLE_MEMORY_OVERCOMMIT", TYPE_VARCHAR, sizeof(StringRef), true},
+{"MIN_CPU_PERCENT", TYPE_BIGINT, sizeof(int64_t), true},
+{"MAX_CPU_PERCENT", TYPE_BIGINT, sizeof(int64_t), true},
+{"MIN_MEMORY_PERCENT", TYPE_BIGINT, sizeof(int64_t), true},
+{"MAX_MEMORY_PERCENT", TYPE_BIGINT, sizeof(int64_t), true},
Review Comment:
I think it is ok. If the memory is very large, use do not need care about
5GB memory.
But this is a bug, the type should be varchar, not int64.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
yiguolei commented on code in PR #54243:
URL: https://github.com/apache/doris/pull/54243#discussion_r2255655972
##
be/src/agent/cgroup_cpu_ctl.cpp:
##
@@ -404,13 +419,14 @@ Status CgroupV2CpuCtl::init() {
}
Status CgroupV2CpuCtl::modify_cg_cpu_hard_limit_no_lock(int cpu_hard_limit) {
-if (cpu_hard_limit <= 0) {
-return Status::InternalError("cpu hard limit must be greater
than 0");
-}
-std::string value = "";
-uint64_t int_val = _cpu_cfs_period_us * _cpu_core_num * cpu_hard_limit /
100;
-value = std::to_string(int_val) + " 10";
-std::string msg = "modify cpu.max to [" + value + "]";
+// If cpu_hard_limit is 0, we set the cpu.max to 1000 10.
Review Comment:
I change the min value for hard limit to 1%
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
yiguolei commented on code in PR #54243:
URL: https://github.com/apache/doris/pull/54243#discussion_r2254027782
##
fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroup.java:
##
@@ -261,69 +293,98 @@ public static WorkloadGroup copyAndUpdate(WorkloadGroup
currentWorkloadGroup, Ma
private static void checkProperties(Map properties) throws
DdlException {
for (String propertyName : properties.keySet()) {
if (!ALL_PROPERTIES_NAME.contains(propertyName)) {
-throw new DdlException("Property " + propertyName + " is not
supported.");
+throw new DdlException("Property " + propertyName + " is not
supported, maybe it is deprecated.");
}
}
-if (properties.containsKey(CPU_SHARE)) {
-String inputValue = properties.get(CPU_SHARE);
+int minCpuPercent = 0;
+if (properties.containsKey(MIN_CPU_PERCENT)) {
+String inputValue = properties.get(MIN_CPU_PERCENT);
+String minCpuPercentStr = inputValue;
try {
-int cpuShareI = Integer.parseInt(inputValue);
-if (cpuShareI <= 0 && cpuShareI != -1) {
+if (minCpuPercentStr.endsWith("%")) {
+minCpuPercentStr = minCpuPercentStr.substring(0,
minCpuPercentStr.length() - 1);
+}
+
+minCpuPercent = Integer.parseInt(minCpuPercentStr);
Review Comment:
它并不是当系统出现资源不足的时候,只能使用min cpu percent 这么多资源,而是至少可以使用这么多资源。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
yiguolei commented on code in PR #54243:
URL: https://github.com/apache/doris/pull/54243#discussion_r2254025857
##
fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroup.java:
##
@@ -261,69 +293,98 @@ public static WorkloadGroup copyAndUpdate(WorkloadGroup
currentWorkloadGroup, Ma
private static void checkProperties(Map properties) throws
DdlException {
for (String propertyName : properties.keySet()) {
if (!ALL_PROPERTIES_NAME.contains(propertyName)) {
-throw new DdlException("Property " + propertyName + " is not
supported.");
+throw new DdlException("Property " + propertyName + " is not
supported, maybe it is deprecated.");
}
}
-if (properties.containsKey(CPU_SHARE)) {
-String inputValue = properties.get(CPU_SHARE);
+int minCpuPercent = 0;
+if (properties.containsKey(MIN_CPU_PERCENT)) {
+String inputValue = properties.get(MIN_CPU_PERCENT);
+String minCpuPercentStr = inputValue;
try {
-int cpuShareI = Integer.parseInt(inputValue);
-if (cpuShareI <= 0 && cpuShareI != -1) {
+if (minCpuPercentStr.endsWith("%")) {
+minCpuPercentStr = minCpuPercentStr.substring(0,
minCpuPercentStr.length() - 1);
+}
+
+minCpuPercent = Integer.parseInt(minCpuPercentStr);
Review Comment:
换个角度理解,min cpu percent的含义是,当系统资源不足的时候,当前group 至少保留的资源量。 所以当配置成0的时候,实际使用50%
是合理的。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
wangbo commented on code in PR #54243:
URL: https://github.com/apache/doris/pull/54243#discussion_r2254013663
##
fe/fe-core/src/main/java/org/apache/doris/resource/workloadgroup/WorkloadGroup.java:
##
@@ -261,69 +293,98 @@ public static WorkloadGroup copyAndUpdate(WorkloadGroup
currentWorkloadGroup, Ma
private static void checkProperties(Map properties) throws
DdlException {
for (String propertyName : properties.keySet()) {
if (!ALL_PROPERTIES_NAME.contains(propertyName)) {
-throw new DdlException("Property " + propertyName + " is not
supported.");
+throw new DdlException("Property " + propertyName + " is not
supported, maybe it is deprecated.");
}
}
-if (properties.containsKey(CPU_SHARE)) {
-String inputValue = properties.get(CPU_SHARE);
+int minCpuPercent = 0;
+if (properties.containsKey(MIN_CPU_PERCENT)) {
+String inputValue = properties.get(MIN_CPU_PERCENT);
+String minCpuPercentStr = inputValue;
try {
-int cpuShareI = Integer.parseInt(inputValue);
-if (cpuShareI <= 0 && cpuShareI != -1) {
+if (minCpuPercentStr.endsWith("%")) {
+minCpuPercentStr = minCpuPercentStr.substring(0,
minCpuPercentStr.length() - 1);
+}
+
+minCpuPercent = Integer.parseInt(minCpuPercentStr);
Review Comment:
If user create wg1(minCpuPercent=0), then create wg2(minCpuPercent=0), How
should users understand this configuration? In CGroup, wg1 is 50% and wg2 is
50% when cpu is full. This means cpushares can not express the min cpu usage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
yiguolei commented on code in PR #54243:
URL: https://github.com/apache/doris/pull/54243#discussion_r2254010586
##
be/src/runtime/workload_group/workload_group.h:
##
@@ -243,29 +231,31 @@ class WorkloadGroup : public
std::enable_shared_from_this {
uint64_t _id;
std::string _name;
int64_t _version;
-int64_t _memory_limit; // bytes
+std::atomic _min_cpu_percent = 0;
+std::atomic _max_cpu_percent = 100;
+bool _enable_memory_overcommit = true;
Review Comment:
Currently it is useful, we only use it to skip memory check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
wangbo commented on code in PR #54243:
URL: https://github.com/apache/doris/pull/54243#discussion_r2253979563
##
be/src/runtime/workload_group/workload_group.h:
##
@@ -243,29 +231,31 @@ class WorkloadGroup : public
std::enable_shared_from_this {
uint64_t _id;
std::string _name;
int64_t _version;
-int64_t _memory_limit; // bytes
+std::atomic _min_cpu_percent = 0;
+std::atomic _max_cpu_percent = 100;
+bool _enable_memory_overcommit = true;
Review Comment:
_enable_memory_overcommit is useless?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
wangbo commented on code in PR #54243:
URL: https://github.com/apache/doris/pull/54243#discussion_r2253963713
##
be/src/runtime/workload_group/workload_group.cpp:
##
@@ -148,40 +153,42 @@ std::string WorkloadGroup::_memory_debug_string() const {
std::string WorkloadGroup::memory_debug_string() const {
return fmt::format(
-"WorkloadGroup[id = {}, name = {}, version = {}, "
Review Comment:
miss end of "["
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
wangbo commented on code in PR #54243:
URL: https://github.com/apache/doris/pull/54243#discussion_r2253949679
##
be/src/exec/schema_scanner/schema_workload_groups_scanner.cpp:
##
@@ -31,13 +31,13 @@ namespace doris {
std::vector
SchemaWorkloadGroupsScanner::_s_tbls_columns = {
{"ID", TYPE_BIGINT, sizeof(int64_t), true},
{"NAME", TYPE_VARCHAR, sizeof(StringRef), true},
-{"CPU_SHARE", TYPE_BIGINT, sizeof(int64_t), true},
-{"MEMORY_LIMIT", TYPE_VARCHAR, sizeof(StringRef), true},
-{"ENABLE_MEMORY_OVERCOMMIT", TYPE_VARCHAR, sizeof(StringRef), true},
+{"MIN_CPU_PERCENT", TYPE_BIGINT, sizeof(int64_t), true},
+{"MAX_CPU_PERCENT", TYPE_BIGINT, sizeof(int64_t), true},
+{"MIN_MEMORY_PERCENT", TYPE_BIGINT, sizeof(int64_t), true},
+{"MAX_MEMORY_PERCENT", TYPE_BIGINT, sizeof(int64_t), true},
Review Comment:
In an 1024G memoy's node, 1% means 10G memory, 0.5% means 5G memory, it's
not a small waste , maybe double is better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
wangbo commented on code in PR #54243:
URL: https://github.com/apache/doris/pull/54243#discussion_r2253937087
##
be/src/agent/cgroup_cpu_ctl.cpp:
##
@@ -404,13 +419,14 @@ Status CgroupV2CpuCtl::init() {
}
Status CgroupV2CpuCtl::modify_cg_cpu_hard_limit_no_lock(int cpu_hard_limit) {
-if (cpu_hard_limit <= 0) {
-return Status::InternalError("cpu hard limit must be greater
than 0");
-}
-std::string value = "";
-uint64_t int_val = _cpu_cfs_period_us * _cpu_core_num * cpu_hard_limit /
100;
-value = std::to_string(int_val) + " 10";
-std::string msg = "modify cpu.max to [" + value + "]";
+// If cpu_hard_limit is 0, we set the cpu.max to 1000 10.
Review Comment:
set cgroup's cpu limit default value 'max 10' when input value is 0 or
invalid value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
wangbo commented on code in PR #54243:
URL: https://github.com/apache/doris/pull/54243#discussion_r2253934966
##
be/src/agent/cgroup_cpu_ctl.cpp:
##
@@ -340,10 +353,12 @@ Status
CgroupV1CpuCtl::modify_cg_cpu_soft_limit_no_lock(int cpu_shares) {
}
Status CgroupV1CpuCtl::modify_cg_cpu_hard_limit_no_lock(int cpu_hard_limit) {
-if (cpu_hard_limit <= 0) {
-return Status::InternalError("cpu hard limit must be greater
than 0");
+// If cpu_hard_limit == 0, we do not know the actual behavior of CGroup
+// just set it to 1% of the cpu core.
+uint64_t val = _cpu_cfs_period_us / 100;
Review Comment:
we'd better set cpu_hard_limit=-1 when input value is 0 or an invalid value.
it's cgroup's default value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
doris-robot commented on PR #54243: URL: https://github.com/apache/doris/pull/54243#issuecomment-3153771191 # Cloud UT Coverage Report Increment line coverage ` ` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/6d205c98642c9d798a10102958833828db874b8a_6d205c98642c9d798a10102958833828db874b8a_cloud/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/6d205c98642c9d798a10102958833828db874b8a_6d205c98642c9d798a10102958833828db874b8a_cloud/report/index.html) | Category | Coverage | |---|| | Function Coverage | 81.16% (1361/1677) | | Line Coverage | 65.80% (23508/35725) | | Region Coverage | 67.01% (11677/17427) | | Branch Coverage | 56.63% (6117/10802) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
yiguolei commented on PR #54243: URL: https://github.com/apache/doris/pull/54243#issuecomment-3153711634 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [workloadgroup](improvement) using min/max cpu and memory percent to unify soft and hard limit [doris]
Thearas commented on PR #54243: URL: https://github.com/apache/doris/pull/54243#issuecomment-3148259937 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
