Re: [PR] [improvement](compaction) split time series compaction score metric [doris]

2026-08-11 Thread via GitHub


Yukang-Lian merged PR #63900:
URL: https://github.com/apache/doris/pull/63900


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [improvement](compaction) split time series compaction score metric [doris]

2026-08-11 Thread via GitHub


hello-stephen commented on PR #63900:
URL: https://github.com/apache/doris/pull/63900#issuecomment-5250465162

   skip check_coverage


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [improvement](compaction) split time series compaction score metric [doris]

2026-08-11 Thread via GitHub


github-actions[bot] commented on PR #63900:
URL: https://github.com/apache/doris/pull/63900#issuecomment-5250400865

   PR approved by anyone and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [improvement](compaction) split time series compaction score metric [doris]

2026-08-11 Thread via GitHub


github-actions[bot] commented on PR #63900:
URL: https://github.com/apache/doris/pull/63900#issuecomment-5250250155

   PR approved by at least one committer and no changes requested.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [improvement](compaction) split time series compaction score metric [doris]

2026-08-10 Thread via GitHub


hello-stephen commented on PR #63900:
URL: https://github.com/apache/doris/pull/63900#issuecomment-5237764292

   
   
   ClickBench: Total hot run time: 23.79 s
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit 9e37d18a4604e78180cb7aa3da637d6ff7c14472, 
data reload: false
   
   query1   0.010.010.00
   query2   0.100.050.05
   query3   0.250.140.14
   query4   1.610.140.14
   query5   0.240.220.21
   query6   1.160.800.83
   query7   0.040.010.00
   query8   0.060.040.03
   query9   0.380.300.31
   query10  0.590.560.54
   query11  0.190.140.13
   query12  0.180.150.14
   query13  0.450.450.48
   query14  1.010.981.00
   query15  0.610.580.58
   query16  0.310.330.34
   query17  1.071.181.08
   query18  0.210.200.19
   query19  2.051.941.97
   query20  0.020.010.01
   query21  15.47   0.200.13
   query22  4.900.060.05
   query23  16.15   0.300.12
   query24  2.970.420.32
   query25  0.110.050.04
   query26  0.740.200.14
   query27  0.040.030.04
   query28  3.570.740.33
   query29  12.47   4.083.22
   query30  0.270.150.15
   query31  2.760.550.32
   query32  3.210.580.48
   query33  3.183.193.15
   query34  15.49   3.983.25
   query35  3.233.233.25
   query36  0.570.440.43
   query37  0.090.070.07
   query38  0.050.040.03
   query39  0.030.040.03
   query40  0.170.150.15
   query41  0.080.030.03
   query42  0.040.020.02
   query43  0.040.030.04
   Total cold run time: 96.17 s
   Total hot run time: 23.79 s
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [improvement](compaction) split time series compaction score metric [doris]

2026-08-10 Thread via GitHub


hello-stephen commented on PR #63900:
URL: https://github.com/apache/doris/pull/63900#issuecomment-5237718928

   
   
   TPC-DS: Total hot run time: 158255 ms
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit 9e37d18a4604e78180cb7aa3da637d6ff7c14472, 
data reload: false
   
   query5   4306579 444 444
   query6   466 230 204 204
   query7   5006565 343 343
   query8   318 169 161 161
   query9   8765401139663966
   query10  491 353 306 306
   query11  5849218220082008
   query12  156 100 95  95
   query13  1250635 418 418
   query14  6079426939403940
   query14_13781377637873776
   query15  202 201 179 179
   query16  972 515 472 472
   query17  934 734 571 571
   query18  2429490 352 352
   query19  209 192 151 151
   query20  113 105 108 105
   query21  232 156 134 134
   query22  13099   12998   12832   12832
   query23  15856   14909   14632   14632
   query23_114634   14557   14648   14557
   query24  7655172312391239
   query24_11225124012361236
   query25  567 501 380 380
   query26  1321359 217 217
   query27  2580608 398 398
   query28  4563203020282028
   query29  1065638 514 514
   query30  346 269 219 219
   query31  1179111710501050
   query32  108 63  66  63
   query33  525 304 239 239
   query34  11771101663 663
   query35  738 761 614 614
   query36  775 786 702 702
   query37  155 108 90  90
   query38  1829176616801680
   query39  825 829 807 807
   query39_1796 776 787 776
   query40  256 164 143 143
   query41  65  64  63  63
   query42  98  95  97  95
   query43  324 317 279 279
   query44  1398775 772 772
   query45  192 174 173 173
   query46  10551169715 715
   query47  1520156614651465
   query48  400 414 297 297
   query49  577 423 320 320
   query50  1062458 321 321
   query51  10412   10600   10505   10505
   query52  94  84  77  77
   query53  269 272 194 194
   query54  297 224 225 224
   query55  73  70  66  66
   query56  305 300 320 300
   query57  10151006978 978
   query58  286 255 248 248
   query59  1570161213911391
   query60  313 289 252 252
   query61  185 145 149 145
   query62  392 317 267 267
   query63  226 198 201 198
   query64  28501119877 877
   query65  3884384538123812
   query66  1818474 366 366
   query67  19868   20190   19939   19939
   query68  3187152410661066
   query69  413 288 276 276
   query70  854 771 819 771
   query71  364 345 315 315
   query72  3016268823882388
   query73  871 780 455 455
   query74  4622448042834283
   query75  2362234519881988
   query76  23361094742 742
   query77  328 370 272 272
   query78  7   10993   10466   10466
   query79  13731149744 744
   query80  1053551 448 448
   query81  504 335 281 281
   query82  632 188 140 140
   query83  410 318 297 297
   query84  322 159 135 135
   query85  988 629 521 521
   query86  369 230 219 219
   query87  1992198418431843
   query88  3721278627582758
   query89  398 323 276 276
   query90  1787205 195 195
   query91  201 189 164 164
   query92  62  59  55  55
   query93  16661684928 928
   query94  619 344 331 331
   query95  822 519 459 459
   query96  1110808 379 379
   query97  2472247123522352
   query98  198 187 185 185
   query99  745 750 608 608
   Total cold run time: 244558 ms
   Total hot run time: 158255 ms
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go 

Re: [PR] [improvement](compaction) split time series compaction score metric [doris]

2026-08-10 Thread via GitHub


hello-stephen commented on PR #63900:
URL: https://github.com/apache/doris/pull/63900#issuecomment-5237616427

   
   
   TPC-H: Total hot run time: 29282 ms
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit 9e37d18a4604e78180cb7aa3da637d6ff7c14472, 
data reload: false
   
   -- Round 1 --
   
   q1   17592   405639573957
   q2   2048321 210 210
   q3   10281   1385821 821
   q4   4682465 336 336
   q5   7490828 562 562
   q6   175 166 135 135
   q7   737 813 590 590
   q8   9328164916731649
   q9   5392411441174114
   q10  6714164313711371
   q11  504 348 331 331
   q12  708 567 449 449
   q13  18141   335327772777
   q14  258 263 246 246
   q15  q16 728 733 657 657
   q17  987 921 994 921
   q18  6571560755705570
   q19  1160121010531053
   q20  767 677 569 569
   q21  5935282026402640
   q22  440 380 324 324
   Total cold run time: 100638 ms
   Total hot run time: 29282 ms
   
   - Round 2, with runtime_filter_mode=off -
   
   q1   4936465145324532
   q2   285 346 211 211
   q3   4924531946104610
   q4   2213226614481448
   q5   4519465743954395
   q6   232 176 125 125
   q7   1813171415271527
   q8   2306200820532008
   q9   7176680566896689
   q10  4258423238303830
   q11  511 371 338 338
   q12  698 717 485 485
   q13  2932320227422742
   q14  281 275 248 248
   q15  q16 655 680 600 600
   q17  1231120512191205
   q18  12025   10993   11839   10993
   q19  1087108710521052
   q20  2189217519081908
   q21  5269446245634462
   q22  506 463 393 393
   Total cold run time: 60046 ms
   Total hot run time: 53801 ms
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [improvement](compaction) split time series compaction score metric [doris]

2026-08-09 Thread via GitHub


Yukang-Lian commented on PR #63900:
URL: https://github.com/apache/doris/pull/63900#issuecomment-5235521640

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [improvement](compaction) split time series compaction score metric [doris]

2026-08-06 Thread via GitHub


hello-stephen commented on PR #63900:
URL: https://github.com/apache/doris/pull/63900#issuecomment-5204540271

   
   
   ClickBench: Total hot run time: 24 s
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
   ClickBench test result on commit d9faa0fad7a0160c67f206ce45fa99a6f21bb095, 
data reload: false
   
   query1   0.000.000.01
   query2   0.090.040.05
   query3   0.250.130.13
   query4   1.600.130.14
   query5   0.230.220.22
   query6   1.160.800.81
   query7   0.030.010.01
   query8   0.060.030.04
   query9   0.380.330.31
   query10  0.540.580.57
   query11  0.190.140.13
   query12  0.190.140.14
   query13  0.450.460.47
   query14  0.990.990.98
   query15  0.610.580.61
   query16  0.310.320.32
   query17  1.121.101.10
   query18  0.210.200.20
   query19  2.052.001.96
   query20  0.020.010.01
   query21  15.44   0.190.13
   query22  4.960.050.06
   query23  16.10   0.300.12
   query24  2.980.430.36
   query25  0.110.050.03
   query26  0.770.200.15
   query27  0.030.040.04
   query28  3.580.770.34
   query29  12.50   3.993.17
   query30  0.280.150.15
   query31  2.770.550.31
   query32  3.220.580.50
   query33  3.283.213.21
   query34  15.50   4.003.33
   query35  3.233.203.21
   query36  0.560.410.43
   query37  0.090.070.07
   query38  0.060.040.04
   query39  0.040.030.04
   query40  0.170.170.15
   query41  0.090.030.03
   query42  0.040.030.03
   query43  0.040.030.04
   Total cold run time: 96.32 s
   Total hot run time: 24 s
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [improvement](compaction) split time series compaction score metric [doris]

2026-08-06 Thread via GitHub


hello-stephen commented on PR #63900:
URL: https://github.com/apache/doris/pull/63900#issuecomment-5204493438

   
   
   TPC-DS: Total hot run time: 166284 ms
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit d9faa0fad7a0160c67f206ce45fa99a6f21bb095, 
data reload: false
   
   query5   4328592 451 451
   query6   464 226 209 209
   query7   4864605 354 354
   query8   323 164 147 147
   query9   8769407140394039
   query10  465 359 295 295
   query11  5815215320112011
   query12  151 96  99  96
   query13  1260611 456 456
   query14  6163429039533953
   query14_13767377937913779
   query15  201 200 181 181
   query16  955 455 437 437
   query17  908 699 541 541
   query18  2476464 323 323
   query19  202 179 141 141
   query20  101 101 99  99
   query21  228 158 135 135
   query22  13104   12980   12806   12806
   query23  15846   15073   14547   14547
   query23_114727   14806   14726   14726
   query24  7539173412241224
   query24_11265121612511216
   query25  541 411 348 348
   query26  1327365 222 222
   query27  2610596 380 380
   query28  4613206920492049
   query29  1065597 470 470
   query30  344 262 222 222
   query31  1185112110571057
   query32  105 60  58  58
   query33  513 300 237 237
   query34  11751162636 636
   query35  711 745 626 626
   query36  773 774 704 704
   query37  155 107 90  90
   query38  1827177017191719
   query39  823 819 805 805
   query39_1810 786 781 781
   query40  252 162 146 146
   query41  66  65  65  65
   query42  102 95  91  91
   query43  317 325 278 278
   query44  1452779 784 779
   query45  195 177 170 170
   query46  11001181704 704
   query47  1590156614941494
   query48  412 420 315 315
   query49  591 422 308 308
   query50  1104425 334 334
   query51  10637   10562   10772   10562
   query52  90  89  87  87
   query53  265 287 208 208
   query54  301 257 233 233
   query55  79  80  72  72
   query56  308 325 302 302
   query57  10291049931 931
   query58  318 275 272 272
   query59  1555164813851385
   query60  315 312 262 262
   query61  171 174 176 174
   query62  405 323 274 274
   query63  235 200 210 200
   query64  30421165984 984
   query65  3917383637993799
   query66  1838505 375 375
   query67  28207   28091   28009   28009
   query68  32261663956 956
   query69  415 309 270 270
   query70  913 738 777 738
   query71  363 338 319 319
   query72  3019259223052305
   query73  873 802 437 437
   query74  4615450242824282
   query75  2363236019871987
   query76  23261195745 745
   query77  357 364 273 273
   query78  11312   11225   10506   10506
   query79  14841174695 695
   query80  1240551 461 461
   query81  538 326 281 281
   query82  639 173 142 142
   query83  374 319 293 293
   query84  334 159 134 134
   query85  981 603 517 517
   query86  406 238 212 212
   query87  1963196318411841
   query88  3772284728032803
   query89  386 325 283 283
   query90  1991198 191 191
   query91  207 195 158 158
   query92  63  61  56  56
   query93  16601537937 937
   query94  722 367 317 317
   query95  787 573 472 472
   query96  1080785 365 365
   query97  2468244223752375
   query98  200 190 184 184
   query99  752 734 608 608
   Total cold run time: 254291 ms
   Total hot run time: 166284 ms
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go 

Re: [PR] [improvement](compaction) split time series compaction score metric [doris]

2026-08-06 Thread via GitHub


hello-stephen commented on PR #63900:
URL: https://github.com/apache/doris/pull/63900#issuecomment-5204393107

   
   
   TPC-H: Total hot run time: 29083 ms
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit d9faa0fad7a0160c67f206ce45fa99a6f21bb095, 
data reload: false
   
   -- Round 1 --
   
   q1   17898   403339243924
   q2   2010334 202 202
   q3   10300   1432810 810
   q4   4679477 345 345
   q5   7549843 554 554
   q6   184 166 139 139
   q7   769 800 602 602
   q8   9337150616321506
   q9   5391412740824082
   q10  6729163213761376
   q11  508 350 322 322
   q12  732 570 446 446
   q13  18112   330127462746
   q14  271 257 239 239
   q15  q16 732 732 664 664
   q17  967 1000928 928
   q18  6662560855795579
   q19  1178127610581058
   q20  804 688 571 571
   q21  6226289326702670
   q22  463 388 320 320
   Total cold run time: 101501 ms
   Total hot run time: 29083 ms
   
   - Round 2, with runtime_filter_mode=off -
   
   q1   5012469646644664
   q2   289 334 217 217
   q3   5019581646514651
   q4   2192227514381438
   q5   472844214421
   q6   238 179 136 136
   q7   1843171815211521
   q8   2369205920082008
   q9   7045675767416741
   q10  4282419738053805
   q11  520 375 344 344
   q12  699 714 493 493
   q13  2947330727242724
   q14  266 279 250 250
   q15  q16 661 683 596 596
   q17  1246121912101210
   q18  12142   10986   11775   10986
   q19  1094106010681060
   q20  2187220418931893
   q21  5301455145274527
   q22  537 453 390 390
   Total cold run time: 60617 ms
   Total hot run time: 54075 ms
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [improvement](compaction) split time series compaction score metric [doris]

2026-08-06 Thread via GitHub


hello-stephen commented on PR #63900:
URL: https://github.com/apache/doris/pull/63900#issuecomment-5204124151

   # BE Regression && UT Coverage Report
   Increment line coverage `97.46% (115/118)` :tada:
   
   [Increment coverage 
report](http://coverage.selectdb-in.cc/coverage/63900_d9faa0fad7a0160c67f206ce45fa99a6f21bb095_merge/increment_report/index.html)
   [Complete coverage 
report](http://coverage.selectdb-in.cc/coverage/63900_d9faa0fad7a0160c67f206ce45fa99a6f21bb095_merge/report/index.html)
   | Category  | Coverage   |
   |---||
   | Function Coverage | 75.96% (32287/42508) |
   | Line Coverage | 60.73% (361398/595136) |
   | Region Coverage   | 57.36% (303916/529836) |
   | Branch Coverage   | 58.81% (137295/233443) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [improvement](compaction) split time series compaction score metric [doris]

2026-08-06 Thread via GitHub


hello-stephen commented on PR #63900:
URL: https://github.com/apache/doris/pull/63900#issuecomment-5204012804

   # BE UT Coverage Report
   Increment line coverage `97.08% (133/137)` :tada:
   
   [Increment coverage 
report](http://coverage.selectdb-in.cc/coverage/d9faa0fad7a0160c67f206ce45fa99a6f21bb095_d9faa0fad7a0160c67f206ce45fa99a6f21bb095/increment_report/index.html)
   [Complete coverage 
report](http://coverage.selectdb-in.cc/coverage/d9faa0fad7a0160c67f206ce45fa99a6f21bb095_d9faa0fad7a0160c67f206ce45fa99a6f21bb095/report/index.html)
   | Category  | Coverage   |
   |---||
   | Function Coverage | 59.81% (26093/43627) |
   | Line Coverage | 44.14% (264737/599741) |
   | Region Coverage   | 39.88% (210813/528582) |
   | Branch Coverage   | 41.31% (96550/233748) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [improvement](compaction) split time series compaction score metric [doris]

2026-08-06 Thread via GitHub


Yukang-Lian commented on PR #63900:
URL: https://github.com/apache/doris/pull/63900#issuecomment-5202048699

   run performance


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [improvement](compaction) split time series compaction score metric [doris]

2026-08-06 Thread via GitHub


Yukang-Lian commented on PR #63900:
URL: https://github.com/apache/doris/pull/63900#issuecomment-5202047708

   run beut


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [improvement](compaction) split time series compaction score metric [doris]

2026-08-05 Thread via GitHub


Yukang-Lian commented on PR #63900:
URL: https://github.com/apache/doris/pull/63900#issuecomment-5189555014

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [improvement](compaction) split time series compaction score metric [doris]

2026-06-24 Thread via GitHub


github-actions[bot] commented on code in PR #63900:
URL: https://github.com/apache/doris/pull/63900#discussion_r3465299209


##
be/src/storage/olap_server.cpp:
##
@@ -915,38 +920,59 @@ std::vector 
StorageEngine::_generate_compaction_tasks(
 
 // Even if need_pick_tablet is false, we still need to call 
find_best_tablet_to_compaction(),
 // So that we can update the max_compaction_score metric.
-if (!data_dir->reach_capacity_limit(0)) {
-uint32_t disk_max_score = 0;
-auto tablet_contexts = 
compaction_registry_snapshot.pick_topn_tablets_for_compaction(
-_tablet_manager.get(), data_dir, compaction_type,
-cumulative_compaction_policies, &disk_max_score);
-int concurrent_num = get_concurrent_per_disk(
-disk_max_score, disk_compaction_slot_num(*data_dir, 
compaction_type));
-need_pick_tablet = need_generate_compaction_tasks(
-executing_task_num, concurrent_num, compaction_type,
-!compaction_registry_snapshot.has_compaction_task(
-data_dir, CompactionType::CUMULATIVE_COMPACTION));
-for (const auto& context : tablet_contexts) {
-if (context.tablet != nullptr) {
-if (need_pick_tablet) {
-tablet_compaction_contexts.emplace_back(context);
-}
-max_compaction_score = std::max(max_compaction_score, 
disk_max_score);
+if (data_dir->reach_capacity_limit(0)) {
+skipped_capacity_limited_dir = true;
+continue;
+}
+
+CompactionScoreStats disk_score_stats;
+auto tablet_contexts = 
compaction_registry_snapshot.pick_topn_tablets_for_compaction(
+_tablet_manager.get(), data_dir, compaction_type, 
cumulative_compaction_policies,
+&disk_score_stats);
+max_score_stats.scanned = max_score_stats.scanned || 
disk_score_stats.scanned;
+max_score_stats.max_score = std::max(max_score_stats.max_score, 
disk_score_stats.max_score);
+max_score_stats.size_based_max_score = 
std::max(max_score_stats.size_based_max_score,
+
disk_score_stats.size_based_max_score);
+max_score_stats.time_series_max_score = 
std::max(max_score_stats.time_series_max_score,
+ 
disk_score_stats.time_series_max_score);
+int concurrent_num = get_concurrent_per_disk(
+disk_score_stats.max_score, 
disk_compaction_slot_num(*data_dir, compaction_type));
+need_pick_tablet = need_generate_compaction_tasks(
+executing_task_num, concurrent_num, compaction_type,
+!compaction_registry_snapshot.has_compaction_task(
+data_dir, CompactionType::CUMULATIVE_COMPACTION));
+for (const auto& context : tablet_contexts) {
+if (context.tablet != nullptr) {
+if (need_pick_tablet) {
+tablet_compaction_contexts.emplace_back(context);
 }
 }
 }
 }
 
-if (max_compaction_score > 0) {
-if (compaction_type == CompactionType::BASE_COMPACTION) {
+if (max_score_stats.scanned) {
+if (compaction_type == CompactionType::BASE_COMPACTION && 
max_score_stats.max_score > 0) {
 
DorisMetrics::instance()->tablet_base_max_compaction_score->set_value(
-max_compaction_score);
+max_score_stats.max_score);
 } else if (compaction_type == CompactionType::CUMULATIVE_COMPACTION) {
-
DorisMetrics::instance()->tablet_cumulative_max_compaction_score->set_value(
-max_compaction_score);
-} else if (compaction_type == CompactionType::BINLOG_COMPACTION) {
+auto update_policy_score = [skipped_capacity_limited_dir, 
check_score](IntGauge* metric,
+   
int64_t score) {
+if (skipped_capacity_limited_dir) {
+if (score > metric->value()) {
+metric->set_value(score);
+}
+} else if (check_score || score > 0) {
+metric->set_value(score);

Review Comment:
   `tablet_cumulative_max_compaction_score` is an existing exported BE metric, 
but after this change it no longer means "max cumulative compaction score"; it 
only tracks size-based cumulative scores while time-series backlog moves to 
`tablet_time_series_max_compaction_score`. In the base code this same metric 
was fed from the single aggregate cumulative `max_compaction_score`, so 
existing dashboards/alerts will silently miss time-series tablets after upgrade 
unless they know to query both metrics. Please either keep the 

Re: [PR] [improvement](compaction) split time series compaction score metric [doris]

2026-06-24 Thread via GitHub


Yukang-Lian commented on PR #63900:
URL: https://github.com/apache/doris/pull/63900#issuecomment-4786796579

   /review


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [improvement](compaction) split time series compaction score metric [doris]

2026-06-08 Thread via GitHub


hello-stephen commented on PR #63900:
URL: https://github.com/apache/doris/pull/63900#issuecomment-4656321867

   # BE Regression && UT Coverage Report
   Increment line coverage `96.52% (111/115)` :tada:
   
   [Increment coverage 
report](http://coverage.selectdb-in.cc/coverage/63900_c45fe72d7898e11f553a077fa29f33b9f3382ba4_merge/increment_report/index.html)
   [Complete coverage 
report](http://coverage.selectdb-in.cc/coverage/63900_c45fe72d7898e11f553a077fa29f33b9f3382ba4_merge/report/index.html)
   | Category  | Coverage   |
   |---||
   | Function Coverage | 63.73% (24407/38300) |
   | Line Coverage | 47.61% (253237/531944) |
   | Region Coverage   | 44.31% (208917/471438) |
   | Branch Coverage   | 45.48% (90736/199493) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [improvement](compaction) split time series compaction score metric [doris]

2026-06-08 Thread via GitHub


Yukang-Lian commented on PR #63900:
URL: https://github.com/apache/doris/pull/63900#issuecomment-4655542938

   run nonConcurrent


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [improvement](compaction) split time series compaction score metric [doris]

2026-06-08 Thread via GitHub


Yukang-Lian commented on PR #63900:
URL: https://github.com/apache/doris/pull/63900#issuecomment-4655543661

   run cloud_p0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [improvement](compaction) split time series compaction score metric [doris]

2026-06-08 Thread via GitHub


hello-stephen commented on PR #63900:
URL: https://github.com/apache/doris/pull/63900#issuecomment-4651301444

   
   
   TPC-DS: Total hot run time: 169015 ms
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
   TPC-DS sf100 test result on commit c45fe72d7898e11f553a077fa29f33b9f3382ba4, 
data reload: false
   
   query5   4316637 484 484
   query6   468 198 179 179
   query7   4846555 302 302
   query8   377 222 207 207
   query9   8780406140434043
   query10  449 315 254 254
   query11  5844235722212221
   query12  159 104 100 100
   query13  1272622 436 436
   query14  6367538550665066
   query14_14388443144194419
   query15  205 201 183 183
   query16  1026463 437 437
   query17  958 708 588 588
   query18  2466477 355 355
   query19  206 197 164 164
   query20  116 110 106 106
   query21  225 140 120 120
   query22  13685   13662   13380   13380
   query23  17280   16614   16186   16186
   query23_116381   16273   16229   16229
   query24  7501175612871287
   query24_11308131113271311
   query25  551 459 384 384
   query26  1349307 161 161
   query27  2740543 309 309
   query28  4477201019941994
   query29  1095611 512 512
   query30  315 249 200 200
   query31  11101077955 955
   query32  163 68  60  60
   query33  516 321 258 258
   query34  11611098663 663
   query35  762 783 679 679
   query36  1358142412281228
   query37  158 117 85  85
   query38  3213314930323032
   query39  938 919 905 905
   query39_1866 906 897 897
   query40  215 124 108 108
   query41  67  67  62  62
   query42  102 94  95  94
   query43  322 319 279 279
   query44  
   query45  196 182 181 181
   query46  10991212743 743
   query47  2362239322522252
   query48  354 406 299 299
   query49  636 469 364 364
   query50  1032343 255 255
   query51  4376427242624262
   query52  87  87  77  77
   query53  242 270 196 196
   query54  275 219 201 201
   query55  76  77  69  69
   query56  234 234 211 211
   query57  1419141513301330
   query58  276 211 196 196
   query59  1578161914641464
   query60  287 259 233 233
   query61  161 160 161 160
   query62  706 642 591 591
   query63  224 192 183 183
   query64  2614793 623 623
   query65  
   query66  1831498 345 345
   query67  29711   29581   29563   29563
   query68  
   query69  423 306 268 268
   query70  958 946 919 919
   query71  293 226 213 213
   query72  3062270324072407
   query73  836 762 415 415
   query74  5107496947724772
   query75  2675258822722272
   query76  23441136761 761
   query77  364 369 289 289
   query78  12382   12297   11798   11798
   query79  13181043760 760
   query80  595 478 396 396
   query81  448 280 247 247
   query82  643 157 120 120
   query83  362 279 252 252
   query84  
   query85  900 542 448 448
   query86  367 292 298 292
   query87  3423335831923192
   query88  3631273127022702
   query89  418 389 329 329
   query90  1937177 182 177
   query91  175 161 141 141
   query92  59  61  60  60
   query93  13961421946 946
   query94  570 345 325 325
   query95  675 373 436 373
   query96  1026762 355 355
   query97  2702269925752575
   query98  214 215 204 204
   query99  1147117910231023
   Total cold run time: 250614 ms
   Total hot run time: 169015 ms
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about 

Re: [PR] [improvement](compaction) split time series compaction score metric [doris]

2026-06-08 Thread via GitHub


hello-stephen commented on PR #63900:
URL: https://github.com/apache/doris/pull/63900#issuecomment-4651206913

   
   
   TPC-H: Total hot run time: 29207 ms
   
   ```
   machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
   scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
   Tpch sf100 test result on commit c45fe72d7898e11f553a077fa29f33b9f3382ba4, 
data reload: false
   
   -- Round 1 --
   orders   Doris   NULLNULL0   0   0   NULL0   
NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8   
NULLNULL
   
   q1   17735   409940214021
   q2   q3  10791   1470854 854
   q4   4686482 345 345
   q5   7628910 589 589
   q6   193 182 142 142
   q7   804 864 650 650
   q8   9374169916041604
   q9   5811455644734473
   q10  6758182615301530
   q11  446 282 257 257
   q12  634 421 300 300
   q13  18152   343228082808
   q14  273 266 251 251
   q15  q16 834 781 712 712
   q17  949 892 935 892
   q18  6917592656635663
   q19  1365127110811081
   q20  547 409 253 253
   q21  5838263924702470
   q22  431 366 312 312
   Total cold run time: 100166 ms
   Total hot run time: 29207 ms
   
   - Round 2, with runtime_filter_mode=off -
   orders   Doris   NULLNULL15000   42  6422171781  
NULL22778155NULLNULL2023-12-26 18:27:23 2023-12-26 
18:42:55 NULLutf-8   NULLNULL
   
   q1   4380435243494349
   q2   q3  4544497143484348
   q4   2090219414051405
   q5   4504435643854356
   q6   233 177 133 133
   q7   1743186919101869
   q8   2658223022302230
   q9   8167837779897989
   q10  4837473942794279
   q11  572 430 384 384
   q12  766 799 549 549
   q13  3374366329682968
   q14  295 314 262 262
   q15  q16 741 750 659 659
   q17  1394134813691348
   q18  8114738871787178
   q19  1186117811351135
   q20  2215222819691969
   q21  5319462445094509
   q22  537 465 409 409
   Total cold run time: 57669 ms
   Total hot run time: 52328 ms
   ```
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [improvement](compaction) split time series compaction score metric [doris]

2026-06-08 Thread via GitHub


hello-stephen commented on PR #63900:
URL: https://github.com/apache/doris/pull/63900#issuecomment-4648914018

   # BE Regression && UT Coverage Report
   Increment line coverage `96.52% (111/115)` :tada:
   
   [Increment coverage 
report](http://coverage.selectdb-in.cc/coverage/63900_c45fe72d7898e11f553a077fa29f33b9f3382ba4_merge/increment_report/index.html)
   [Complete coverage 
report](http://coverage.selectdb-in.cc/coverage/63900_c45fe72d7898e11f553a077fa29f33b9f3382ba4_merge/report/index.html)
   | Category  | Coverage   |
   |---||
   | Function Coverage | 63.73% (24407/38300) |
   | Line Coverage | 47.61% (253237/531944) |
   | Region Coverage   | 44.31% (208917/471438) |
   | Branch Coverage   | 45.48% (90736/199493) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [improvement](compaction) split time series compaction score metric [doris]

2026-06-08 Thread via GitHub


hello-stephen commented on PR #63900:
URL: https://github.com/apache/doris/pull/63900#issuecomment-4647661716

   # BE UT Coverage Report
   Increment line coverage `93.28% (125/134)` :tada:
   
   [Increment coverage 
report](http://coverage.selectdb-in.cc/coverage/c45fe72d7898e11f553a077fa29f33b9f3382ba4_c45fe72d7898e11f553a077fa29f33b9f3382ba4/increment_report/index.html)
   [Complete coverage 
report](http://coverage.selectdb-in.cc/coverage/c45fe72d7898e11f553a077fa29f33b9f3382ba4_c45fe72d7898e11f553a077fa29f33b9f3382ba4/report/index.html)
   | Category  | Coverage   |
   |---||
   | Function Coverage | 53.96% (21156/39209) |
   | Line Coverage | 37.67% (201533/535009) |
   | Region Coverage   | 33.68% (158017/469239) |
   | Branch Coverage   | 34.73% (69189/199212) |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [improvement](compaction) split time series compaction score metric [doris]

2026-06-07 Thread via GitHub


Yukang-Lian commented on PR #63900:
URL: https://github.com/apache/doris/pull/63900#issuecomment-4646088748

   run buildall


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]



Re: [PR] [improvement](compaction) split time series compaction score metric [doris]

2026-05-29 Thread via GitHub


hello-stephen commented on PR #63900:
URL: https://github.com/apache/doris/pull/63900#issuecomment-4573498013

   
   Thank you for your contribution to Apache Doris.
   Don't know what should be done next? See [How to process your 
PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR).
   
   Please clearly describe your PR:
   1. What problem was fixed (it's best to include specific error reporting 
information). How it was fixed.
   2. Which behaviors were modified. What was the previous behavior, what is it 
now, why was it modified, and what possible impacts might there be.
   3. What features were added. Why was this function added?
   4. Which code was refactored and why was this part of the code refactored?
   5. Which functions were optimized and what is the difference before and 
after the optimization?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]