Re: [PR] [improvement](compaction) split time series compaction score metric [doris]
Yukang-Lian merged PR #63900: URL: https://github.com/apache/doris/pull/63900 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](compaction) split time series compaction score metric [doris]
hello-stephen commented on PR #63900: URL: https://github.com/apache/doris/pull/63900#issuecomment-5250465162 skip check_coverage -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](compaction) split time series compaction score metric [doris]
github-actions[bot] commented on PR #63900: URL: https://github.com/apache/doris/pull/63900#issuecomment-5250400865 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](compaction) split time series compaction score metric [doris]
github-actions[bot] commented on PR #63900: URL: https://github.com/apache/doris/pull/63900#issuecomment-5250250155 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](compaction) split time series compaction score metric [doris]
hello-stephen commented on PR #63900: URL: https://github.com/apache/doris/pull/63900#issuecomment-5237764292 ClickBench: Total hot run time: 23.79 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 9e37d18a4604e78180cb7aa3da637d6ff7c14472, data reload: false query1 0.010.010.00 query2 0.100.050.05 query3 0.250.140.14 query4 1.610.140.14 query5 0.240.220.21 query6 1.160.800.83 query7 0.040.010.00 query8 0.060.040.03 query9 0.380.300.31 query10 0.590.560.54 query11 0.190.140.13 query12 0.180.150.14 query13 0.450.450.48 query14 1.010.981.00 query15 0.610.580.58 query16 0.310.330.34 query17 1.071.181.08 query18 0.210.200.19 query19 2.051.941.97 query20 0.020.010.01 query21 15.47 0.200.13 query22 4.900.060.05 query23 16.15 0.300.12 query24 2.970.420.32 query25 0.110.050.04 query26 0.740.200.14 query27 0.040.030.04 query28 3.570.740.33 query29 12.47 4.083.22 query30 0.270.150.15 query31 2.760.550.32 query32 3.210.580.48 query33 3.183.193.15 query34 15.49 3.983.25 query35 3.233.233.25 query36 0.570.440.43 query37 0.090.070.07 query38 0.050.040.03 query39 0.030.040.03 query40 0.170.150.15 query41 0.080.030.03 query42 0.040.020.02 query43 0.040.030.04 Total cold run time: 96.17 s Total hot run time: 23.79 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](compaction) split time series compaction score metric [doris]
hello-stephen commented on PR #63900: URL: https://github.com/apache/doris/pull/63900#issuecomment-5237718928 TPC-DS: Total hot run time: 158255 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 9e37d18a4604e78180cb7aa3da637d6ff7c14472, data reload: false query5 4306579 444 444 query6 466 230 204 204 query7 5006565 343 343 query8 318 169 161 161 query9 8765401139663966 query10 491 353 306 306 query11 5849218220082008 query12 156 100 95 95 query13 1250635 418 418 query14 6079426939403940 query14_13781377637873776 query15 202 201 179 179 query16 972 515 472 472 query17 934 734 571 571 query18 2429490 352 352 query19 209 192 151 151 query20 113 105 108 105 query21 232 156 134 134 query22 13099 12998 12832 12832 query23 15856 14909 14632 14632 query23_114634 14557 14648 14557 query24 7655172312391239 query24_11225124012361236 query25 567 501 380 380 query26 1321359 217 217 query27 2580608 398 398 query28 4563203020282028 query29 1065638 514 514 query30 346 269 219 219 query31 1179111710501050 query32 108 63 66 63 query33 525 304 239 239 query34 11771101663 663 query35 738 761 614 614 query36 775 786 702 702 query37 155 108 90 90 query38 1829176616801680 query39 825 829 807 807 query39_1796 776 787 776 query40 256 164 143 143 query41 65 64 63 63 query42 98 95 97 95 query43 324 317 279 279 query44 1398775 772 772 query45 192 174 173 173 query46 10551169715 715 query47 1520156614651465 query48 400 414 297 297 query49 577 423 320 320 query50 1062458 321 321 query51 10412 10600 10505 10505 query52 94 84 77 77 query53 269 272 194 194 query54 297 224 225 224 query55 73 70 66 66 query56 305 300 320 300 query57 10151006978 978 query58 286 255 248 248 query59 1570161213911391 query60 313 289 252 252 query61 185 145 149 145 query62 392 317 267 267 query63 226 198 201 198 query64 28501119877 877 query65 3884384538123812 query66 1818474 366 366 query67 19868 20190 19939 19939 query68 3187152410661066 query69 413 288 276 276 query70 854 771 819 771 query71 364 345 315 315 query72 3016268823882388 query73 871 780 455 455 query74 4622448042834283 query75 2362234519881988 query76 23361094742 742 query77 328 370 272 272 query78 7 10993 10466 10466 query79 13731149744 744 query80 1053551 448 448 query81 504 335 281 281 query82 632 188 140 140 query83 410 318 297 297 query84 322 159 135 135 query85 988 629 521 521 query86 369 230 219 219 query87 1992198418431843 query88 3721278627582758 query89 398 323 276 276 query90 1787205 195 195 query91 201 189 164 164 query92 62 59 55 55 query93 16661684928 928 query94 619 344 331 331 query95 822 519 459 459 query96 1110808 379 379 query97 2472247123522352 query98 198 187 185 185 query99 745 750 608 608 Total cold run time: 244558 ms Total hot run time: 158255 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go
Re: [PR] [improvement](compaction) split time series compaction score metric [doris]
hello-stephen commented on PR #63900: URL: https://github.com/apache/doris/pull/63900#issuecomment-5237616427 TPC-H: Total hot run time: 29282 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 9e37d18a4604e78180cb7aa3da637d6ff7c14472, data reload: false -- Round 1 -- q1 17592 405639573957 q2 2048321 210 210 q3 10281 1385821 821 q4 4682465 336 336 q5 7490828 562 562 q6 175 166 135 135 q7 737 813 590 590 q8 9328164916731649 q9 5392411441174114 q10 6714164313711371 q11 504 348 331 331 q12 708 567 449 449 q13 18141 335327772777 q14 258 263 246 246 q15 q16 728 733 657 657 q17 987 921 994 921 q18 6571560755705570 q19 1160121010531053 q20 767 677 569 569 q21 5935282026402640 q22 440 380 324 324 Total cold run time: 100638 ms Total hot run time: 29282 ms - Round 2, with runtime_filter_mode=off - q1 4936465145324532 q2 285 346 211 211 q3 4924531946104610 q4 2213226614481448 q5 4519465743954395 q6 232 176 125 125 q7 1813171415271527 q8 2306200820532008 q9 7176680566896689 q10 4258423238303830 q11 511 371 338 338 q12 698 717 485 485 q13 2932320227422742 q14 281 275 248 248 q15 q16 655 680 600 600 q17 1231120512191205 q18 12025 10993 11839 10993 q19 1087108710521052 q20 2189217519081908 q21 5269446245634462 q22 506 463 393 393 Total cold run time: 60046 ms Total hot run time: 53801 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](compaction) split time series compaction score metric [doris]
Yukang-Lian commented on PR #63900: URL: https://github.com/apache/doris/pull/63900#issuecomment-5235521640 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](compaction) split time series compaction score metric [doris]
hello-stephen commented on PR #63900: URL: https://github.com/apache/doris/pull/63900#issuecomment-5204540271 ClickBench: Total hot run time: 24 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit d9faa0fad7a0160c67f206ce45fa99a6f21bb095, data reload: false query1 0.000.000.01 query2 0.090.040.05 query3 0.250.130.13 query4 1.600.130.14 query5 0.230.220.22 query6 1.160.800.81 query7 0.030.010.01 query8 0.060.030.04 query9 0.380.330.31 query10 0.540.580.57 query11 0.190.140.13 query12 0.190.140.14 query13 0.450.460.47 query14 0.990.990.98 query15 0.610.580.61 query16 0.310.320.32 query17 1.121.101.10 query18 0.210.200.20 query19 2.052.001.96 query20 0.020.010.01 query21 15.44 0.190.13 query22 4.960.050.06 query23 16.10 0.300.12 query24 2.980.430.36 query25 0.110.050.03 query26 0.770.200.15 query27 0.030.040.04 query28 3.580.770.34 query29 12.50 3.993.17 query30 0.280.150.15 query31 2.770.550.31 query32 3.220.580.50 query33 3.283.213.21 query34 15.50 4.003.33 query35 3.233.203.21 query36 0.560.410.43 query37 0.090.070.07 query38 0.060.040.04 query39 0.040.030.04 query40 0.170.170.15 query41 0.090.030.03 query42 0.040.030.03 query43 0.040.030.04 Total cold run time: 96.32 s Total hot run time: 24 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](compaction) split time series compaction score metric [doris]
hello-stephen commented on PR #63900: URL: https://github.com/apache/doris/pull/63900#issuecomment-5204493438 TPC-DS: Total hot run time: 166284 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit d9faa0fad7a0160c67f206ce45fa99a6f21bb095, data reload: false query5 4328592 451 451 query6 464 226 209 209 query7 4864605 354 354 query8 323 164 147 147 query9 8769407140394039 query10 465 359 295 295 query11 5815215320112011 query12 151 96 99 96 query13 1260611 456 456 query14 6163429039533953 query14_13767377937913779 query15 201 200 181 181 query16 955 455 437 437 query17 908 699 541 541 query18 2476464 323 323 query19 202 179 141 141 query20 101 101 99 99 query21 228 158 135 135 query22 13104 12980 12806 12806 query23 15846 15073 14547 14547 query23_114727 14806 14726 14726 query24 7539173412241224 query24_11265121612511216 query25 541 411 348 348 query26 1327365 222 222 query27 2610596 380 380 query28 4613206920492049 query29 1065597 470 470 query30 344 262 222 222 query31 1185112110571057 query32 105 60 58 58 query33 513 300 237 237 query34 11751162636 636 query35 711 745 626 626 query36 773 774 704 704 query37 155 107 90 90 query38 1827177017191719 query39 823 819 805 805 query39_1810 786 781 781 query40 252 162 146 146 query41 66 65 65 65 query42 102 95 91 91 query43 317 325 278 278 query44 1452779 784 779 query45 195 177 170 170 query46 11001181704 704 query47 1590156614941494 query48 412 420 315 315 query49 591 422 308 308 query50 1104425 334 334 query51 10637 10562 10772 10562 query52 90 89 87 87 query53 265 287 208 208 query54 301 257 233 233 query55 79 80 72 72 query56 308 325 302 302 query57 10291049931 931 query58 318 275 272 272 query59 1555164813851385 query60 315 312 262 262 query61 171 174 176 174 query62 405 323 274 274 query63 235 200 210 200 query64 30421165984 984 query65 3917383637993799 query66 1838505 375 375 query67 28207 28091 28009 28009 query68 32261663956 956 query69 415 309 270 270 query70 913 738 777 738 query71 363 338 319 319 query72 3019259223052305 query73 873 802 437 437 query74 4615450242824282 query75 2363236019871987 query76 23261195745 745 query77 357 364 273 273 query78 11312 11225 10506 10506 query79 14841174695 695 query80 1240551 461 461 query81 538 326 281 281 query82 639 173 142 142 query83 374 319 293 293 query84 334 159 134 134 query85 981 603 517 517 query86 406 238 212 212 query87 1963196318411841 query88 3772284728032803 query89 386 325 283 283 query90 1991198 191 191 query91 207 195 158 158 query92 63 61 56 56 query93 16601537937 937 query94 722 367 317 317 query95 787 573 472 472 query96 1080785 365 365 query97 2468244223752375 query98 200 190 184 184 query99 752 734 608 608 Total cold run time: 254291 ms Total hot run time: 166284 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go
Re: [PR] [improvement](compaction) split time series compaction score metric [doris]
hello-stephen commented on PR #63900: URL: https://github.com/apache/doris/pull/63900#issuecomment-5204393107 TPC-H: Total hot run time: 29083 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit d9faa0fad7a0160c67f206ce45fa99a6f21bb095, data reload: false -- Round 1 -- q1 17898 403339243924 q2 2010334 202 202 q3 10300 1432810 810 q4 4679477 345 345 q5 7549843 554 554 q6 184 166 139 139 q7 769 800 602 602 q8 9337150616321506 q9 5391412740824082 q10 6729163213761376 q11 508 350 322 322 q12 732 570 446 446 q13 18112 330127462746 q14 271 257 239 239 q15 q16 732 732 664 664 q17 967 1000928 928 q18 6662560855795579 q19 1178127610581058 q20 804 688 571 571 q21 6226289326702670 q22 463 388 320 320 Total cold run time: 101501 ms Total hot run time: 29083 ms - Round 2, with runtime_filter_mode=off - q1 5012469646644664 q2 289 334 217 217 q3 5019581646514651 q4 2192227514381438 q5 472844214421 q6 238 179 136 136 q7 1843171815211521 q8 2369205920082008 q9 7045675767416741 q10 4282419738053805 q11 520 375 344 344 q12 699 714 493 493 q13 2947330727242724 q14 266 279 250 250 q15 q16 661 683 596 596 q17 1246121912101210 q18 12142 10986 11775 10986 q19 1094106010681060 q20 2187220418931893 q21 5301455145274527 q22 537 453 390 390 Total cold run time: 60617 ms Total hot run time: 54075 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](compaction) split time series compaction score metric [doris]
hello-stephen commented on PR #63900: URL: https://github.com/apache/doris/pull/63900#issuecomment-5204124151 # BE Regression && UT Coverage Report Increment line coverage `97.46% (115/118)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/63900_d9faa0fad7a0160c67f206ce45fa99a6f21bb095_merge/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/63900_d9faa0fad7a0160c67f206ce45fa99a6f21bb095_merge/report/index.html) | Category | Coverage | |---|| | Function Coverage | 75.96% (32287/42508) | | Line Coverage | 60.73% (361398/595136) | | Region Coverage | 57.36% (303916/529836) | | Branch Coverage | 58.81% (137295/233443) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](compaction) split time series compaction score metric [doris]
hello-stephen commented on PR #63900: URL: https://github.com/apache/doris/pull/63900#issuecomment-5204012804 # BE UT Coverage Report Increment line coverage `97.08% (133/137)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/d9faa0fad7a0160c67f206ce45fa99a6f21bb095_d9faa0fad7a0160c67f206ce45fa99a6f21bb095/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/d9faa0fad7a0160c67f206ce45fa99a6f21bb095_d9faa0fad7a0160c67f206ce45fa99a6f21bb095/report/index.html) | Category | Coverage | |---|| | Function Coverage | 59.81% (26093/43627) | | Line Coverage | 44.14% (264737/599741) | | Region Coverage | 39.88% (210813/528582) | | Branch Coverage | 41.31% (96550/233748) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](compaction) split time series compaction score metric [doris]
Yukang-Lian commented on PR #63900: URL: https://github.com/apache/doris/pull/63900#issuecomment-5202048699 run performance -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](compaction) split time series compaction score metric [doris]
Yukang-Lian commented on PR #63900: URL: https://github.com/apache/doris/pull/63900#issuecomment-5202047708 run beut -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](compaction) split time series compaction score metric [doris]
Yukang-Lian commented on PR #63900: URL: https://github.com/apache/doris/pull/63900#issuecomment-5189555014 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](compaction) split time series compaction score metric [doris]
github-actions[bot] commented on code in PR #63900:
URL: https://github.com/apache/doris/pull/63900#discussion_r3465299209
##
be/src/storage/olap_server.cpp:
##
@@ -915,38 +920,59 @@ std::vector
StorageEngine::_generate_compaction_tasks(
// Even if need_pick_tablet is false, we still need to call
find_best_tablet_to_compaction(),
// So that we can update the max_compaction_score metric.
-if (!data_dir->reach_capacity_limit(0)) {
-uint32_t disk_max_score = 0;
-auto tablet_contexts =
compaction_registry_snapshot.pick_topn_tablets_for_compaction(
-_tablet_manager.get(), data_dir, compaction_type,
-cumulative_compaction_policies, &disk_max_score);
-int concurrent_num = get_concurrent_per_disk(
-disk_max_score, disk_compaction_slot_num(*data_dir,
compaction_type));
-need_pick_tablet = need_generate_compaction_tasks(
-executing_task_num, concurrent_num, compaction_type,
-!compaction_registry_snapshot.has_compaction_task(
-data_dir, CompactionType::CUMULATIVE_COMPACTION));
-for (const auto& context : tablet_contexts) {
-if (context.tablet != nullptr) {
-if (need_pick_tablet) {
-tablet_compaction_contexts.emplace_back(context);
-}
-max_compaction_score = std::max(max_compaction_score,
disk_max_score);
+if (data_dir->reach_capacity_limit(0)) {
+skipped_capacity_limited_dir = true;
+continue;
+}
+
+CompactionScoreStats disk_score_stats;
+auto tablet_contexts =
compaction_registry_snapshot.pick_topn_tablets_for_compaction(
+_tablet_manager.get(), data_dir, compaction_type,
cumulative_compaction_policies,
+&disk_score_stats);
+max_score_stats.scanned = max_score_stats.scanned ||
disk_score_stats.scanned;
+max_score_stats.max_score = std::max(max_score_stats.max_score,
disk_score_stats.max_score);
+max_score_stats.size_based_max_score =
std::max(max_score_stats.size_based_max_score,
+
disk_score_stats.size_based_max_score);
+max_score_stats.time_series_max_score =
std::max(max_score_stats.time_series_max_score,
+
disk_score_stats.time_series_max_score);
+int concurrent_num = get_concurrent_per_disk(
+disk_score_stats.max_score,
disk_compaction_slot_num(*data_dir, compaction_type));
+need_pick_tablet = need_generate_compaction_tasks(
+executing_task_num, concurrent_num, compaction_type,
+!compaction_registry_snapshot.has_compaction_task(
+data_dir, CompactionType::CUMULATIVE_COMPACTION));
+for (const auto& context : tablet_contexts) {
+if (context.tablet != nullptr) {
+if (need_pick_tablet) {
+tablet_compaction_contexts.emplace_back(context);
}
}
}
}
-if (max_compaction_score > 0) {
-if (compaction_type == CompactionType::BASE_COMPACTION) {
+if (max_score_stats.scanned) {
+if (compaction_type == CompactionType::BASE_COMPACTION &&
max_score_stats.max_score > 0) {
DorisMetrics::instance()->tablet_base_max_compaction_score->set_value(
-max_compaction_score);
+max_score_stats.max_score);
} else if (compaction_type == CompactionType::CUMULATIVE_COMPACTION) {
-
DorisMetrics::instance()->tablet_cumulative_max_compaction_score->set_value(
-max_compaction_score);
-} else if (compaction_type == CompactionType::BINLOG_COMPACTION) {
+auto update_policy_score = [skipped_capacity_limited_dir,
check_score](IntGauge* metric,
+
int64_t score) {
+if (skipped_capacity_limited_dir) {
+if (score > metric->value()) {
+metric->set_value(score);
+}
+} else if (check_score || score > 0) {
+metric->set_value(score);
Review Comment:
`tablet_cumulative_max_compaction_score` is an existing exported BE metric,
but after this change it no longer means "max cumulative compaction score"; it
only tracks size-based cumulative scores while time-series backlog moves to
`tablet_time_series_max_compaction_score`. In the base code this same metric
was fed from the single aggregate cumulative `max_compaction_score`, so
existing dashboards/alerts will silently miss time-series tablets after upgrade
unless they know to query both metrics. Please either keep the
Re: [PR] [improvement](compaction) split time series compaction score metric [doris]
Yukang-Lian commented on PR #63900: URL: https://github.com/apache/doris/pull/63900#issuecomment-4786796579 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](compaction) split time series compaction score metric [doris]
hello-stephen commented on PR #63900: URL: https://github.com/apache/doris/pull/63900#issuecomment-4656321867 # BE Regression && UT Coverage Report Increment line coverage `96.52% (111/115)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/63900_c45fe72d7898e11f553a077fa29f33b9f3382ba4_merge/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/63900_c45fe72d7898e11f553a077fa29f33b9f3382ba4_merge/report/index.html) | Category | Coverage | |---|| | Function Coverage | 63.73% (24407/38300) | | Line Coverage | 47.61% (253237/531944) | | Region Coverage | 44.31% (208917/471438) | | Branch Coverage | 45.48% (90736/199493) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](compaction) split time series compaction score metric [doris]
Yukang-Lian commented on PR #63900: URL: https://github.com/apache/doris/pull/63900#issuecomment-4655542938 run nonConcurrent -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](compaction) split time series compaction score metric [doris]
Yukang-Lian commented on PR #63900: URL: https://github.com/apache/doris/pull/63900#issuecomment-4655543661 run cloud_p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](compaction) split time series compaction score metric [doris]
hello-stephen commented on PR #63900: URL: https://github.com/apache/doris/pull/63900#issuecomment-4651301444 TPC-DS: Total hot run time: 169015 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit c45fe72d7898e11f553a077fa29f33b9f3382ba4, data reload: false query5 4316637 484 484 query6 468 198 179 179 query7 4846555 302 302 query8 377 222 207 207 query9 8780406140434043 query10 449 315 254 254 query11 5844235722212221 query12 159 104 100 100 query13 1272622 436 436 query14 6367538550665066 query14_14388443144194419 query15 205 201 183 183 query16 1026463 437 437 query17 958 708 588 588 query18 2466477 355 355 query19 206 197 164 164 query20 116 110 106 106 query21 225 140 120 120 query22 13685 13662 13380 13380 query23 17280 16614 16186 16186 query23_116381 16273 16229 16229 query24 7501175612871287 query24_11308131113271311 query25 551 459 384 384 query26 1349307 161 161 query27 2740543 309 309 query28 4477201019941994 query29 1095611 512 512 query30 315 249 200 200 query31 11101077955 955 query32 163 68 60 60 query33 516 321 258 258 query34 11611098663 663 query35 762 783 679 679 query36 1358142412281228 query37 158 117 85 85 query38 3213314930323032 query39 938 919 905 905 query39_1866 906 897 897 query40 215 124 108 108 query41 67 67 62 62 query42 102 94 95 94 query43 322 319 279 279 query44 query45 196 182 181 181 query46 10991212743 743 query47 2362239322522252 query48 354 406 299 299 query49 636 469 364 364 query50 1032343 255 255 query51 4376427242624262 query52 87 87 77 77 query53 242 270 196 196 query54 275 219 201 201 query55 76 77 69 69 query56 234 234 211 211 query57 1419141513301330 query58 276 211 196 196 query59 1578161914641464 query60 287 259 233 233 query61 161 160 161 160 query62 706 642 591 591 query63 224 192 183 183 query64 2614793 623 623 query65 query66 1831498 345 345 query67 29711 29581 29563 29563 query68 query69 423 306 268 268 query70 958 946 919 919 query71 293 226 213 213 query72 3062270324072407 query73 836 762 415 415 query74 5107496947724772 query75 2675258822722272 query76 23441136761 761 query77 364 369 289 289 query78 12382 12297 11798 11798 query79 13181043760 760 query80 595 478 396 396 query81 448 280 247 247 query82 643 157 120 120 query83 362 279 252 252 query84 query85 900 542 448 448 query86 367 292 298 292 query87 3423335831923192 query88 3631273127022702 query89 418 389 329 329 query90 1937177 182 177 query91 175 161 141 141 query92 59 61 60 60 query93 13961421946 946 query94 570 345 325 325 query95 675 373 436 373 query96 1026762 355 355 query97 2702269925752575 query98 214 215 204 204 query99 1147117910231023 Total cold run time: 250614 ms Total hot run time: 169015 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about
Re: [PR] [improvement](compaction) split time series compaction score metric [doris]
hello-stephen commented on PR #63900: URL: https://github.com/apache/doris/pull/63900#issuecomment-4651206913 TPC-H: Total hot run time: 29207 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit c45fe72d7898e11f553a077fa29f33b9f3382ba4, data reload: false -- Round 1 -- orders Doris NULLNULL0 0 0 NULL0 NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 17735 409940214021 q2 q3 10791 1470854 854 q4 4686482 345 345 q5 7628910 589 589 q6 193 182 142 142 q7 804 864 650 650 q8 9374169916041604 q9 5811455644734473 q10 6758182615301530 q11 446 282 257 257 q12 634 421 300 300 q13 18152 343228082808 q14 273 266 251 251 q15 q16 834 781 712 712 q17 949 892 935 892 q18 6917592656635663 q19 1365127110811081 q20 547 409 253 253 q21 5838263924702470 q22 431 366 312 312 Total cold run time: 100166 ms Total hot run time: 29207 ms - Round 2, with runtime_filter_mode=off - orders Doris NULLNULL15000 42 6422171781 NULL22778155NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 4380435243494349 q2 q3 4544497143484348 q4 2090219414051405 q5 4504435643854356 q6 233 177 133 133 q7 1743186919101869 q8 2658223022302230 q9 8167837779897989 q10 4837473942794279 q11 572 430 384 384 q12 766 799 549 549 q13 3374366329682968 q14 295 314 262 262 q15 q16 741 750 659 659 q17 1394134813691348 q18 8114738871787178 q19 1186117811351135 q20 2215222819691969 q21 5319462445094509 q22 537 465 409 409 Total cold run time: 57669 ms Total hot run time: 52328 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](compaction) split time series compaction score metric [doris]
hello-stephen commented on PR #63900: URL: https://github.com/apache/doris/pull/63900#issuecomment-4648914018 # BE Regression && UT Coverage Report Increment line coverage `96.52% (111/115)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/63900_c45fe72d7898e11f553a077fa29f33b9f3382ba4_merge/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/63900_c45fe72d7898e11f553a077fa29f33b9f3382ba4_merge/report/index.html) | Category | Coverage | |---|| | Function Coverage | 63.73% (24407/38300) | | Line Coverage | 47.61% (253237/531944) | | Region Coverage | 44.31% (208917/471438) | | Branch Coverage | 45.48% (90736/199493) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](compaction) split time series compaction score metric [doris]
hello-stephen commented on PR #63900: URL: https://github.com/apache/doris/pull/63900#issuecomment-4647661716 # BE UT Coverage Report Increment line coverage `93.28% (125/134)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/c45fe72d7898e11f553a077fa29f33b9f3382ba4_c45fe72d7898e11f553a077fa29f33b9f3382ba4/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/c45fe72d7898e11f553a077fa29f33b9f3382ba4_c45fe72d7898e11f553a077fa29f33b9f3382ba4/report/index.html) | Category | Coverage | |---|| | Function Coverage | 53.96% (21156/39209) | | Line Coverage | 37.67% (201533/535009) | | Region Coverage | 33.68% (158017/469239) | | Branch Coverage | 34.73% (69189/199212) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](compaction) split time series compaction score metric [doris]
Yukang-Lian commented on PR #63900: URL: https://github.com/apache/doris/pull/63900#issuecomment-4646088748 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](compaction) split time series compaction score metric [doris]
hello-stephen commented on PR #63900: URL: https://github.com/apache/doris/pull/63900#issuecomment-4573498013 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
