Re: [PR] [fix](mtmv) release read lock when align mvmv's partition [doris]
morrySnow merged PR #53069: URL: https://github.com/apache/doris/pull/53069 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) release read lock when align mvmv's partition [doris]
github-actions[bot] commented on PR #53069: URL: https://github.com/apache/doris/pull/53069#issuecomment-3086582010 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) release read lock when align mvmv's partition [doris]
liutang123 commented on PR #53069: URL: https://github.com/apache/doris/pull/53069#issuecomment-3083788785 @zddr Hello, do you have time to see this PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) release read lock when align mvmv's partition [doris]
liutang123 commented on PR #53069: URL: https://github.com/apache/doris/pull/53069#issuecomment-3073331896 run feut -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) release read lock when align mvmv's partition [doris]
doris-robot commented on PR #53069: URL: https://github.com/apache/doris/pull/53069#issuecomment-3072939531 ClickBench: Total hot run time: 32.51 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit d321455f039df97968c5ba34f88261666a2900ae, data reload: false query1 0.040.030.04 query2 0.080.040.03 query3 0.240.070.07 query4 1.620.110.11 query5 0.440.420.41 query6 1.170.670.66 query7 0.020.020.02 query8 0.040.030.04 query9 0.610.510.52 query10 0.570.560.58 query11 0.150.110.11 query12 0.140.110.12 query13 0.630.620.61 query14 0.780.800.82 query15 0.910.880.88 query16 0.380.380.39 query17 1.101.091.06 query18 0.220.210.20 query19 1.921.871.89 query20 0.010.010.01 query21 15.38 0.930.54 query22 0.741.280.68 query23 14.88 1.380.62 query24 6.901.530.88 query25 0.490.130.18 query26 0.610.160.15 query27 0.060.050.05 query28 9.720.900.45 query29 12.56 3.923.31 query30 3.032.982.88 query31 2.810.600.40 query32 3.240.550.47 query33 3.143.053.13 query34 15.94 5.384.81 query35 4.854.814.83 query36 0.690.520.48 query37 0.090.070.07 query38 0.050.040.04 query39 0.040.020.03 query40 0.170.130.13 query41 0.080.030.03 query42 0.040.020.03 query43 0.040.030.03 Total cold run time: 106.62 s Total hot run time: 32.51 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) release read lock when align mvmv's partition [doris]
doris-robot commented on PR #53069: URL: https://github.com/apache/doris/pull/53069#issuecomment-3072922244 TPC-DS: Total hot run time: 187653 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit d321455f039df97968c5ba34f88261666a2900ae, data reload: false query1 1001383 419 383 query2 6531164616421642 query3 6739209 204 204 query4 25907 23462 23362 23362 query5 4377565 433 433 query6 313 227 207 207 query7 4623491 291 291 query8 281 231 209 209 query9 8602260626102606 query10 475 344 266 266 query11 15740 15174 14815 14815 query12 164 109 103 103 query13 1650529 394 394 query14 9485592659355926 query15 198 196 167 167 query16 7661436 255 255 query17 1324719 552 552 query18 2008409 293 293 query19 192 201 163 163 query20 123 118 113 113 query21 214 127 117 117 query22 4245424242394239 query23 34213 33157 33178 33157 query24 8473232223972322 query25 509 449 403 403 query26 1227268 143 143 query27 2771516 337 337 query28 4298212121012101 query29 720 553 443 443 query30 287 225 195 195 query31 940 841 768 768 query32 70 64 60 60 query33 551 343 290 290 query34 802 853 506 506 query35 597 658 565 565 query36 962 991 888 888 query37 123 102 73 73 query38 4117409640514051 query39 1502142014291420 query40 205 115 105 105 query41 58 56 52 52 query42 121 105 113 105 query43 507 490 481 481 query44 1367818 823 818 query45 180 170 168 168 query46 851 1025631 631 query47 1786184517311731 query48 379 421 305 305 query49 732 461 404 404 query50 656 688 427 427 query51 5550563053335333 query52 130 108 99 99 query53 225 262 182 182 query54 564 561 491 491 query55 82 79 82 79 query56 289 293 303 293 query57 1208121111401140 query58 267 273 274 273 query59 2534267724892489 query60 346 337 314 314 query61 134 124 126 124 query62 797 719 648 648 query63 214 185 184 184 query64 4336137310021002 query65 4271415741824157 query66 1072431 320 320 query67 15925 15727 15504 15504 query68 8069900 538 538 query69 514 399 268 268 query70 1158116311131113 query71 472 316 293 293 query72 5529468147644681 query73 689 594 348 348 query74 9293908889818981 query75 3866322126572657 query76 37091236760 760 query77 803 393 301 301 query78 11008 10866 10319 10319 query79 2678828 584 584 query80 622 510 449 449 query81 482 256 219 219 query82 485 125 98 98 query83 283 249 235 235 query84 304 103 81 81 query85 804 370 351 351 query86 386 300 292 292 query87 4390446143174317 query88 3687224722642247 query89 386 304 286 286 query90 1856211 204 204 query91 137 141 108 108 query92 74 58 54 54 query93 1976945 587 587 query94 682 309 201 201 query95 373 295 284 284 query96 504 578 277 277 query97 2737275527222722 query98 238 217 205 205 query99 1415138112721272 Total cold run time: 278410 ms Total hot run time: 187653 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t
Re: [PR] [fix](mtmv) release read lock when align mvmv's partition [doris]
doris-robot commented on PR #53069: URL: https://github.com/apache/doris/pull/53069#issuecomment-3072884325 TPC-H: Total hot run time: 33461 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit d321455f039df97968c5ba34f88261666a2900ae, data reload: false -- Round 1 -- q1 17553 531651045104 q2 1963296 186 186 q3 10272 1304757 757 q4 10274 1042508 508 q5 8449236223592359 q6 212 156 131 131 q7 902 756 599 599 q8 9346128110781078 q9 7283513050925092 q10 6942239219801980 q11 492 294 268 268 q12 370 351 209 209 q13 17755 371031093109 q14 226 235 213 213 q15 550 479 479 479 q16 440 424 373 373 q17 626 882 378 378 q18 7478713271627132 q19 1365944 562 562 q20 354 356 223 223 q21 4073312924292429 q22 368 325 292 292 Total cold run time: 107293 ms Total hot run time: 33461 ms - Round 2, with runtime_filter_mode=off - q1 5331519451455145 q2 250 335 218 218 q3 2165263923122312 q4 1351174713241324 q5 4240445544554455 q6 223 172 128 128 q7 2092196018691869 q8 2687257426182574 q9 7280731671937193 q10 3110337929122912 q11 568 507 501 501 q12 730 774 611 611 q13 3591400235383538 q14 285 323 288 288 q15 508 498 465 465 q16 450 489 451 451 q17 1189156413841384 q18 7980780374347434 q19 830 800 927 800 q20 1939195118031803 q21 4744438144914381 q22 617 574 532 532 Total cold run time: 52160 ms Total hot run time: 50318 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) release read lock when align mvmv's partition [doris]
liutang123 commented on PR #53069: URL: https://github.com/apache/doris/pull/53069#issuecomment-3072809953 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) release read lock when align mvmv's partition [doris]
doris-robot commented on PR #53069: URL: https://github.com/apache/doris/pull/53069#issuecomment-3071784881 ClickBench: Total hot run time: 31.92 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 28e1aebb130bc686dbc73e6a8e856f0b4a0b3c87, data reload: false query1 0.050.030.04 query2 0.080.040.04 query3 0.250.080.07 query4 1.620.110.11 query5 0.450.420.43 query6 1.180.670.66 query7 0.030.020.03 query8 0.060.040.04 query9 0.600.520.52 query10 0.570.580.57 query11 0.160.110.11 query12 0.150.120.11 query13 0.620.610.61 query14 0.790.820.81 query15 0.900.870.88 query16 0.390.410.40 query17 1.081.041.03 query18 0.230.230.22 query19 1.911.801.85 query20 0.020.010.01 query21 15.43 0.910.56 query22 0.761.210.63 query23 14.97 1.370.66 query24 8.291.040.35 query25 0.470.230.08 query26 0.620.180.15 query27 0.070.060.05 query28 8.740.870.44 query29 12.53 4.033.34 query30 2.982.952.91 query31 2.820.600.38 query32 3.230.560.48 query33 3.152.993.15 query34 16.20 5.414.79 query35 4.834.854.89 query36 0.680.510.48 query37 0.090.070.07 query38 0.050.040.03 query39 0.030.020.02 query40 0.180.140.14 query41 0.080.020.02 query42 0.040.030.02 query43 0.030.040.03 Total cold run time: 107.41 s Total hot run time: 31.92 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) release read lock when align mvmv's partition [doris]
hello-stephen commented on PR #53069: URL: https://github.com/apache/doris/pull/53069#issuecomment-3071781572 # FE UT Coverage Report Increment line coverage ` 50.00% (8/16)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/53069_28e1aebb130bc686dbc73e6a8e856f0b4a0b3c87/fe_increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/53069_28e1aebb130bc686dbc73e6a8e856f0b4a0b3c87/fe_report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) release read lock when align mvmv's partition [doris]
doris-robot commented on PR #53069: URL: https://github.com/apache/doris/pull/53069#issuecomment-3071777437 TPC-DS: Total hot run time: 188542 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 28e1aebb130bc686dbc73e6a8e856f0b4a0b3c87, data reload: false query1 997 391 375 375 query2 6502177117851771 query3 6754215 224 215 query4 26900 23378 22943 22943 query5 4350570 434 434 query6 308 225 219 219 query7 4620517 289 289 query8 263 222 227 222 query9 8607259126212591 query10 493 343 264 264 query11 15743 15101 14887 14887 query12 161 108 104 104 query13 1660530 420 420 query14 8870600460816004 query15 208 198 172 172 query16 7305438 267 267 query17 1318738 596 596 query18 1999418 311 311 query19 209 194 167 167 query20 130 128 117 117 query21 220 133 117 117 query22 4384451246614512 query23 34949 33970 33396 33396 query24 8437233123882331 query25 560 456 384 384 query26 1228265 143 143 query27 2757537 338 338 query28 4263211920892089 query29 757 550 433 433 query30 282 216 192 192 query31 949 848 769 769 query32 76 62 72 62 query33 537 324 292 292 query34 803 843 517 517 query35 627 625 556 556 query36 977 1036932 932 query37 111 96 71 71 query38 4236418741154115 query39 1455145115261451 query40 210 118 107 107 query41 55 52 52 52 query42 121 109 105 105 query43 510 530 490 490 query44 1290809 815 809 query45 174 179 164 164 query46 854 1020626 626 query47 1775184217451745 query48 385 410 308 308 query49 735 471 392 392 query50 633 686 404 404 query51 5453553354985498 query52 107 105 94 94 query53 213 253 186 186 query54 567 570 487 487 query55 84 83 78 78 query56 299 294 290 290 query57 1198121411431143 query58 263 246 252 246 query59 2641273426982698 query60 338 335 307 307 query61 123 119 123 119 query62 815 715 660 660 query63 226 180 183 180 query64 44121212824 824 query65 4294418142504181 query66 1161411 322 322 query67 16237 15905 15698 15698 query68 8472919 524 524 query69 515 313 268 268 query70 1265120511611161 query71 493 319 293 293 query72 5565470047884700 query73 713 597 349 349 query74 8921903187498749 query75 3986320126882688 query76 37271143709 709 query77 779 442 299 299 query78 10948 11066 10490 10490 query79 2647795 577 577 query80 662 507 448 448 query81 460 264 223 223 query82 443 129 96 96 query83 284 248 232 232 query84 296 102 85 85 query85 779 359 317 317 query86 332 311 314 311 query87 4433450943864386 query88 2885228422662266 query89 415 325 285 285 query90 1909200 200 200 query91 140 136 109 109 query92 75 57 59 57 query93 1850956 590 590 query94 672 359 199 199 query95 385 291 282 282 query96 497 575 278 278 query97 2725274926392639 query98 229 207 199 199 query99 1443140112521252 Total cold run time: 279123 ms Total hot run time: 188542 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [fix](mtmv) release read lock when align mvmv's partition [doris]
doris-robot commented on PR #53069: URL: https://github.com/apache/doris/pull/53069#issuecomment-3071762226 TPC-H: Total hot run time: 33579 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 28e1aebb130bc686dbc73e6a8e856f0b4a0b3c87, data reload: false -- Round 1 -- q1 17580 516050805080 q2 1943279 183 183 q3 10420 1305723 723 q4 10234 1029558 558 q5 7701240823912391 q6 175 162 130 130 q7 923 752 596 596 q8 9320132211011101 q9 6906519851995198 q10 6884240119651965 q11 487 299 264 264 q12 328 352 217 217 q13 17764 366430903090 q14 239 236 218 218 q15 534 488 482 482 q16 426 460 382 382 q17 596 889 363 363 q18 7546717872307178 q19 1210952 563 563 q20 349 344 221 221 q21 3899322123802380 q22 368 321 296 296 Total cold run time: 105832 ms Total hot run time: 33579 ms - Round 2, with runtime_filter_mode=off - q1 5102510451065104 q2 249 321 223 223 q3 2198273123232323 q4 1393177413391339 q5 4265439547114395 q6 220 167 122 122 q7 2070200618261826 q8 2710258825862586 q9 7422728572797279 q10 3172334129182918 q11 593 512 470 470 q12 711 817 663 663 q13 3563401235003500 q14 291 320 272 272 q15 537 476 493 476 q16 447 489 477 477 q17 1206158513981398 q18 8034787876647664 q19 830 818 931 818 q20 2005213319141914 q21 5137462645854585 q22 631 627 583 583 Total cold run time: 52786 ms Total hot run time: 50935 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) release read lock when align mvmv's partition [doris]
liutang123 commented on PR #53069: URL: https://github.com/apache/doris/pull/53069#issuecomment-3071683423 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) release read lock when align mvmv's partition [doris]
morrySnow commented on code in PR #53069:
URL: https://github.com/apache/doris/pull/53069#discussion_r2204410979
##
fe/fe-core/src/main/java/org/apache/doris/job/extensions/mtmv/MTMVTask.java:
##
@@ -211,13 +214,28 @@ public void run() throws JobException {
+ " e.g. Table has multiple partition columns"
+ " or including not supported transform
functions.");
}
-MTMVPartitionUtil.alignMvPartition(mtmv);
+syncPartitions = MTMVPartitionUtil.alignMvPartition(mtmv);
}
+} finally {
+MetaLockUtils.readUnlockTables(tableIfs);
+}
+if (syncPartitions != null) {
+for (String pName : syncPartitions.first) {
+MTMVPartitionUtil.dropPartition(mtmv, pName);
+}
+for (PartitionKeyDesc partitionKeyDesc :
syncPartitions.second) {
+MTMVPartitionUtil.addPartition(mtmv, partitionKeyDesc);
+}
+}
+MetaLockUtils.readLockTables(tableIfs);
+try {
context = MTMVRefreshContext.buildContext(mtmv);
this.needRefreshPartitions =
calculateNeedRefreshPartitions(context);
} finally {
MetaLockUtils.readUnlockTables(tableIfs);
}
+
+
Review Comment:
more than one blank line, this cannot pass CheckStyle
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) release read lock when align mvmv's partition [doris]
liutang123 commented on PR #53069: URL: https://github.com/apache/doris/pull/53069#issuecomment-3068156858 > I see the main change is moving the logic for adding and deleting partitions from MTMVPartitionUtil to MTMVTask? Can you help explain the benefits of doing this or what issues might arise if we don't make this change? I added a bug description. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) release read lock when align mvmv's partition [doris]
zddr commented on PR #53069: URL: https://github.com/apache/doris/pull/53069#issuecomment-3067687958 I see the main change is moving the logic for adding and deleting partitions from MTMVPartitionUtil to MTMVTask? Can you help explain the benefits of doing this or what issues might arise if we don't make this change? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) release read lock when align mvmv's partition [doris]
github-actions[bot] commented on PR #53069: URL: https://github.com/apache/doris/pull/53069#issuecomment-3060257664 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) release read lock when align mvmv's partition [doris]
github-actions[bot] commented on PR #53069: URL: https://github.com/apache/doris/pull/53069#issuecomment-3060257700 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) release read lock when align mvmv's partition [doris]
Thearas commented on PR #53069: URL: https://github.com/apache/doris/pull/53069#issuecomment-3057521818 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
