Re: [PR] [enhance](mtmv)cache table snapshot in refresh context [doris]
morrySnow merged PR #50855: URL: https://github.com/apache/doris/pull/50855 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](mtmv)cache table snapshot in refresh context [doris]
github-actions[bot] commented on PR #50855: URL: https://github.com/apache/doris/pull/50855#issuecomment-2893220084 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](mtmv)cache table snapshot in refresh context [doris]
doris-robot commented on PR #50855: URL: https://github.com/apache/doris/pull/50855#issuecomment-2892813786 ClickBench: Total hot run time: 28.74 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 94ae9c96a763c05ce1ae272497c588bad8813ad8, data reload: false query1 0.040.040.03 query2 0.120.100.11 query3 0.260.190.19 query4 1.600.200.19 query5 0.450.430.44 query6 1.550.640.67 query7 0.020.020.01 query8 0.040.030.03 query9 0.590.530.52 query10 0.570.590.58 query11 0.150.120.11 query12 0.150.120.12 query13 0.610.600.59 query14 0.830.800.81 query15 0.880.860.87 query16 0.390.380.38 query17 1.071.031.01 query18 0.220.210.21 query19 1.961.811.85 query20 0.010.010.01 query21 15.40 0.900.58 query22 0.771.260.64 query23 14.85 1.350.61 query24 7.202.150.49 query25 0.500.200.06 query26 0.520.160.13 query27 0.050.050.05 query28 9.320.860.45 query29 12.57 3.963.35 query30 0.260.090.06 query31 2.830.590.38 query32 3.230.560.48 query33 3.083.043.11 query34 15.82 5.084.53 query35 4.524.504.46 query36 0.660.500.49 query37 0.080.060.06 query38 0.060.040.04 query39 0.030.020.02 query40 0.170.140.13 query41 0.080.030.02 query42 0.030.020.02 query43 0.040.030.03 Total cold run time: 103.58 s Total hot run time: 28.74 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](mtmv)cache table snapshot in refresh context [doris]
doris-robot commented on PR #50855: URL: https://github.com/apache/doris/pull/50855#issuecomment-2892807330 TPC-DS: Total hot run time: 186865 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 94ae9c96a763c05ce1ae272497c588bad8813ad8, data reload: false query1 1028491 506 491 query2 6566184118321832 query3 6770223 242 223 query4 26211 23313 23406 23313 query5 4337623 463 463 query6 317 218 196 196 query7 4625500 300 300 query8 304 261 245 245 query9 8645265326612653 query10 491 345 282 282 query11 15816 15087 15086 15086 query12 166 114 110 110 query13 1669560 420 420 query14 9335623862066206 query15 203 192 179 179 query16 7345626 457 457 query17 1169699 559 559 query18 1980409 327 327 query19 190 185 160 160 query20 129 123 121 121 query21 225 126 108 108 query22 4124411941704119 query23 34024 33388 33196 33196 query24 8507241924332419 query25 532 459 395 395 query26 1227269 157 157 query27 2746493 352 352 query28 4291210720952095 query29 792 563 436 436 query30 284 210 190 190 query31 950 863 741 741 query32 73 67 62 62 query33 554 376 316 316 query34 805 851 539 539 query35 791 812 755 755 query36 937 994 893 893 query37 113 97 76 76 query38 4115411840424042 query39 1463143514131413 query40 212 120 110 110 query41 58 57 55 55 query42 129 116 107 107 query43 504 500 472 472 query44 1348835 822 822 query45 177 178 169 169 query46 859 1021665 665 query47 1777176416891689 query48 400 455 319 319 query49 766 500 418 418 query50 637 686 437 437 query51 4140412241164116 query52 101 100 99 99 query53 231 263 195 195 query54 592 591 515 515 query55 84 90 87 87 query56 319 309 279 279 query57 1174114310821082 query58 270 253 259 253 query59 2552261625172517 query60 333 319 313 313 query61 127 125 127 125 query62 801 705 666 666 query63 230 193 194 193 query64 4261999 717 717 query65 4296424842624248 query66 1149425 330 330 query67 15862 15786 15448 15448 query68 7352913 538 538 query69 491 330 286 286 query70 1139119511151115 query71 403 347 315 315 query72 5684462946444629 query73 629 595 359 359 query74 8967915988448844 query75 3182321627022702 query76 32481204776 776 query77 528 404 381 381 query78 10212 10192 92699269 query79 2671816 579 579 query80 635 518 446 446 query81 507 250 214 214 query82 708 129 94 94 query83 263 250 247 247 query84 243 109 81 81 query85 815 398 312 312 query86 382 305 265 265 query87 4548445644154415 query88 4033232423092309 query89 388 315 281 281 query90 1929212 208 208 query91 142 147 112 112 query92 77 59 61 59 query93 2475922 587 587 query94 727 419 314 314 query95 384 304 284 284 query96 493 567 283 283 query97 2726274226882688 query98 238 213 206 206 query99 1326142312941294 Total cold run time: 274783 ms Total hot run time: 186865 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [enhance](mtmv)cache table snapshot in refresh context [doris]
doris-robot commented on PR #50855: URL: https://github.com/apache/doris/pull/50855#issuecomment-2892794566 TPC-H: Total hot run time: 35251 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 94ae9c96a763c05ce1ae272497c588bad8813ad8, data reload: false -- Round 1 -- q1 26746 527952465246 q2 2085291 184 184 q3 10396 1353713 713 q4 10253 1068570 570 q5 7911245324992453 q6 205 176 132 132 q7 967 769 609 609 q8 9327152712391239 q9 6858526352995263 q10 6871234818801880 q11 513 284 274 274 q12 348 378 219 219 q13 17784 380830933093 q14 256 247 213 213 q15 545 506 487 487 q16 428 437 373 373 q17 610 921 374 374 q18 7741727471357135 q19 12061090613 613 q20 355 381 227 227 q21 4107331229972997 q22 10801027957 957 Total cold run time: 116592 ms Total hot run time: 35251 ms - Round 2, with runtime_filter_mode=off - q1 5388536653835366 q2 271 345 233 233 q3 2209268523212321 q4 1581205415271527 q5 4562438044254380 q6 252 182 131 131 q7 2108192818221822 q8 2845283027482748 q9 7223713471047104 q10 3043321927532753 q11 621 512 494 494 q12 707 801 643 643 q13 3664402132953295 q14 281 304 274 274 q15 537 482 479 479 q16 448 509 439 439 q17 1217168613791379 q18 7554759074917491 q19 875 863 908 863 q20 1935199018891889 q21 4893439543914391 q22 10541034979 979 Total cold run time: 53268 ms Total hot run time: 51001 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](mtmv)cache table snapshot in refresh context [doris]
zddr commented on PR #50855: URL: https://github.com/apache/doris/pull/50855#issuecomment-2892771546 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](mtmv)cache table snapshot in refresh context [doris]
zddr commented on PR #50855: URL: https://github.com/apache/doris/pull/50855#issuecomment-2889434334 run external -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](mtmv)cache table snapshot in refresh context [doris]
seawinde commented on code in PR #50855:
URL: https://github.com/apache/doris/pull/50855#discussion_r2092645826
##
fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPartitionUtil.java:
##
@@ -488,12 +488,34 @@ private static boolean
isSyncWithBaseTable(MTMVRefreshContext context, String mt
if (!baseTable.needAutoRefresh()) {
return true;
}
-MTMVSnapshotIf baseTableCurrentSnapshot =
baseTable.getTableSnapshot(context,
-MvccUtil.getSnapshotFromContext(baseTable));
+MTMVSnapshotIf baseTableCurrentSnapshot =
getTableSnapshotFromContext(baseTable, context);
return mtmv.getRefreshSnapshot()
.equalsWithBaseTable(mtmvPartitionName, new
BaseTableInfo(baseTable), baseTableCurrentSnapshot);
}
+/**
+ * Try context first, then load via getTableSnapshot and cache
+ *
+ * @param mtmvRelatedTableIf Base table of materialized views
+ * @param context The context data persists for the duration of either a
refresh task
+ * or a transparent rewrite operation
+ * @return The snapshot information of the MTMV
+ * @throws AnalysisException
+ */
+public static MTMVSnapshotIf
getTableSnapshotFromContext(MTMVRelatedTableIf mtmvRelatedTableIf,
+MTMVRefreshContext context)
Review Comment:
in query statement, `MTMVRefreshContext` is also used? the only cache mv
partition version ?
##
fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPartitionUtil.java:
##
@@ -488,12 +488,34 @@ private static boolean
isSyncWithBaseTable(MTMVRefreshContext context, String mt
if (!baseTable.needAutoRefresh()) {
return true;
}
-MTMVSnapshotIf baseTableCurrentSnapshot =
baseTable.getTableSnapshot(context,
-MvccUtil.getSnapshotFromContext(baseTable));
+MTMVSnapshotIf baseTableCurrentSnapshot =
getTableSnapshotFromContext(baseTable, context);
return mtmv.getRefreshSnapshot()
.equalsWithBaseTable(mtmvPartitionName, new
BaseTableInfo(baseTable), baseTableCurrentSnapshot);
}
+/**
+ * Try context first, then load via getTableSnapshot and cache
+ *
+ * @param mtmvRelatedTableIf Base table of materialized views
+ * @param context The context data persists for the duration of either a
refresh task
+ * or a transparent rewrite operation
+ * @return The snapshot information of the MTMV
+ * @throws AnalysisException
+ */
+public static MTMVSnapshotIf
getTableSnapshotFromContext(MTMVRelatedTableIf mtmvRelatedTableIf,
+MTMVRefreshContext context)
Review Comment:
in query statement, `MTMVRefreshContext` is also used? only cache mv
partition version ? the table partition version which mv used maybe should not
cache
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](mtmv)cache table snapshot in refresh context [doris]
morrySnow commented on code in PR #50855:
URL: https://github.com/apache/doris/pull/50855#discussion_r2092617066
##
fe/fe-core/src/test/java/org/apache/doris/mtmv/MTMVPartitionUtilTest.java:
##
@@ -296,4 +301,19 @@ public void testIsTableNamelike() {
Assert.assertFalse(MTMVPartitionUtil.isTableNamelike(new
TableName("db1"), tableNameToCheck));
Assert.assertFalse(MTMVPartitionUtil.isTableNamelike(new
TableName("ctl1"), tableNameToCheck));
}
+
+@Test
+public void testGetTableSnapshotFromContext() throws AnalysisException {
+Map cache = Maps.newHashMap();
+new Expectations() {
+{
+context.getBaseTableSnapshotCache();
+minTimes = 0;
+result = cache;
+}
+};
+Assert.assertTrue(cache.isEmpty());
+MTMVPartitionUtil.getTableSnapshotFromContext(baseOlapTable, context);
Review Comment:
should test cache's object is correct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](mtmv)cache table snapshot in refresh context [doris]
zddr commented on PR #50855: URL: https://github.com/apache/doris/pull/50855#issuecomment-2875602745 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [enhance](mtmv)cache table snapshot in refresh context [doris]
hello-stephen commented on PR #50855: URL: https://github.com/apache/doris/pull/50855#issuecomment-2875600159 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
