Re: [PR] [fix](mtmv) Fix compensate union wrongly when direct query is empty relation [doris]
morningman merged PR #51700: URL: https://github.com/apache/doris/pull/51700 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) Fix compensate union wrongly when direct query is empty relation [doris]
github-actions[bot] commented on PR #51700: URL: https://github.com/apache/doris/pull/51700#issuecomment-2982913137 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) Fix compensate union wrongly when direct query is empty relation [doris]
github-actions[bot] commented on PR #51700: URL: https://github.com/apache/doris/pull/51700#issuecomment-2982913192 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) Fix compensate union wrongly when direct query is empty relation [doris]
seawinde commented on code in PR #51700:
URL: https://github.com/apache/doris/pull/51700#discussion_r2153733097
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/InitMaterializationContextHook.java:
##
@@ -199,12 +209,17 @@ private List
createAsyncMaterializationContext(CascadesC
// so regenerate the struct info table bitset
StructInfo mvStructInfo = mtmvCache.getStructInfo();
BitSet tableBitSetInCurrentCascadesContext = new BitSet();
-mvStructInfo.getRelations().forEach(relation ->
tableBitSetInCurrentCascadesContext.set(
-
cascadesContext.getStatementContext().getTableId(relation.getTable()).asInt()));
+BitSet relationIdBitSetInCurrentCascadesContext = new BitSet();
+mvStructInfo.getRelations().forEach(relation -> {
+tableBitSetInCurrentCascadesContext.set(
+
cascadesContext.getStatementContext().getTableId(relation.getTable()).asInt());
+
relationIdBitSetInCurrentCascadesContext.set(relation.getRelationId().asInt());
+});
Review Comment:
this store tableId , the tableId is transfed and is 0 based. this bitset may
store relationId, the relationId also is 0 based, i think this would not be
very large
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) Fix compensate union wrongly when direct query is empty relation [doris]
924060929 commented on code in PR #51700:
URL: https://github.com/apache/doris/pull/51700#discussion_r2153596172
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/InitMaterializationContextHook.java:
##
@@ -199,12 +209,17 @@ private List
createAsyncMaterializationContext(CascadesC
// so regenerate the struct info table bitset
StructInfo mvStructInfo = mtmvCache.getStructInfo();
BitSet tableBitSetInCurrentCascadesContext = new BitSet();
-mvStructInfo.getRelations().forEach(relation ->
tableBitSetInCurrentCascadesContext.set(
-
cascadesContext.getStatementContext().getTableId(relation.getTable()).asInt()));
+BitSet relationIdBitSetInCurrentCascadesContext = new BitSet();
+mvStructInfo.getRelations().forEach(relation -> {
+tableBitSetInCurrentCascadesContext.set(
+
cascadesContext.getStatementContext().getTableId(relation.getTable()).asInt());
+
relationIdBitSetInCurrentCascadesContext.set(relation.getRelationId().asInt());
+});
Review Comment:
the BitSet maybe very big because the table id > 1 million, you should use
RoaringBitmap instead of
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) Fix compensate union wrongly when direct query is empty relation [doris]
924060929 commented on code in PR #51700:
URL: https://github.com/apache/doris/pull/51700#discussion_r2153596172
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/InitMaterializationContextHook.java:
##
@@ -199,12 +209,17 @@ private List
createAsyncMaterializationContext(CascadesC
// so regenerate the struct info table bitset
StructInfo mvStructInfo = mtmvCache.getStructInfo();
BitSet tableBitSetInCurrentCascadesContext = new BitSet();
-mvStructInfo.getRelations().forEach(relation ->
tableBitSetInCurrentCascadesContext.set(
-
cascadesContext.getStatementContext().getTableId(relation.getTable()).asInt()));
+BitSet relationIdBitSetInCurrentCascadesContext = new BitSet();
+mvStructInfo.getRelations().forEach(relation -> {
+tableBitSetInCurrentCascadesContext.set(
+
cascadesContext.getStatementContext().getTableId(relation.getTable()).asInt());
+
relationIdBitSetInCurrentCascadesContext.set(relation.getRelationId().asInt());
+});
Review Comment:
the BitSet of `tableBitSetInCurrentCascadesContext` maybe very big because
the table id > 1 million? you can use RoaringBitmap instead of
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) Fix compensate union wrongly when direct query is empty relation [doris]
924060929 commented on code in PR #51700:
URL: https://github.com/apache/doris/pull/51700#discussion_r2153596172
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/InitMaterializationContextHook.java:
##
@@ -199,12 +209,17 @@ private List
createAsyncMaterializationContext(CascadesC
// so regenerate the struct info table bitset
StructInfo mvStructInfo = mtmvCache.getStructInfo();
BitSet tableBitSetInCurrentCascadesContext = new BitSet();
-mvStructInfo.getRelations().forEach(relation ->
tableBitSetInCurrentCascadesContext.set(
-
cascadesContext.getStatementContext().getTableId(relation.getTable()).asInt()));
+BitSet relationIdBitSetInCurrentCascadesContext = new BitSet();
+mvStructInfo.getRelations().forEach(relation -> {
+tableBitSetInCurrentCascadesContext.set(
+
cascadesContext.getStatementContext().getTableId(relation.getTable()).asInt());
+
relationIdBitSetInCurrentCascadesContext.set(relation.getRelationId().asInt());
+});
Review Comment:
the BitSet of `tableBitSetInCurrentCascadesContext` maybe very big because
the table id > 1 million, you should use RoaringBitmap instead of
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) Fix compensate union wrongly when direct query is empty relation [doris]
hello-stephen commented on PR #51700: URL: https://github.com/apache/doris/pull/51700#issuecomment-2979974743 # FE UT Coverage Report Increment line coverage ` 80.00% (44/55)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/51700_fe19d56f1f9609cb180b891f4d2a0257c87abed2/fe_increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/51700_fe19d56f1f9609cb180b891f4d2a0257c87abed2/fe_report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) Fix compensate union wrongly when direct query is empty relation [doris]
doris-robot commented on PR #51700: URL: https://github.com/apache/doris/pull/51700#issuecomment-2979783074 ClickBench: Total hot run time: 29.11 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit fe19d56f1f9609cb180b891f4d2a0257c87abed2, data reload: false query1 0.030.030.03 query2 0.070.040.03 query3 0.240.060.07 query4 1.610.110.11 query5 0.430.410.40 query6 1.180.660.66 query7 0.020.020.01 query8 0.040.030.03 query9 0.590.530.52 query10 0.560.590.57 query11 0.150.110.11 query12 0.140.110.11 query13 0.640.600.61 query14 0.800.810.82 query15 0.880.860.86 query16 0.440.390.38 query17 1.071.071.05 query18 0.220.210.20 query19 2.031.851.86 query20 0.010.020.01 query21 15.40 0.890.55 query22 0.751.260.63 query23 14.93 1.350.63 query24 6.832.300.54 query25 0.510.110.11 query26 0.640.160.14 query27 0.060.040.05 query28 9.320.880.43 query29 12.57 3.893.27 query30 0.260.080.06 query31 2.840.590.38 query32 3.230.540.47 query33 3.043.083.24 query34 16.14 5.424.77 query35 4.884.784.80 query36 0.670.500.48 query37 0.090.060.06 query38 0.050.040.04 query39 0.030.030.03 query40 0.190.150.15 query41 0.070.020.02 query42 0.030.030.02 query43 0.040.030.03 Total cold run time: 103.72 s Total hot run time: 29.11 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) Fix compensate union wrongly when direct query is empty relation [doris]
doris-robot commented on PR #51700: URL: https://github.com/apache/doris/pull/51700#issuecomment-2979766550 TPC-DS: Total hot run time: 192136 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit fe19d56f1f9609cb180b891f4d2a0257c87abed2, data reload: false query1 14071020977 977 query2 6278182618541826 query3 10997 436844454368 query4 52742 24285 23096 23096 query5 5222515 470 470 query6 355 230 206 206 query7 5161512 294 294 query8 295 229 209 209 query9 6060265626372637 query10 437 315 272 272 query11 15076 14938 14933 14933 query12 159 104 102 102 query13 1058510 411 411 query14 10042 636362796279 query15 211 196 181 181 query16 7112692 473 473 query17 1068762 617 617 query18 1556423 334 334 query19 204 194 173 173 query20 135 129 138 129 query21 216 126 111 111 query22 4311438342794279 query23 34584 33736 33653 33653 query24 6556243824042404 query25 449 486 401 401 query26 673 277 146 146 query27 2343521 347 347 query28 3008219021622162 query29 584 552 444 444 query30 278 233 194 194 query31 879 872 771 771 query32 73 59 66 59 query33 469 380 330 330 query34 782 877 514 514 query35 811 850 769 769 query36 962 992 905 905 query37 121 101 90 90 query38 4287427641864186 query39 1567143814611438 query40 221 125 117 117 query41 70 69 62 62 query42 137 123 119 119 query43 515 516 472 472 query44 1335839 832 832 query45 191 181 179 179 query46 867 1067651 651 query47 1883185917951795 query48 405 427 376 376 query49 648 513 406 406 query50 699 702 409 409 query51 4209424341994199 query52 122 109 101 101 query53 235 264 191 191 query54 617 598 539 539 query55 91 85 80 80 query56 303 319 299 299 query57 1182124611621162 query58 269 264 252 252 query59 2717287028122812 query60 329 324 323 323 query61 135 129 131 129 query62 774 711 664 664 query63 226 192 186 186 query64 17121039677 677 query65 4287416341534153 query66 742 404 296 296 query67 15922 15640 15412 15412 query68 6806907 517 517 query69 542 314 280 280 query70 1195107910931079 query71 496 323 299 299 query72 6056480247514751 query73 1352602 358 358 query74 8863911887568756 query75 3809327126582658 query76 42811192758 758 query77 615 376 296 296 query78 10074 10117 92199219 query79 5799812 563 563 query80 651 507 472 472 query81 511 262 223 223 query82 620 126 102 102 query83 357 248 234 234 query84 286 104 80 80 query85 804 354 315 315 query86 383 292 299 292 query87 4457442643154315 query88 3418227622332233 query89 461 322 296 296 query90 1915208 209 208 query91 140 140 113 113 query92 77 62 56 56 query93 3138921 570 570 query94 669 410 285 285 query95 365 296 286 286 query96 501 576 285 285 query97 2713276126792679 query98 236 208 200 200 query99 1438138812751275 Total cold run time: 302399 ms Total hot run time: 192136 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go
Re: [PR] [fix](mtmv) Fix compensate union wrongly when direct query is empty relation [doris]
doris-robot commented on PR #51700: URL: https://github.com/apache/doris/pull/51700#issuecomment-2979724225 TPC-H: Total hot run time: 33682 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit fe19d56f1f9609cb180b891f4d2a0257c87abed2, data reload: false -- Round 1 -- q1 17624 511650055005 q2 1942283 173 173 q3 10313 1221746 746 q4 10211 983 518 518 q5 7542227823342278 q6 180 158 131 131 q7 897 754 611 611 q8 9308129210511051 q9 6786505551315055 q10 6891239419631963 q11 487 286 273 273 q12 338 352 208 208 q13 17764 369530873087 q14 228 223 219 219 q15 569 479 477 477 q16 428 434 385 385 q17 578 854 347 347 q18 7822715472207154 q19 1261954 540 540 q20 334 327 220 220 q21 3723255522782278 q22 10291007963 963 Total cold run time: 106255 ms Total hot run time: 33682 ms - Round 2, with runtime_filter_mode=off - q1 5019496850354968 q2 247 327 224 224 q3 2145264023132313 q4 1327176813521352 q5 4197411642664116 q6 208 169 128 128 q7 1986191817521752 q8 2589261725342534 q9 7252717071457145 q10 3040332128042804 q11 561 508 485 485 q12 643 777 638 638 q13 3522389132733273 q14 272 290 270 270 q15 518 481 472 472 q16 464 493 447 447 q17 1145157113751375 q18 7681746074497449 q19 789 806 950 806 q20 1983202819891989 q21 505343254325 q22 1088107210371037 Total cold run time: 51729 ms Total hot run time: 49902 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) Fix compensate union wrongly when direct query is empty relation [doris]
seawinde commented on PR #51700: URL: https://github.com/apache/doris/pull/51700#issuecomment-2979662640 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) Fix compensate union wrongly when direct query is empty relation [doris]
seawinde commented on PR #51700: URL: https://github.com/apache/doris/pull/51700#issuecomment-2979429679 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) Fix compensate union wrongly when direct query is empty relation [doris]
doris-robot commented on PR #51700: URL: https://github.com/apache/doris/pull/51700#issuecomment-2976293499 ClickBench: Total hot run time: 28.89 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit ab3292f64f79bdf2846ff64409108b2a565c6510, data reload: false query1 0.040.030.03 query2 0.060.040.04 query3 0.230.080.06 query4 1.600.110.10 query5 0.430.410.41 query6 1.170.660.66 query7 0.020.020.01 query8 0.040.030.03 query9 0.590.510.51 query10 0.580.580.56 query11 0.150.110.10 query12 0.140.110.11 query13 0.610.600.60 query14 0.780.810.80 query15 0.860.860.85 query16 0.370.380.39 query17 1.061.011.02 query18 0.220.210.21 query19 1.941.831.77 query20 0.020.010.01 query21 15.41 0.870.53 query22 0.751.040.75 query23 14.99 1.400.63 query24 7.240.900.93 query25 0.500.240.08 query26 0.580.160.14 query27 0.050.050.05 query28 9.410.930.46 query29 12.59 4.063.33 query30 0.260.090.07 query31 2.820.600.41 query32 3.230.550.48 query33 3.023.073.12 query34 15.92 5.094.47 query35 4.554.524.47 query36 0.670.510.48 query37 0.090.060.06 query38 0.050.040.04 query39 0.040.030.03 query40 0.180.130.12 query41 0.080.020.03 query42 0.040.020.02 query43 0.040.030.03 Total cold run time: 103.42 s Total hot run time: 28.89 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) Fix compensate union wrongly when direct query is empty relation [doris]
doris-robot commented on PR #51700: URL: https://github.com/apache/doris/pull/51700#issuecomment-2976264977 TPC-DS: Total hot run time: 184911 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit ab3292f64f79bdf2846ff64409108b2a565c6510, data reload: false query1 990 392 383 383 query2 6561180717571757 query3 6739228 224 224 query4 26386 23691 23357 23357 query5 4363621 444 444 query6 312 205 191 191 query7 4616499 275 275 query8 263 226 219 219 query9 8614258225832582 query10 484 341 268 268 query11 15626 15315 14759 14759 query12 159 109 101 101 query13 1652519 416 416 query14 9314599160135991 query15 206 191 171 171 query16 7233625 466 466 query17 1201732 574 574 query18 1968400 303 303 query19 197 196 179 179 query20 120 119 119 119 query21 218 134 112 112 query22 4445425941604160 query23 34020 33078 33097 33078 query24 8420237724122377 query25 530 446 384 384 query26 1239278 147 147 query27 2755497 337 337 query28 4369211520982098 query29 760 564 424 424 query30 290 219 185 185 query31 923 829 793 793 query32 70 67 61 61 query33 569 357 324 324 query34 790 832 539 539 query35 778 793 721 721 query36 949 998 882 882 query37 110 101 73 73 query38 4084408639853985 query39 1478139614051396 query40 210 120 107 107 query41 61 61 56 56 query42 124 104 104 104 query43 483 493 443 443 query44 1285808 815 808 query45 173 172 161 161 query46 831 1009612 612 query47 1759182416791679 query48 384 409 313 313 query49 747 485 385 385 query50 634 672 396 396 query51 4111419340464046 query52 113 106 98 98 query53 220 252 181 181 query54 592 568 494 494 query55 83 84 87 84 query56 298 287 285 285 query57 1201119211231123 query58 260 257 257 257 query59 2547259425282528 query60 346 324 306 306 query61 126 126 138 126 query62 794 712 663 663 query63 224 183 185 183 query64 4383985 666 666 query65 4242411041724110 query66 1146401 310 310 query67 15744 15525 15394 15394 query68 8021880 524 524 query69 467 302 265 265 query70 1242106710871067 query71 484 321 300 300 query72 5659479046024602 query73 689 576 355 355 query74 9004921588428842 query75 3873316726762676 query76 36531189741 741 query77 784 359 288 288 query78 10066 10228 94009400 query79 1969880 588 588 query80 577 509 501 501 query81 464 253 215 215 query82 419 124 96 96 query83 252 244 226 226 query84 244 108 89 89 query85 782 352 312 312 query86 334 298 281 281 query87 4320443742854285 query88 3447227522542254 query89 367 317 284 284 query90 1947209 203 203 query91 149 136 116 116 query92 72 60 63 60 query93 1176939 580 580 query94 675 387 296 296 query95 370 291 281 281 query96 499 555 288 288 query97 2674275126772677 query98 234 207 202 202 query99 1420141712441244 Total cold run time: 273255 ms Total hot run time: 184911 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [fix](mtmv) Fix compensate union wrongly when direct query is empty relation [doris]
doris-robot commented on PR #51700: URL: https://github.com/apache/doris/pull/51700#issuecomment-2976212057 TPC-H: Total hot run time: 34190 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit ab3292f64f79bdf2846ff64409108b2a565c6510, data reload: false -- Round 1 -- q1 17622 515850365036 q2 1917277 193 193 q3 10302 1394750 750 q4 10223 1022540 540 q5 7575241523342334 q6 177 164 135 135 q7 914 743 613 613 q8 9325131011761176 q9 6755503851715038 q10 6814239319231923 q11 474 295 279 279 q12 345 355 227 227 q13 17774 363731503150 q14 224 230 219 219 q15 550 480 477 477 q16 417 446 368 368 q17 595 869 361 361 q18 7592724570797079 q19 1215948 541 541 q20 345 346 226 226 q21 4222320525752575 q22 1034956 950 950 Total cold run time: 106411 ms Total hot run time: 34190 ms - Round 2, with runtime_filter_mode=off - q1 5072511050445044 q2 245 317 219 219 q3 2209265822472247 q4 1388175713441344 q5 4242408846224088 q6 214 171 128 128 q7 2072192218321832 q8 2631261825202520 q9 7216712871117111 q10 2987322027632763 q11 593 507 494 494 q12 666 767 634 634 q13 3459382532253225 q14 293 310 262 262 q15 511 484 463 463 q16 458 483 437 437 q17 1209151113651365 q18 7670752572907290 q19 809 791 892 791 q20 2003203118981898 q21 4946456544334433 q22 10691031973 973 Total cold run time: 51962 ms Total hot run time: 49561 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) Fix compensate union wrongly when direct query is empty relation [doris]
seawinde commented on PR #51700: URL: https://github.com/apache/doris/pull/51700#issuecomment-2976167837 run external 10 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) Fix compensate union wrongly when direct query is empty relation [doris]
hello-stephen commented on PR #51700: URL: https://github.com/apache/doris/pull/51700#issuecomment-2975686092 # FE UT Coverage Report Increment line coverage ` 77.50% (31/40)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/51700_ab3292f64f79bdf2846ff64409108b2a565c6510/fe_increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/51700_ab3292f64f79bdf2846ff64409108b2a565c6510/fe_report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) Fix compensate union wrongly when direct query is empty relation [doris]
seawinde commented on PR #51700: URL: https://github.com/apache/doris/pull/51700#issuecomment-2975366932 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) Fix compensate union wrongly when direct query is empty relation [doris]
doris-robot commented on PR #51700: URL: https://github.com/apache/doris/pull/51700#issuecomment-2975356710 TPC-H: Total hot run time: 33763 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit b3d79b218c8e81fc3726aa4876bf78cde9edf2c2, data reload: false -- Round 1 -- q1 17572 516449524952 q2 1940271 169 169 q3 10311 1235771 771 q4 10209 1029527 527 q5 7530238822932293 q6 186 162 130 130 q7 919 742 616 616 q8 9312133310721072 q9 6803505951475059 q10 6839232019151915 q11 480 285 273 273 q12 351 360 224 224 q13 17787 365830893089 q14 239 232 212 212 q15 570 485 475 475 q16 426 440 378 378 q17 606 867 368 368 q18 7534709670587058 q19 2065979 576 576 q20 334 335 216 216 q21 3922337424332433 q22 10101001957 957 Total cold run time: 106945 ms Total hot run time: 33763 ms - Round 2, with runtime_filter_mode=off - q1 5133500350195003 q2 255 317 219 219 q3 2150268422912291 q4 1315175813801380 q5 4197415943404159 q6 219 168 129 129 q7 1986195817571757 q8 2605267725162516 q9 7317722272787222 q10 3023314827112711 q11 584 523 510 510 q12 682 756 640 640 q13 3586390732463246 q14 279 310 281 281 q15 527 486 494 486 q16 441 515 453 453 q17 1169150013411341 q18 7886765174147414 q19 820 812 874 812 q20 1984206719601960 q21 5014442645394426 q22 1106101310241013 Total cold run time: 52278 ms Total hot run time: 49969 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) Fix compensate union wrongly when direct query is empty relation [doris]
seawinde commented on code in PR #51700:
URL: https://github.com/apache/doris/pull/51700#discussion_r2149188116
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/StructInfo.java:
##
@@ -752,10 +752,13 @@ public static Pair
addFilterOnTableScan(Plan queryPlan, Map {
+Plan filterAddedPlan =
MaterializedViewUtils.rewriteByRules(parentCascadesContext, context -> {
Rewriter.getWholeTreeRewriter(context).execute();
return context.getRewritePlan();
-}, queryPlanWithUnionFilter, queryPlan), true);
+}, queryPlanWithUnionFilter, queryPlan);
+// need to collect table partition again
Review Comment:
have added comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) Fix compensate union wrongly when direct query is empty relation [doris]
seawinde commented on code in PR #51700:
URL: https://github.com/apache/doris/pull/51700#discussion_r2149187612
##
fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java:
##
@@ -427,8 +413,13 @@ protected void rewrite(boolean showPlanProcess) {
statementContext.getConnectContext().getExecutor().getSummaryProfile().setNereidsRewriteTime();
}
// collect partitions table used, this is for query rewrite by
materialized view
-// this is needed before init hook
-collectTableUsedPartitions(showPlanProcess);
+// this is needed before init hook, because
+
MaterializedViewUtils.collectTableUsedPartitions(cascadesContext.getRewritePlan(),
cascadesContext);
Review Comment:
move it in hook
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) Fix compensate union wrongly when direct query is empty relation [doris]
seawinde commented on code in PR #51700:
URL: https://github.com/apache/doris/pull/51700#discussion_r2149185431
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/StructInfo.java:
##
@@ -752,10 +752,13 @@ public static Pair
addFilterOnTableScan(Plan queryPlan, Map {
+Plan filterAddedPlan =
MaterializedViewUtils.rewriteByRules(parentCascadesContext, context -> {
Rewriter.getWholeTreeRewriter(context).execute();
return context.getRewritePlan();
-}, queryPlanWithUnionFilter, queryPlan), true);
+}, queryPlanWithUnionFilter, queryPlan);
+// need to collect table partition again
+MaterializedViewUtils.collectTableUsedPartitions(filterAddedPlan,
parentCascadesContext);
Review Comment:
should collect table partition for all rewritten plan by mv, have fixed it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) Fix compensate union wrongly when direct query is empty relation [doris]
seawinde commented on PR #51700: URL: https://github.com/apache/doris/pull/51700#issuecomment-2975273005 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) Fix compensate union wrongly when direct query is empty relation [doris]
doris-robot commented on PR #51700: URL: https://github.com/apache/doris/pull/51700#issuecomment-2975069438 TPC-H: Total hot run time: 34052 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit b95c035a9960a976732c93b7cd96043e85c46e94, data reload: false -- Round 1 -- q1 17580 514550825082 q2 1925268 172 172 q3 10349 1222778 778 q4 10256 998 537 537 q5 7954239824192398 q6 182 164 137 137 q7 930 750 633 633 q8 9295130410571057 q9 6707507450965074 q10 6862231419221922 q11 486 286 277 277 q12 342 357 219 219 q13 17758 366930673067 q14 235 223 212 212 q15 569 483 484 483 q16 411 449 372 372 q17 580 847 367 367 q18 7499706071627060 q19 1366935 569 569 q20 341 341 232 232 q21 3794321424302430 q22 10731031974 974 Total cold run time: 106494 ms Total hot run time: 34052 ms - Round 2, with runtime_filter_mode=off - q1 5065505250275027 q2 232 315 219 219 q3 2193266822682268 q4 1416185213721372 q5 4178410142794101 q6 213 169 130 130 q7 2207191517731773 q8 2621274226182618 q9 7155710572467105 q10 3044312727152715 q11 568 517 492 492 q12 724 774 636 636 q13 3516386532823282 q14 276 306 276 276 q15 529 488 471 471 q16 438 496 449 449 q17 1160150814071407 q18 7714755175297529 q19 801 809 853 809 q20 1953210518701870 q21 5016437744844377 q22 1108106610321032 Total cold run time: 52127 ms Total hot run time: 49958 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) Fix compensate union wrongly when direct query is empty relation [doris]
seawinde commented on PR #51700: URL: https://github.com/apache/doris/pull/51700#issuecomment-2975024520 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) Fix compensate union wrongly when direct query is empty relation [doris]
hello-stephen commented on PR #51700: URL: https://github.com/apache/doris/pull/51700#issuecomment-2974992502 run external 5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) Fix compensate union wrongly when direct query is empty relation [doris]
doris-robot commented on PR #51700: URL: https://github.com/apache/doris/pull/51700#issuecomment-2969488194 TPC-H: Total hot run time: 36238 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 4a2a8c6edb79a886313aa0e6bd8ba0919b95bd81, data reload: false -- Round 1 -- q1 17664 548949904990 q2 2014313 189 189 q3 10238 1305831 831 q4 10220 1048572 572 q5 7527242824052405 q6 190 168 139 139 q7 958 786 657 657 q8 9384137313251325 q9 6900514751795147 q10 6874236319601960 q11 552 335 322 322 q12 391 385 243 243 q13 17793 380432893289 q14 258 257 227 227 q15 570 497 484 484 q16 464 479 412 412 q17 732 904 446 446 q18 8167732072347234 q19 1239991 632 632 q20 410 392 273 273 q21 4682406434293429 q22 1120106510321032 Total cold run time: 108347 ms Total hot run time: 36238 ms - Round 2, with runtime_filter_mode=off - q1 5103503950505039 q2 261 327 237 237 q3 2203275223312331 q4 1516187114311431 q5 4502441644864416 q6 225 172 135 135 q7 2111198018181818 q8 2682271126282628 q9 7367733571227122 q10 3103328328112811 q11 634 534 521 521 q12 734 821 661 661 q13 3735401834633463 q14 305 325 301 301 q15 528 492 492 492 q16 480 533 487 487 q17 1228155013951395 q18 7997774476787678 q19 942 106810461046 q20 2108211419471947 q21 5236477646974697 q22 1161108810861086 Total cold run time: 54161 ms Total hot run time: 51742 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) Fix compensate union wrongly when direct query is empty relation [doris]
hello-stephen commented on PR #51700: URL: https://github.com/apache/doris/pull/51700#issuecomment-2969424064 run external 10 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) Fix compensate union wrongly when direct query is empty relation [doris]
hello-stephen commented on PR #51700: URL: https://github.com/apache/doris/pull/51700#issuecomment-2969107664 # FE UT Coverage Report Increment line coverage ` 73.33% (11/15)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/51700_4a2a8c6edb79a886313aa0e6bd8ba0919b95bd81/fe_increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/51700_4a2a8c6edb79a886313aa0e6bd8ba0919b95bd81/fe_report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) Fix compensate union wrongly when direct query is empty relation [doris]
doris-robot commented on PR #51700: URL: https://github.com/apache/doris/pull/51700#issuecomment-2968986014 TPC-H: Total hot run time: 35905 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 4a2a8c6edb79a886313aa0e6bd8ba0919b95bd81, data reload: false -- Round 1 -- q1 17596 515450695069 q2 1994304 189 189 q3 10314 1312801 801 q4 10204 1058583 583 q5 7517237424362374 q6 203 169 140 140 q7 955 773 646 646 q8 9346137812021202 q9 6933512551955125 q10 7098238319821982 q11 540 336 328 328 q12 387 399 254 254 q13 17869 377532203220 q14 247 243 238 238 q15 586 495 503 495 q16 468 470 421 421 q17 719 897 451 451 q18 7805711474077114 q19 1442991 604 604 q20 411 382 262 262 q21 4516398833993399 q22 1141107210081008 Total cold run time: 108291 ms Total hot run time: 35905 ms - Round 2, with runtime_filter_mode=off - q1 5087502150765021 q2 263 328 241 241 q3 2215275423322332 q4 1490188414471447 q5 4394445744274427 q6 244 177 131 131 q7 2028203918451845 q8 2659266226472647 q9 7314736270727072 q10 3118319328672867 q11 630 537 522 522 q12 722 820 658 658 q13 3718399134223422 q14 308 308 310 308 q15 518 507 494 494 q16 494 560 476 476 q17 1244156714151415 q18 7826765076377637 q19 976 968 1153968 q20 2072210020122012 q21 5081483547154715 q22 1099110410591059 Total cold run time: 53500 ms Total hot run time: 51716 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) Fix compensate union wrongly when direct query is empty relation [doris]
morrySnow commented on code in PR #51700:
URL: https://github.com/apache/doris/pull/51700#discussion_r2144164437
##
fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java:
##
@@ -427,8 +413,13 @@ protected void rewrite(boolean showPlanProcess) {
statementContext.getConnectContext().getExecutor().getSummaryProfile().setNereidsRewriteTime();
}
// collect partitions table used, this is for query rewrite by
materialized view
-// this is needed before init hook
-collectTableUsedPartitions(showPlanProcess);
+// this is needed before init hook, because
+
MaterializedViewUtils.collectTableUsedPartitions(cascadesContext.getRewritePlan(),
cascadesContext);
Review Comment:
why not do it in hook?
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/StructInfo.java:
##
@@ -752,10 +752,13 @@ public static Pair
addFilterOnTableScan(Plan queryPlan, Map {
+Plan filterAddedPlan =
MaterializedViewUtils.rewriteByRules(parentCascadesContext, context -> {
Rewriter.getWholeTreeRewriter(context).execute();
return context.getRewritePlan();
-}, queryPlanWithUnionFilter, queryPlan), true);
+}, queryPlanWithUnionFilter, queryPlan);
+// need to collect table partition again
Review Comment:
write why need re-collect here in comment
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/StructInfo.java:
##
@@ -752,10 +752,13 @@ public static Pair
addFilterOnTableScan(Plan queryPlan, Map {
+Plan filterAddedPlan =
MaterializedViewUtils.rewriteByRules(parentCascadesContext, context -> {
Rewriter.getWholeTreeRewriter(context).execute();
return context.getRewritePlan();
-}, queryPlanWithUnionFilter, queryPlan), true);
+}, queryPlanWithUnionFilter, queryPlan);
+// need to collect table partition again
+MaterializedViewUtils.collectTableUsedPartitions(filterAddedPlan,
parentCascadesContext);
Review Comment:
if u need collect partition info multi times, u should sum the time used by
it in summary profile
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/exploration/mv/StructInfo.java:
##
@@ -752,10 +752,13 @@ public static Pair
addFilterOnTableScan(Plan queryPlan, Map {
+Plan filterAddedPlan =
MaterializedViewUtils.rewriteByRules(parentCascadesContext, context -> {
Rewriter.getWholeTreeRewriter(context).execute();
return context.getRewritePlan();
-}, queryPlanWithUnionFilter, queryPlan), true);
+}, queryPlanWithUnionFilter, queryPlan);
+// need to collect table partition again
+MaterializedViewUtils.collectTableUsedPartitions(filterAddedPlan,
parentCascadesContext);
Review Comment:
only collect for sub plan is enough?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) Fix compensate union wrongly when direct query is empty relation [doris]
seawinde commented on PR #51700: URL: https://github.com/apache/doris/pull/51700#issuecomment-2968934236 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](mtmv) Fix compensate union wrongly when direct query is empty relation [doris]
hello-stephen commented on PR #51700: URL: https://github.com/apache/doris/pull/51700#issuecomment-2968931407 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
