Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
924060929 merged PR #44586: URL: https://github.com/apache/doris/pull/44586 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
doris-robot commented on PR #44586: URL: https://github.com/apache/doris/pull/44586#issuecomment-2560804158 ClickBench: Total hot run time: 32.55 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit c57505e8881de7d113381649f1719360e5690243, data reload: false query1 0.030.030.03 query2 0.080.030.04 query3 0.240.070.07 query4 1.610.100.10 query5 0.440.430.40 query6 1.170.650.64 query7 0.020.020.01 query8 0.040.040.03 query9 0.560.510.51 query10 0.550.580.56 query11 0.140.100.10 query12 0.140.110.10 query13 0.600.600.61 query14 2.762.772.72 query15 0.890.820.82 query16 0.370.390.36 query17 1.041.041.05 query18 0.230.220.21 query19 1.981.881.97 query20 0.010.010.01 query21 15.36 0.590.58 query22 2.811.862.42 query23 16.93 0.930.77 query24 3.430.751.67 query25 0.230.140.22 query26 0.400.140.14 query27 0.060.040.04 query28 10.53 1.091.06 query29 12.58 3.303.26 query30 0.250.060.06 query31 2.850.390.39 query32 3.240.460.44 query33 3.103.083.15 query34 17.15 4.514.49 query35 4.504.494.48 query36 0.680.480.50 query37 0.090.060.05 query38 0.040.040.04 query39 0.040.020.02 query40 0.170.130.14 query41 0.090.020.02 query42 0.040.020.02 query43 0.040.030.03 Total cold run time: 107.51 s Total hot run time: 32.55 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
doris-robot commented on PR #44586: URL: https://github.com/apache/doris/pull/44586#issuecomment-2560798837 TPC-DS: Total hot run time: 190328 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit c57505e8881de7d113381649f1719360e5690243, data reload: false query1 975 390 372 372 query2 6517235323572353 query3 6716215 215 215 query4 33880 23543 23966 23543 query5 4293459 472 459 query6 293 192 192 192 query7 4629321 325 321 query8 308 244 238 238 query9 9426278227712771 query10 478 267 250 250 query11 18338 15150 15123 15123 query12 159 107 111 107 query13 1731423 435 423 query14 10955 672269736722 query15 294 178 182 178 query16 8278460 472 460 query17 1738586 575 575 query18 2158314 302 302 query19 389 150 150 150 query20 113 101 109 101 query21 205 99 105 99 query22 4651438045424380 query23 34264 33056 33235 33056 query24 11430 245124292429 query25 674 376 406 376 query26 1755156 153 153 query27 2951335 332 332 query28 8146246724932467 query29 1027420 415 415 query30 307 148 160 148 query31 1014810 784 784 query32 95 59 60 59 query33 777 314 285 285 query34 967 514 503 503 query35 950 725 730 725 query36 1075944 960 944 query37 291 77 79 77 query38 4164435640334033 query39 1447145614271427 query40 281 102 103 102 query41 48 47 47 47 query42 116 101 101 101 query43 524 498 492 492 query44 1251815 839 815 query45 183 158 164 158 query46 1142716 714 714 query47 1937183818551838 query48 403 330 313 313 query49 1298410 385 385 query50 810 382 384 382 query51 7221711970227022 query52 100 96 88 88 query53 257 185 185 185 query54 1174410 412 410 query55 78 78 83 78 query56 254 251 237 237 query57 1243112810941094 query58 232 236 243 236 query59 3244304329672967 query60 285 244 249 244 query61 113 149 109 109 query62 933 728 738 728 query63 219 185 188 185 query64 5077668 705 668 query65 3237318032163180 query66 1415324 302 302 query67 15817 15526 15601 15526 query68 5869578 556 556 query69 420 253 248 248 query70 1186116411491149 query71 450 279 243 243 query72 6552415540074007 query73 787 361 363 361 query74 10415 911691279116 query75 3470268426922684 query76 3561106910841069 query77 554 283 282 282 query78 10420 952494169416 query79 2430609 613 609 query80 1094419 493 419 query81 532 232 225 225 query82 944 119 122 119 query83 254 155 144 144 query84 243 73 68 68 query85 1348309 312 309 query86 442 302 303 302 query87 4446426843434268 query88 4064222822752228 query89 419 295 303 295 query90 2134188 190 188 query91 142 111 109 109 query92 63 52 52 52 query93 1758563 563 563 query94 1124285 301 285 query95 354 254 263 254 query96 609 279 285 279 query97 2856269226852685 query98 226 201 202 201 query99 1677144214591442 Total cold run time: 307593 ms Total hot run time: 190328 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
doris-robot commented on PR #44586: URL: https://github.com/apache/doris/pull/44586#issuecomment-2560787578 TPC-H: Total hot run time: 39820 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit c57505e8881de7d113381649f1719360e5690243, data reload: false -- Round 1 -- q1 17691 744272827282 q2 2051179 165 165 q3 10619 111811041104 q4 10220 715 725 715 q5 7606268326592659 q6 237 151 146 146 q7 997 620 598 598 q8 9259185019511850 q9 6608648564566456 q10 7033228923432289 q11 466 273 271 271 q12 429 218 223 218 q13 17780 293329582933 q14 241 209 210 209 q15 557 500 501 500 q16 659 576 566 566 q17 973 590 543 543 q18 7296674867416741 q19 13361033872 872 q20 453 181 188 181 q21 3963329232073207 q22 369 315 317 315 Total cold run time: 106843 ms Total hot run time: 39820 ms - Round 2, with runtime_filter_mode=off - q1 7297721076277210 q2 327 230 231 230 q3 2875277030332770 q4 2109189118091809 q5 5526557355915573 q6 226 142 140 140 q7 2194177017591759 q8 3377355134813481 q9 8900889688818881 q10 3681357135003500 q11 604 500 497 497 q12 810 586 584 584 q13 12449 308629682968 q14 298 269 296 269 q15 555 491 518 491 q16 698 648 659 648 q17 1802156715571557 q18 7811759976727599 q19 1659143914921439 q20 2069181218241812 q21 5371523652745236 q22 620 606 584 584 Total cold run time: 71258 ms Total hot run time: 59037 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
924060929 commented on PR #44586: URL: https://github.com/apache/doris/pull/44586#issuecomment-2560769639 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
github-actions[bot] commented on PR #44586: URL: https://github.com/apache/doris/pull/44586#issuecomment-2559313184 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
doris-robot commented on PR #44586: URL: https://github.com/apache/doris/pull/44586#issuecomment-2559255639 ClickBench: Total hot run time: 32.06 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit c57505e8881de7d113381649f1719360e5690243, data reload: false query1 0.040.040.03 query2 0.080.030.04 query3 0.230.070.07 query4 1.620.100.11 query5 0.430.400.41 query6 1.160.650.65 query7 0.020.020.02 query8 0.040.030.03 query9 0.560.500.49 query10 0.570.560.57 query11 0.150.100.11 query12 0.140.110.11 query13 0.620.620.61 query14 2.732.722.75 query15 0.900.820.83 query16 0.370.390.39 query17 1.061.071.04 query18 0.230.210.22 query19 1.951.822.00 query20 0.020.020.01 query21 15.37 0.600.60 query22 2.631.751.68 query23 17.03 1.080.83 query24 2.601.620.27 query25 0.230.200.26 query26 0.300.140.14 query27 0.050.050.05 query28 11.31 1.101.06 query29 12.59 3.243.27 query30 0.250.060.06 query31 2.860.390.38 query32 3.240.480.47 query33 3.103.163.13 query34 17.09 4.504.45 query35 4.554.564.51 query36 0.670.470.48 query37 0.090.060.06 query38 0.040.040.03 query39 0.030.020.02 query40 0.180.130.13 query41 0.080.020.03 query42 0.040.020.02 query43 0.040.040.03 Total cold run time: 107.29 s Total hot run time: 32.06 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
doris-robot commented on PR #44586: URL: https://github.com/apache/doris/pull/44586#issuecomment-2559246156 TPC-DS: Total hot run time: 190521 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit c57505e8881de7d113381649f1719360e5690243, data reload: false query1 957 376 370 370 query2 6526241423652365 query3 6709209 212 209 query4 33941 23348 23348 23348 query5 4419462 460 460 query6 277 198 184 184 query7 4621307 305 305 query8 297 231 236 231 query9 9486274627472746 query10 469 284 249 249 query11 18045 15099 15107 15099 query12 149 111 106 106 query13 1671457 435 435 query14 9923785568566856 query15 271 179 190 179 query16 8117448 452 448 query17 1627602 575 575 query18 1735309 315 309 query19 365 164 162 162 query20 126 110 113 110 query21 215 107 104 104 query22 4507432241694169 query23 34588 34097 33877 33877 query24 11432 246325002463 query25 668 381 386 381 query26 1776150 151 150 query27 2642324 326 324 query28 7908244024592440 query29 978 403 410 403 query30 302 153 151 151 query31 1013782 833 782 query32 101 59 57 57 query33 774 295 286 286 query34 1024520 516 516 query35 902 740 757 740 query36 1082985 962 962 query37 267 83 78 78 query38 4085423640504050 query39 1464148214201420 query40 283 109 105 105 query41 51 46 50 46 query42 113 103 107 103 query43 554 511 500 500 query44 1195799 804 799 query45 183 172 168 168 query46 1184706 726 706 query47 1985183618801836 query48 409 319 315 315 query49 1275391 387 387 query50 831 381 401 381 query51 7224698869266926 query52 105 96 88 88 query53 260 184 189 184 query54 1313413 415 413 query55 85 75 80 75 query56 260 241 236 236 query57 1245112711011101 query58 237 238 220 220 query59 3266304129332933 query60 275 250 243 243 query61 113 106 105 105 query62 904 672 683 672 query63 218 189 191 189 query64 5265670 668 668 query65 3246320232373202 query66 1384307 320 307 query67 15880 15574 15478 15478 query68 5722569 606 569 query69 435 250 249 249 query70 1213112911311129 query71 392 262 258 258 query72 6325413142354131 query73 781 372 368 368 query74 10238 880488828804 query75 3407267326772673 query76 3553113011151115 query77 548 273 277 273 query78 10132 940796699407 query79 1472624 605 605 query80 1011435 454 435 query81 522 229 230 229 query82 804 126 119 119 query83 240 155 144 144 query84 243 75 76 75 query85 1294314 301 301 query86 356 297 306 297 query87 4730446844284428 query88 3472226922362236 query89 417 301 305 301 query90 2147193 187 187 query91 144 111 107 107 query92 67 51 54 51 query93 1132559 548 548 query94 1091295 288 288 query95 349 267 258 258 query96 609 287 281 281 query97 2878269826672667 query98 214 197 195 195 query99 1584131713081308 Total cold run time: 302324 ms Total hot run time: 190521 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t
Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
doris-robot commented on PR #44586: URL: https://github.com/apache/doris/pull/44586#issuecomment-2559226245 TPC-H: Total hot run time: 40120 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit c57505e8881de7d113381649f1719360e5690243, data reload: false -- Round 1 -- q1 17606 742673867386 q2 2047178 173 173 q3 10555 115411541154 q4 10238 674 700 674 q5 7575271426602660 q6 238 153 152 152 q7 982 639 613 613 q8 9267182019191820 q9 6524643564866435 q10 6972232423612324 q11 459 255 265 255 q12 415 228 226 226 q13 17754 290429582904 q14 269 215 218 215 q15 544 512 498 498 q16 670 618 605 605 q17 987 656 542 542 q18 7285667266656665 q19 1350998 1047998 q20 497 180 179 179 q21 4481339533323332 q22 373 310 316 310 Total cold run time: 107088 ms Total hot run time: 40120 ms - Round 2, with runtime_filter_mode=off - q1 7285724872367236 q2 332 230 235 230 q3 2887277130652771 q4 2122194219101910 q5 5588560856515608 q6 225 139 143 139 q7 2233179418101794 q8 3377353434913491 q9 8977885789198857 q10 3603355335363536 q11 611 505 508 505 q12 814 653 614 614 q13 13181 305731243057 q14 314 266 269 266 q15 557 501 504 501 q16 676 624 642 624 q17 1818161115861586 q18 8050725473797254 q19 1688149915871499 q20 2071181018491810 q21 5476527153015271 q22 616 578 565 565 Total cold run time: 72501 ms Total hot run time: 59124 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
924060929 commented on PR #44586: URL: https://github.com/apache/doris/pull/44586#issuecomment-2559151503 > should we add some UTs? the exists UT/RT is enough and it already made me find some bugs -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
924060929 commented on PR #44586: URL: https://github.com/apache/doris/pull/44586#issuecomment-2559151736 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
github-actions[bot] commented on PR #44586: URL: https://github.com/apache/doris/pull/44586#issuecomment-2559115405 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
morrySnow commented on code in PR #44586:
URL: https://github.com/apache/doris/pull/44586#discussion_r1895308368
##
fe/fe-core/src/main/java/org/apache/doris/common/cache/NereidsSortedPartitionsCacheManager.java:
##
@@ -0,0 +1,191 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.common.cache;
+
+import org.apache.doris.catalog.DatabaseIf;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.catalog.PartitionInfo;
+import org.apache.doris.catalog.PartitionItem;
+import org.apache.doris.common.Config;
+import org.apache.doris.common.ConfigBase.DefaultConfHandler;
+import org.apache.doris.datasource.CatalogIf;
+import org.apache.doris.nereids.rules.expression.rules.MultiColumnBound;
+import org.apache.doris.nereids.rules.expression.rules.PartitionItemToRange;
+import org.apache.doris.nereids.rules.expression.rules.SortedPartitionRanges;
+import
org.apache.doris.nereids.rules.expression.rules.SortedPartitionRanges.PartitionItemAndId;
+import
org.apache.doris.nereids.rules.expression.rules.SortedPartitionRanges.PartitionItemAndRange;
+
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import com.google.common.collect.Range;
+import lombok.AllArgsConstructor;
+import lombok.Data;
+import org.apache.hadoop.util.Lists;
+
+import java.lang.reflect.Field;
+import java.time.Duration;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Optional;
+
+/** NereidsSortedPartitionsCacheManager */
+public class NereidsSortedPartitionsCacheManager {
+private volatile Cache
partitionCaches;
+
+public NereidsSortedPartitionsCacheManager() {
+partitionCaches = buildCaches(
+Config.cache_partition_meta_table_manage_num,
+Config.expire_cache_partition_meta_table_in_fe_second
+);
+}
+
+public Optional> get(OlapTable table) {
+DatabaseIf database = table.getDatabase();
+if (database == null) {
+return Optional.empty();
+}
+CatalogIf catalog = database.getCatalog();
+if (catalog == null) {
+return Optional.empty();
+}
+TableIdentifier key = new TableIdentifier(
+catalog.getName(), database.getFullName(), table.getName());
+PartitionCacheContext partitionCacheContext =
partitionCaches.getIfPresent(key);
+if (partitionCacheContext == null) {
+return Optional.of(loadCache(key, table));
+}
+if (table.getId() != partitionCacheContext.tableId
+|| table.getVisibleVersion() !=
partitionCacheContext.tableVersion) {
+partitionCaches.invalidate(key);
+return Optional.empty();
Review Comment:
why not load again?
##
fe/fe-core/src/main/java/org/apache/doris/common/cache/NereidsSortedPartitionsCacheManager.java:
##
@@ -0,0 +1,191 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.common.cache;
+
+import org.apache.doris.catalog.DatabaseIf;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.catalog.PartitionInfo;
+import org.apache.doris.catalog.PartitionItem;
+import org.apache.doris.common.Config;
+import org.apache.doris.common.ConfigBase.DefaultConfHandler;
+import org.apache.doris.datasourc
Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
doris-robot commented on PR #44586: URL: https://github.com/apache/doris/pull/44586#issuecomment-2558079159 ClickBench: Total hot run time: 33.04 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 9bb2e041435b28a6b0bd9b7532295f2fd4d98fa8, data reload: false query1 0.030.030.03 query2 0.070.040.04 query3 0.230.080.07 query4 1.610.100.10 query5 0.430.410.42 query6 1.140.650.65 query7 0.020.020.02 query8 0.050.030.03 query9 0.570.510.51 query10 0.540.600.55 query11 0.150.110.10 query12 0.130.110.12 query13 0.610.600.60 query14 2.732.722.81 query15 0.890.830.82 query16 0.400.410.40 query17 1.061.071.03 query18 0.230.210.22 query19 1.981.812.03 query20 0.010.010.01 query21 15.36 0.610.56 query22 2.871.741.85 query23 17.11 0.860.85 query24 3.221.441.42 query25 0.250.070.19 query26 0.520.130.13 query27 0.050.040.03 query28 9.801.101.07 query29 12.56 3.233.23 query30 0.250.060.05 query31 2.860.390.39 query32 3.260.460.47 query33 3.083.113.15 query34 16.77 4.454.48 query35 4.564.494.52 query36 0.680.500.48 query37 0.100.060.05 query38 0.050.040.03 query39 0.030.020.03 query40 0.180.130.12 query41 0.080.030.02 query42 0.040.020.02 query43 0.030.030.03 Total cold run time: 106.59 s Total hot run time: 33.04 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
doris-robot commented on PR #44586: URL: https://github.com/apache/doris/pull/44586#issuecomment-2558077792 TPC-DS: Total hot run time: 190276 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 9bb2e041435b28a6b0bd9b7532295f2fd4d98fa8, data reload: false query1 961 390 375 375 query2 6526241423322332 query3 6717218 232 218 query4 33746 23316 23720 23316 query5 4409476 468 468 query6 305 213 190 190 query7 4631304 310 304 query8 308 247 240 240 query9 9341272026902690 query10 446 245 238 238 query11 17981 15138 15228 15138 query12 172 100 100 100 query13 1671406 406 406 query14 11643 728068736873 query15 315 180 192 180 query16 8059417 443 417 query17 1641577 556 556 query18 2049294 290 290 query19 370 147 146 146 query20 119 108 110 108 query21 209 103 108 103 query22 4633439242874287 query23 34486 33619 33507 33507 query24 11197 251824832483 query25 734 386 411 386 query26 1801149 153 149 query27 2954328 336 328 query28 8115242324322423 query29 1031403 447 403 query30 296 148 150 148 query31 1049821 819 819 query32 98 56 61 56 query33 775 298 291 291 query34 994 519 498 498 query35 900 734 733 733 query36 1110943 934 934 query37 271 78 81 78 query38 4286414042024140 query39 1511143214941432 query40 286 100 100 100 query41 48 43 48 43 query42 114 104 97 97 query43 539 495 504 495 query44 1255837 818 818 query45 191 173 166 166 query46 1237697 702 697 query47 1947183318451833 query48 427 318 326 318 query49 1282410 398 398 query50 832 385 385 385 query51 7217702969316931 query52 109 96 98 96 query53 266 186 194 186 query54 1318413 429 413 query55 82 79 80 79 query56 262 236 231 231 query57 1275112511181118 query58 247 220 224 220 query59 3196297530032975 query60 285 260 245 245 query61 114 108 110 108 query62 908 666 682 666 query63 219 191 194 191 query64 5186664 654 654 query65 3301319132703191 query66 1386313 316 313 query67 15905 15652 15592 15592 query68 5172558 566 558 query69 453 259 263 259 query70 1214113010961096 query71 454 256 252 252 query72 6322421840644064 query73 821 356 368 356 query74 10470 880288758802 query75 3632269126892689 query76 3003115510841084 query77 534 279 281 279 query78 10254 991694239423 query79 2316616 614 614 query80 882 434 443 434 query81 536 245 242 242 query82 674 117 122 117 query83 241 149 157 149 query84 241 77 82 77 query85 1324377 388 377 query86 434 302 294 294 query87 4647475843944394 query88 4105221321892189 query89 420 291 304 291 query90 2202191 189 189 query91 137 105 107 105 query92 68 54 52 52 query93 1301545 550 545 query94 1053282 282 282 query95 355 251 256 251 query96 667 280 287 280 query97 2834270326702670 query98 221 199 220 199 query99 1589140213211321 Total cold run time: 305870 ms Total hot run time: 190276 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
doris-robot commented on PR #44586: URL: https://github.com/apache/doris/pull/44586#issuecomment-2558075254 TPC-H: Total hot run time: 40369 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 9bb2e041435b28a6b0bd9b7532295f2fd4d98fa8, data reload: false -- Round 1 -- q1 17630 744073377337 q2 2061179 169 169 q3 10545 115411811154 q4 10236 702 747 702 q5 7622274826962696 q6 242 156 152 152 q7 1009641 613 613 q8 9265191019961910 q9 6633645864806458 q10 6991231623252316 q11 473 264 271 264 q12 425 225 229 225 q13 17783 296829582958 q14 246 223 229 223 q15 552 508 495 495 q16 661 592 609 592 q17 991 598 527 527 q18 7555679568276795 q19 13581015973 973 q20 481 178 181 178 q21 4398337833243324 q22 387 308 319 308 Total cold run time: 107544 ms Total hot run time: 40369 ms - Round 2, with runtime_filter_mode=off - q1 7590723772057205 q2 329 235 231 231 q3 2941283328132813 q4 1975173716861686 q5 5374541454095409 q6 216 137 140 137 q7 2158169016691669 q8 3250340534223405 q9 8646868686808680 q10 3502349634503450 q11 602 500 514 500 q12 798 568 591 568 q13 6 298029782978 q14 308 260 265 260 q15 560 496 497 496 q16 669 656 631 631 q17 1827158716161587 q18 7886750574317431 q19 1719157814921492 q20 2074178818401788 q21 5547535852435243 q22 624 587 605 587 Total cold run time: 69711 ms Total hot run time: 58246 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
924060929 commented on PR #44586: URL: https://github.com/apache/doris/pull/44586#issuecomment-2558070726 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
doris-robot commented on PR #44586: URL: https://github.com/apache/doris/pull/44586#issuecomment-2557223558 ClickBench: Total hot run time: 33.1 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 197aa64cd0977e9b1a84031c86a154153f893ac2, data reload: false query1 0.040.030.03 query2 0.080.030.04 query3 0.230.070.07 query4 1.600.110.10 query5 0.440.420.41 query6 1.180.640.64 query7 0.020.020.01 query8 0.040.030.03 query9 0.570.500.50 query10 0.540.580.55 query11 0.150.110.10 query12 0.140.110.10 query13 0.610.610.59 query14 2.712.782.86 query15 0.900.830.82 query16 0.400.380.39 query17 1.061.031.08 query18 0.220.210.21 query19 1.901.912.03 query20 0.010.010.01 query21 15.42 0.600.59 query22 2.972.172.37 query23 17.07 0.800.88 query24 3.360.841.87 query25 0.230.160.16 query26 0.480.130.14 query27 0.040.050.04 query28 9.711.101.07 query29 12.59 3.313.30 query30 0.250.060.06 query31 2.860.390.38 query32 3.250.480.46 query33 3.113.023.08 query34 16.86 4.464.56 query35 4.544.524.52 query36 0.680.480.51 query37 0.100.060.06 query38 0.050.040.04 query39 0.030.020.02 query40 0.170.130.13 query41 0.080.020.03 query42 0.040.020.02 query43 0.030.030.03 Total cold run time: 106.76 s Total hot run time: 33.1 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
doris-robot commented on PR #44586: URL: https://github.com/apache/doris/pull/44586#issuecomment-2557214108 TPC-DS: Total hot run time: 190054 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 197aa64cd0977e9b1a84031c86a154153f893ac2, data reload: false query1 960 412 380 380 query2 6510237123852371 query3 6710238 219 219 query4 34000 23230 23310 23230 query5 4351470 455 455 query6 272 193 207 193 query7 4632303 304 303 query8 306 242 237 237 query9 9597270427022702 query10 478 261 250 250 query11 17874 15346 15198 15198 query12 158 108 107 107 query13 1676432 425 425 query14 10220 720068696869 query15 312 194 205 194 query16 8337451 472 451 query17 1686608 575 575 query18 2149327 305 305 query19 367 150 151 150 query20 125 111 109 109 query21 211 103 129 103 query22 4663445242844284 query23 34402 33579 33596 33579 query24 11477 258124202420 query25 674 375 394 375 query26 1769155 148 148 query27 2830320 333 320 query28 8025241124092409 query29 1028409 428 409 query30 298 146 146 146 query31 1046787 811 787 query32 92 57 59 57 query33 775 293 308 293 query34 973 508 524 508 query35 904 736 746 736 query36 1091924 913 913 query37 296 79 80 79 query38 4373411742714117 query39 1509143414341434 query40 276 102 100 100 query41 46 43 45 43 query42 107 99 99 99 query43 530 491 480 480 query44 1235813 810 810 query45 184 168 165 165 query46 1161718 695 695 query47 1969183418981834 query48 399 312 314 312 query49 1300402 382 382 query50 813 385 388 385 query51 7100708168756875 query52 105 91 88 88 query53 253 184 181 181 query54 1296406 414 406 query55 83 82 81 81 query56 256 237 234 234 query57 1266112911441129 query58 243 217 223 217 query59 3251301629262926 query60 282 248 245 245 query61 118 112 113 112 query62 883 680 684 680 query63 219 186 191 186 query64 5071677 699 677 query65 3246318032383180 query66 1467314 308 308 query67 15790 15633 15387 15387 query68 5091545 573 545 query69 434 250 255 250 query70 1210114411451144 query71 438 249 249 249 query72 6458421740884088 query73 779 367 358 358 query74 9578900388848884 query75 3457260326412603 query76 3085103011061030 query77 547 274 280 274 query78 11791 10348 94939493 query79 1332618 597 597 query80 853 446 439 439 query81 524 295 239 239 query82 673 117 127 117 query83 241 149 143 143 query84 233 70 76 70 query85 787 309 307 307 query86 373 300 275 275 query87 4511461644914491 query88 3382223721972197 query89 464 289 297 289 query90 2091196 190 190 query91 144 107 106 106 query92 65 52 54 52 query93 1094542 542 542 query94 919 287 296 287 query95 357 258 260 258 query96 622 280 284 280 query97 2840269027282690 query98 218 199 195 195 query99 1544131513311315 Total cold run time: 302240 ms Total hot run time: 190054 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
doris-robot commented on PR #44586: URL: https://github.com/apache/doris/pull/44586#issuecomment-2557192467 TPC-H: Total hot run time: 40019 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 197aa64cd0977e9b1a84031c86a154153f893ac2, data reload: false -- Round 1 -- q1 17587 743572697269 q2 2063186 191 186 q3 10521 117111971171 q4 10222 783 770 770 q5 7597273226602660 q6 242 147 148 147 q7 1006624 630 624 q8 9254186619171866 q9 6724649964616461 q10 7042230323192303 q11 471 260 258 258 q12 435 232 225 225 q13 17798 297729722972 q14 245 212 215 212 q15 548 516 494 494 q16 665 588 582 582 q17 1012490 605 490 q18 7457678766076607 q19 1346964 934 934 q20 477 190 190 190 q21 4027333532783278 q22 377 320 327 320 Total cold run time: 107116 ms Total hot run time: 40019 ms - Round 2, with runtime_filter_mode=off - q1 7222719972167199 q2 328 238 230 230 q3 2959274130592741 q4 2146192718221822 q5 5655569757075697 q6 227 139 138 138 q7 2235180517961796 q8 3402361635163516 q9 8961896690568966 q10 3610358935463546 q11 594 510 498 498 q12 828 632 590 590 q13 13912 309330673067 q14 299 290 289 289 q15 554 493 504 493 q16 692 624 634 624 q17 1800158316041583 q18 7961745175937451 q19 1716157015381538 q20 2063180418231804 q21 5432523752975237 q22 658 576 563 563 Total cold run time: 73254 ms Total hot run time: 59388 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
924060929 commented on PR #44586: URL: https://github.com/apache/doris/pull/44586#issuecomment-2557125229 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
doris-robot commented on PR #44586: URL: https://github.com/apache/doris/pull/44586#issuecomment-2556529920 ClickBench: Total hot run time: 33.33 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 264f4363ae65f0256a42b0160f7e793b7af599d5, data reload: false query1 0.030.030.03 query2 0.080.030.03 query3 0.230.080.07 query4 1.600.110.11 query5 0.440.410.40 query6 1.170.660.64 query7 0.020.010.01 query8 0.040.030.03 query9 0.610.520.51 query10 0.550.590.56 query11 0.150.110.11 query12 0.130.110.11 query13 0.600.610.59 query14 2.872.872.80 query15 0.890.840.83 query16 0.390.370.38 query17 1.041.031.03 query18 0.210.200.21 query19 1.921.791.93 query20 0.010.020.01 query21 15.38 0.570.58 query22 2.642.562.42 query23 17.05 0.880.79 query24 2.910.971.23 query25 0.260.230.07 query26 0.420.160.13 query27 0.050.050.04 query28 10.65 1.101.07 query29 12.56 3.293.28 query30 0.250.060.06 query31 2.870.380.39 query32 3.230.480.46 query33 3.073.123.12 query34 16.97 4.474.52 query35 4.504.474.54 query36 0.660.490.48 query37 0.100.060.06 query38 0.050.040.04 query39 0.030.030.03 query40 0.160.120.12 query41 0.080.030.02 query42 0.030.020.03 query43 0.030.030.03 Total cold run time: 106.93 s Total hot run time: 33.33 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
doris-robot commented on PR #44586: URL: https://github.com/apache/doris/pull/44586#issuecomment-2556521762 TPC-DS: Total hot run time: 196789 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 264f4363ae65f0256a42b0160f7e793b7af599d5, data reload: false query1 13131000906 906 query2 6245240023182318 query3 2 485847244724 query4 32801 23423 23287 23287 query5 3652457 460 457 query6 285 191 190 190 query7 3996300 306 300 query8 311 237 239 237 query9 9512272127222721 query10 462 250 255 250 query11 17731 15224 15127 15127 query12 178 107 101 101 query13 1601424 412 412 query14 9081754473677367 query15 265 204 192 192 query16 8204476 453 453 query17 1696567 573 567 query18 2035298 319 298 query19 407 156 159 156 query20 122 116 115 115 query21 205 112 104 104 query22 4759473244354435 query23 34423 33718 33682 33682 query24 9991257025202520 query25 593 386 408 386 query26 706 159 157 157 query27 2139331 336 331 query28 7220249124832483 query29 739 403 414 403 query30 245 147 152 147 query31 1066859 840 840 query32 94 55 58 55 query33 670 360 294 294 query34 974 515 524 515 query35 884 755 761 755 query36 1130966 950 950 query37 138 76 71 71 query38 4198429341574157 query39 1511148014521452 query40 206 108 101 101 query41 44 42 43 42 query42 114 105 130 105 query43 535 492 492 492 query44 1203815 832 815 query45 190 173 166 166 query46 1181735 731 731 query47 2077191319481913 query48 412 333 324 324 query49 973 382 388 382 query50 841 419 408 408 query51 7314728372677267 query52 104 95 98 95 query53 255 187 184 184 query54 1291420 420 420 query55 83 75 88 75 query56 269 238 254 238 query57 1317118511661166 query58 254 230 234 230 query59 3394305931283059 query60 280 246 260 246 query61 106 105 128 105 query62 880 686 721 686 query63 223 190 194 190 query64 3687695 654 654 query65 3294321132783211 query66 771 316 319 316 query67 16585 15599 15563 15563 query68 4318551 557 551 query69 428 261 248 248 query70 1215111611531116 query71 395 252 245 245 query72 6559415740814081 query73 777 360 362 360 query74 9690881389068813 query75 3404265626212621 query76 2798108111441081 query77 436 277 282 277 query78 10198 939493229322 query79 1098599 590 590 query80 700 429 436 429 query81 508 238 235 235 query82 245 120 120 120 query83 163 143 142 142 query84 246 83 76 76 query85 923 377 374 374 query86 361 283 287 283 query87 4468453143974397 query88 3576225422292229 query89 424 294 290 290 query90 2117193 183 183 query91 138 101 104 101 query92 65 50 54 50 query93 1373547 537 537 query94 812 280 286 280 query95 357 256 252 252 query96 621 279 280 279 query97 2862261326612613 query98 220 205 193 193 query99 1571135813221322 Total cold run time: 293643 ms Total hot run time: 196789 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go
Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
doris-robot commented on PR #44586: URL: https://github.com/apache/doris/pull/44586#issuecomment-2556504127 TPC-H: Total hot run time: 39878 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 264f4363ae65f0256a42b0160f7e793b7af599d5, data reload: false -- Round 1 -- q1 17714 752073317331 q2 2061181 171 171 q3 10631 109111921091 q4 10561 754 702 702 q5 7623276326802680 q6 246 147 144 144 q7 981 617 593 593 q8 9247186519561865 q9 6648643664276427 q10 6980231023452310 q11 477 252 261 252 q12 424 227 223 223 q13 17758 300529282928 q14 251 223 219 219 q15 563 502 511 502 q16 645 608 595 595 q17 986 523 519 519 q18 7295668467516684 q19 13351013989 989 q20 459 180 184 180 q21 4147319731583158 q22 375 329 315 315 Total cold run time: 107407 ms Total hot run time: 39878 ms - Round 2, with runtime_filter_mode=off - q1 7264747672737273 q2 332 228 232 228 q3 2928280728992807 q4 2086184118461841 q5 5654564056545640 q6 228 147 152 147 q7 2244181117991799 q8 3391358334983498 q9 8932895990098959 q10 3618357535273527 q11 611 539 501 501 q12 845 621 597 597 q13 11924 316731153115 q14 317 279 269 269 q15 573 508 516 508 q16 678 633 666 633 q17 1869162916191619 q18 8395785577837783 q19 1730164714671467 q20 2032186818751868 q21 5637561054265426 q22 632 613 602 602 Total cold run time: 71920 ms Total hot run time: 60107 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
924060929 commented on PR #44586: URL: https://github.com/apache/doris/pull/44586#issuecomment-255648 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
924060929 commented on PR #44586: URL: https://github.com/apache/doris/pull/44586#issuecomment-2556407765 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
924060929 commented on code in PR #44586:
URL: https://github.com/apache/doris/pull/44586#discussion_r1891044459
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/MultiColumnBound.java:
##
@@ -0,0 +1,54 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.expression.rules;
+
+import java.util.List;
+
+/** MultiColumnBound */
+public class MultiColumnBound implements Comparable {
+private final List columnBounds;
+
+public MultiColumnBound(List columnBounds) {
+this.columnBounds = columnBounds;
+}
+
+@Override
+public int compareTo(MultiColumnBound o) {
+for (int i = 0; i < columnBounds.size(); i++) {
+ColumnBound columnBound = columnBounds.get(i);
+ColumnBound otherColumnBound = o.columnBounds.get(i);
Review Comment:
not need，the size always equals because the PartitionPredicateToRange align
it
https://github.com/apache/doris/pull/44586/files#diff-73a2f40b7fe2c26d3a80ccbe8b1e5152a4d961bc7b5e045b94640327377e1506R247-R266
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
924060929 commented on code in PR #44586:
URL: https://github.com/apache/doris/pull/44586#discussion_r1891044459
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/MultiColumnBound.java:
##
@@ -0,0 +1,54 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.expression.rules;
+
+import java.util.List;
+
+/** MultiColumnBound */
+public class MultiColumnBound implements Comparable {
+private final List columnBounds;
+
+public MultiColumnBound(List columnBounds) {
+this.columnBounds = columnBounds;
+}
+
+@Override
+public int compareTo(MultiColumnBound o) {
+for (int i = 0; i < columnBounds.size(); i++) {
+ColumnBound columnBound = columnBounds.get(i);
+ColumnBound otherColumnBound = o.columnBounds.get(i);
Review Comment:
not need，the size always equals because the PartitionPredicateToRange align
it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
github-actions[bot] commented on PR #44586:
URL: https://github.com/apache/doris/pull/44586#issuecomment-2551786369
`sh-checker report`
To get the full details, please check in the
[job]("https://github.com/apache/doris/actions/runs/12397305848";) output.
shellcheck errors
```
'shellcheck ' returned error 1 finding the following syntactical issues:
--
In gensrc/script/gen_build_version.sh line 38:
if [[ ${build_version_hotfix} > 0 ]]; then
^-- SC2071 (error): > is for string
comparisons. Use -gt instead.
In gensrc/script/gen_build_version.sh line 228:
if [[ ${build_version_hotfix} > 0 ]]; then
^-- SC2071 (error): > is for string
comparisons. Use -gt instead.
For more information:
https://www.shellcheck.net/wiki/SC2071 -- > is for string comparisons. Use
...
--
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=
above the line that contains the issue, where is the error code;
3. Add '-e ' to the SHELLCHECK_OPTS setting in your .yml action file.
```
shfmt errors
```
'shfmt ' found no issues.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
924060929 commented on PR #44586: URL: https://github.com/apache/doris/pull/44586#issuecomment-2551785776 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
github-actions[bot] commented on PR #44586:
URL: https://github.com/apache/doris/pull/44586#issuecomment-2551295757
`sh-checker report`
To get the full details, please check in the
[job]("https://github.com/apache/doris/actions/runs/12393726562";) output.
shellcheck errors
```
'shellcheck ' returned error 1 finding the following syntactical issues:
--
In gensrc/script/gen_build_version.sh line 38:
if [[ ${build_version_hotfix} > 0 ]]; then
^-- SC2071 (error): > is for string
comparisons. Use -gt instead.
In gensrc/script/gen_build_version.sh line 228:
if [[ ${build_version_hotfix} > 0 ]]; then
^-- SC2071 (error): > is for string
comparisons. Use -gt instead.
For more information:
https://www.shellcheck.net/wiki/SC2071 -- > is for string comparisons. Use
...
--
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=
above the line that contains the issue, where is the error code;
3. Add '-e ' to the SHELLCHECK_OPTS setting in your .yml action file.
```
shfmt errors
```
'shfmt ' found no issues.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
924060929 commented on PR #44586: URL: https://github.com/apache/doris/pull/44586#issuecomment-2551294599 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
github-actions[bot] commented on PR #44586:
URL: https://github.com/apache/doris/pull/44586#issuecomment-2551273259
`sh-checker report`
To get the full details, please check in the
[job]("https://github.com/apache/doris/actions/runs/12393541131";) output.
shellcheck errors
```
'shellcheck ' returned error 1 finding the following syntactical issues:
--
In gensrc/script/gen_build_version.sh line 38:
if [[ ${build_version_hotfix} > 0 ]]; then
^-- SC2071 (error): > is for string
comparisons. Use -gt instead.
In gensrc/script/gen_build_version.sh line 228:
if [[ ${build_version_hotfix} > 0 ]]; then
^-- SC2071 (error): > is for string
comparisons. Use -gt instead.
For more information:
https://www.shellcheck.net/wiki/SC2071 -- > is for string comparisons. Use
...
--
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=
above the line that contains the issue, where is the error code;
3. Add '-e ' to the SHELLCHECK_OPTS setting in your .yml action file.
```
shfmt errors
```
'shfmt ' found no issues.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
924060929 commented on PR #44586: URL: https://github.com/apache/doris/pull/44586#issuecomment-2551271996 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
github-actions[bot] commented on PR #44586:
URL: https://github.com/apache/doris/pull/44586#issuecomment-2550356328
`sh-checker report`
To get the full details, please check in the
[job]("https://github.com/apache/doris/actions/runs/1238687";) output.
shellcheck errors
```
'shellcheck ' returned error 1 finding the following syntactical issues:
--
In gensrc/script/gen_build_version.sh line 38:
if [[ ${build_version_hotfix} > 0 ]]; then
^-- SC2071 (error): > is for string
comparisons. Use -gt instead.
In gensrc/script/gen_build_version.sh line 228:
if [[ ${build_version_hotfix} > 0 ]]; then
^-- SC2071 (error): > is for string
comparisons. Use -gt instead.
For more information:
https://www.shellcheck.net/wiki/SC2071 -- > is for string comparisons. Use
...
--
You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
# shellcheck disable=
above the line that contains the issue, where is the error code;
3. Add '-e ' to the SHELLCHECK_OPTS setting in your .yml action file.
```
shfmt errors
```
'shfmt ' found no issues.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
924060929 commented on PR #44586: URL: https://github.com/apache/doris/pull/44586#issuecomment-2550355689 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
924060929 commented on PR #44586: URL: https://github.com/apache/doris/pull/44586#issuecomment-2548397617 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
924060929 commented on code in PR #44586:
URL: https://github.com/apache/doris/pull/44586#discussion_r1888388219
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/PartitionItemToRange.java:
##
@@ -0,0 +1,78 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.expression.rules;
+
+import org.apache.doris.analysis.LiteralExpr;
+import org.apache.doris.catalog.ListPartitionItem;
+import org.apache.doris.catalog.PartitionItem;
+import org.apache.doris.catalog.PartitionKey;
+import org.apache.doris.catalog.RangePartitionItem;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Range;
+import com.google.common.collect.TreeRangeSet;
+
+import java.util.List;
+
+/** PartitionItemToRange */
+public class PartitionItemToRange {
+/** toRangeSets */
+public static TreeRangeSet toRangeSets(PartitionItem
partitionItem) {
+if (partitionItem instanceof RangePartitionItem) {
+Range range = partitionItem.getItems();
+TreeRangeSet oneRangePartitionRanges =
TreeRangeSet.create();
+PartitionKey lowerKey = range.lowerEndpoint();
+ImmutableList.Builder lowerBounds
+=
ImmutableList.builderWithExpectedSize(lowerKey.getKeys().size());
+for (LiteralExpr key : lowerKey.getKeys()) {
+Literal literal = Literal.fromLegacyLiteral(key,
key.getType());
Review Comment:
This solution maybe use more memory and maybe cause OOM when upgrade doris.
I think we should replace PartitionKey to Expression after remove Legacy
expression
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
924060929 commented on code in PR #44586:
URL: https://github.com/apache/doris/pull/44586#discussion_r1888388219
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/PartitionItemToRange.java:
##
@@ -0,0 +1,78 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.expression.rules;
+
+import org.apache.doris.analysis.LiteralExpr;
+import org.apache.doris.catalog.ListPartitionItem;
+import org.apache.doris.catalog.PartitionItem;
+import org.apache.doris.catalog.PartitionKey;
+import org.apache.doris.catalog.RangePartitionItem;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Range;
+import com.google.common.collect.TreeRangeSet;
+
+import java.util.List;
+
+/** PartitionItemToRange */
+public class PartitionItemToRange {
+/** toRangeSets */
+public static TreeRangeSet toRangeSets(PartitionItem
partitionItem) {
+if (partitionItem instanceof RangePartitionItem) {
+Range range = partitionItem.getItems();
+TreeRangeSet oneRangePartitionRanges =
TreeRangeSet.create();
+PartitionKey lowerKey = range.lowerEndpoint();
+ImmutableList.Builder lowerBounds
+=
ImmutableList.builderWithExpectedSize(lowerKey.getKeys().size());
+for (LiteralExpr key : lowerKey.getKeys()) {
+Literal literal = Literal.fromLegacyLiteral(key,
key.getType());
Review Comment:
This solution maybe use more memory and cause OOM when upgrade doris.
I think we should replace PartitionKey to Expression after remove Legacy
expression
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
feiniaofeiafei commented on code in PR #44586:
URL: https://github.com/apache/doris/pull/44586#discussion_r1866997179
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/PartitionPruner.java:
##
@@ -134,39 +143,111 @@ public static List prune(List
partitionSlots, Expression partitionP
return ImmutableList.of();
}
-List evaluators =
Lists.newArrayListWithCapacity(idToPartitions.size());
-for (Entry kv : idToPartitions.entrySet()) {
-evaluators.add(toPartitionEvaluator(
-kv.getKey(), kv.getValue(), partitionSlots,
cascadesContext, expandThreshold));
+if (sortedPartitionRanges != null) {
+RangeSet predicateRanges =
partitionPredicate.accept(
+new PartitionPredicateToRange(partitionSlots), null);
Review Comment:
dt<'2020-02-01' will compute a empty predicateRanges, left bound is closed
and right bound is open. e.g.['2020-02-01','2020-02-01')
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
feiniaofeiafei commented on code in PR #44586:
URL: https://github.com/apache/doris/pull/44586#discussion_r1866951021
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PruneOlapScanPartition.java:
##
@@ -76,10 +81,18 @@ public Rule build() {
partitionSlots.add(partitionSlot);
}
}
+NereidsSortedPartitionsCacheManager sortedPartitionsCacheManager =
Env.getCurrentEnv()
+.getSortedPartitionsCacheManager();
List manuallySpecifiedPartitions =
scan.getManuallySpecifiedPartitions();
-Map idToPartitions;
+Map idToPartitions = ImmutableMap.of();
+SortedPartitionRanges sortedPartitionRanges = null;
if (manuallySpecifiedPartitions.isEmpty()) {
-idToPartitions = partitionInfo.getIdToItem(false);
+Optional> sortedPartitionRangesOpt =
sortedPartitionsCacheManager.get(table);
+if (sortedPartitionRangesOpt.isPresent()) {
+sortedPartitionRanges = (SortedPartitionRanges)
sortedPartitionRangesOpt.get();
+} else {
+idToPartitions = partitionInfo.getIdToItem(false);
+}
Review Comment:
when sortedPartitionRangesOpt.isPresent(), idToPartitions will be empty map,
in PartitionPruner.prune, may lead to result wrong when
BooleanLiteral.TRUE.equals(partitionPredicate) is true.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
morrySnow commented on code in PR #44586:
URL: https://github.com/apache/doris/pull/44586#discussion_r1862302951
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/MultiColumnBound.java:
##
@@ -0,0 +1,54 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.expression.rules;
+
+import java.util.List;
+
+/** MultiColumnBound */
+public class MultiColumnBound implements Comparable {
+private final List columnBounds;
+
+public MultiColumnBound(List columnBounds) {
+this.columnBounds = columnBounds;
Review Comment:
need immutable?
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/SortedPartitionRanges.java:
##
@@ -0,0 +1,46 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.expression.rules;
+
+import org.apache.doris.catalog.PartitionItem;
+
+import com.google.common.collect.RangeSet;
+
+import java.util.List;
+
+/** SortedPartitionRanges */
+public class SortedPartitionRanges {
+public final List> sortedPartitions;
+
+public SortedPartitionRanges(List>
sortedPartitions) {
+this.sortedPartitions = sortedPartitions;
Review Comment:
check null and immutable?
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/PartitionPruner.java:
##
@@ -105,19 +108,25 @@ public Expression
visitComparisonPredicate(ComparisonPredicate cp, Void context)
public List prune() {
Builder scanPartitionIdents = ImmutableList.builder();
for (OnePartitionEvaluator partition : partitions) {
-if (!canBePrunedOut(partition)) {
+if (!canBePrunedOut(partitionPredicate, partition)) {
scanPartitionIdents.add((K) partition.getPartitionIdent());
}
}
return scanPartitionIdents.build();
}
+public static List prune(List partitionSlots, Expression
partitionPredicate,
+Map idToPartitions, CascadesContext
cascadesContext,
+PartitionTableType partitionTableType) {
+return prune(partitionSlots, partitionPredicate, idToPartitions,
cascadesContext, partitionTableType, null);
+}
+
/**
* prune partition with `idToPartitions` as parameter.
*/
public static List prune(List partitionSlots, Expression
partitionPredicate,
Map idToPartitions, CascadesContext
cascadesContext,
-PartitionTableType partitionTableType) {
+PartitionTableType partitionTableType, SortedPartitionRanges
sortedPartitionRanges) {
Review Comment:
use Optional> ?
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rules/PartitionPredicateToRange.java:
##
@@ -0,0 +1,249 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS
Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
924060929 commented on PR #44586: URL: https://github.com/apache/doris/pull/44586#issuecomment-2499544562 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [opt](nereids) use binary search to prune partitions [doris]
doris-robot commented on PR #44586: URL: https://github.com/apache/doris/pull/44586#issuecomment-2499516549 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
