Re: [PR] [improvement](parquet) Prune array_contains with row group statistics [doris]
Gabriel39 merged PR #67441: URL: https://github.com/apache/doris/pull/67441 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](parquet) Prune array_contains with row group statistics [doris]
hello-stephen commented on PR #67441: URL: https://github.com/apache/doris/pull/67441#issuecomment-5537087584 # BE Regression && UT Coverage Report Increment line coverage `69.23% (9/13)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/67441_cf8309ea71c9f4a68473ea4f87095942d78e25c6_merge/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/67441_cf8309ea71c9f4a68473ea4f87095942d78e25c6_merge/report/index.html) | Category | Coverage | |---|| | Function Coverage | 76.29% (34657/45428) | | Line Coverage | 61.36% (391172/637534) | | Region Coverage | 57.67% (329597/571552) | | Branch Coverage | 58.40% (150049/256938) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](parquet) Prune array_contains with row group statistics [doris]
hello-stephen commented on PR #67441: URL: https://github.com/apache/doris/pull/67441#issuecomment-5535604504 TPC-DS: Total hot run time: 81790 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit cf8309ea71c9f4a68473ea4f87095942d78e25c6, data reload: false query5 4251405 349 349 query6 401 138 133 133 query7 4917428 237 237 query8 309 124 116 116 query9 8685287328832873 query10 393 206 202 202 query11 53891020921 921 query12 112 69 72 69 query13 1188443 306 306 query14 6036218320902090 query14_11974195419601954 query15 172 122 110 110 query16 940 368 355 355 query17 825 462 373 373 query18 2331330 241 241 query19 172 145 112 112 query20 74 71 70 70 query21 207 104 88 88 query22 5376532654215326 query23 6810616659215921 query23_16188605259895989 query24 72941095764 764 query24_1792 773 777 773 query25 405 294 232 232 query26 1212238 123 123 query27 2793425 255 255 query28 4696149114991491 query29 912 420 335 335 query30 246 152 127 127 query31 814 389 317 317 query32 123 72 72 72 query33 450 210 164 164 query34 980 830 502 502 query35 400 398 339 339 query36 580 568 539 539 query37 120 82 70 70 query38 994 836 824 824 query39 471 465 472 465 query39_1477 483 465 465 query40 205 88 76 76 query41 55 50 50 50 query42 72 70 71 70 query43 237 240 210 210 query44 1015539 566 539 query45 108 104 99 99 query46 753 885 570 570 query47 753 747 698 698 query48 307 281 235 235 query49 526 240 190 190 query50 735 263 188 188 query51 8141803080518030 query52 67 67 64 64 query53 188 197 146 146 query54 215 159 162 159 query55 74 57 54 54 query56 210 172 169 169 query57 751 701 662 662 query58 201 187 171 171 query59 1240122910921092 query60 224 194 177 177 query61 113 118 129 118 query62 358 205 186 186 query63 163 138 143 138 query64 2706738 603 603 query65 1582157415491549 query66 1894257 212 212 query67 9751964596829645 query68 27581260719 719 query69 339 238 199 199 query70 673 650 627 627 query71 251 173 177 173 query72 2427172415451545 query73 666 561 341 341 query74 1565120711341134 query75 11641084963 963 query76 2285728 526 526 query77 255 252 216 216 query78 4267359832423242 query79 1220796 583 583 query80 1184314 279 279 query81 487 151 128 128 query82 682 128 94 94 query83 317 209 189 189 query84 287 111 88 88 query85 911 353 296 296 query86 377 176 173 173 query87 1001951 895 895 query88 2750211321112111 query89 288 197 177 177 query90 1907129 129 129 query91 131 121 100 100 query92 77 70 66 66 query93 12701119658 658 query94 637 257 216 216 query95 506 324 229 229 query96 855 591 270 270 query97 1071102910451029 query98 145 141 138 138 query99 423 351 309 309 Total cold run time: 175779 ms Total hot run time: 81790 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific
Re: [PR] [improvement](parquet) Prune array_contains with row group statistics [doris]
hello-stephen commented on PR #67441: URL: https://github.com/apache/doris/pull/67441#issuecomment-5535635354 ClickBench: Total hot run time: 14.63 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit cf8309ea71c9f4a68473ea4f87095942d78e25c6, data reload: false query1 0.000.010.00 query2 0.070.030.04 query3 0.250.110.11 query4 1.600.100.09 query5 0.170.180.15 query6 1.250.670.67 query7 0.030.010.00 query8 0.050.030.03 query9 0.300.210.21 query10 0.340.360.34 query11 0.160.110.12 query12 0.150.130.13 query13 0.310.310.30 query14 0.460.450.43 query15 0.360.350.34 query16 0.220.200.22 query17 0.670.680.70 query18 0.180.170.18 query19 1.201.191.19 query20 0.020.010.01 query21 15.47 0.150.12 query22 5.080.050.05 query23 16.18 0.250.11 query24 2.970.310.24 query25 0.110.050.04 query26 0.800.170.11 query27 0.040.030.04 query28 3.620.560.28 query29 12.46 3.212.59 query30 0.250.110.12 query31 2.750.370.17 query32 3.580.330.23 query33 1.331.411.55 query34 15.38 2.141.77 query35 1.711.681.71 query36 0.450.300.28 query37 0.060.040.04 query38 0.040.020.02 query39 0.030.030.02 query40 0.120.080.07 query41 0.070.030.02 query42 0.030.020.02 query43 0.030.030.03 Total cold run time: 90.35 s Total hot run time: 14.63 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](parquet) Prune array_contains with row group statistics [doris]
hello-stephen commented on PR #67441: URL: https://github.com/apache/doris/pull/67441#issuecomment-5535557760 TPC-H: Total hot run time: 16808 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit cf8309ea71c9f4a68473ea4f87095942d78e25c6, data reload: false -- Round 1 -- q1 17578 309530823082 q2 2092267 241 241 q3 10219 916 511 511 q4 4675243 209 209 q5 7672554 384 384 q6 134 113 91 91 q7 546 497 389 389 q8 9238903 946 903 q9 3442240123942394 q10 6511862 743 743 q11 395 203 176 176 q12 608 257 196 196 q13 18139 152211541154 q14 153 155 137 137 q15 q16 433 393 365 365 q17 1371941 800 800 q18 3231225122272227 q19 1288886 741 741 q20 373 281 197 197 q21 5650164218521642 q22 327 269 226 226 Total cold run time: 94075 ms Total hot run time: 16808 ms - Round 2, with runtime_filter_mode=off - q1 3467337733653365 q2 513 398 361 361 q3 2164231921782178 q4 11881159895 895 q5 2172210520782078 q6 169 119 85 85 q7 1003917 873 873 q8 1608141014151410 q9 3159308430973084 q10 1878179916281628 q11 356 274 250 250 q12 457 422 357 357 q13 1450152611621162 q14 169 175 163 163 q15 q16 396 394 351 351 q17 3622335831743174 q18 4783441146994411 q19 844 915 882 882 q20 984 987 823 823 q21 3870325831873187 q22 397 345 332 332 Total cold run time: 34649 ms Total hot run time: 31049 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](parquet) Prune array_contains with row group statistics [doris]
Gabriel39 commented on PR #67441: URL: https://github.com/apache/doris/pull/67441#issuecomment-5534657281 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](parquet) Prune array_contains with row group statistics [doris]
Gabriel39 commented on PR #67441: URL: https://github.com/apache/doris/pull/67441#issuecomment-5534656549 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](parquet) Prune array_contains with row group statistics [doris]
Gabriel39 commented on PR #67441: URL: https://github.com/apache/doris/pull/67441#issuecomment-5527032739 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](parquet) Prune array_contains with row group statistics [doris]
Gabriel39 commented on PR #67441: URL: https://github.com/apache/doris/pull/67441#issuecomment-5527034299 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](parquet) Prune array_contains with row group statistics [doris]
Gabriel39 commented on code in PR #67441:
URL: https://github.com/apache/doris/pull/67441#discussion_r3925385970
##
be/src/exprs/vectorized_fn_call.cpp:
##
@@ -683,12 +683,13 @@ bool
VectorizedFnCall::is_safe_to_execute_on_selected_rows() const {
"is_null_pred",
"is_not_null_pred",
"element_at",
-
"struct_element"};
+
"struct_element",
+
"array_contains"};
Review Comment:
Fixed in 214ba43510. The safe-prefix classification now mirrors the current
BE dispatch domain, including string compatibility, and keeps unsupported
complex element signatures unsafe. Added a regression for ARRAY>
with an ARRAY needle.
##
be/src/format_v2/parquet/parquet_statistics.cpp:
##
@@ -863,6 +863,31 @@ bool has_expr_zonemap_filter(const
format::FileScanRequest& request, const Runti
return has_variant_shredded_filter(request);
}
+bool can_evaluate_native_page_index(const VExprSPtr& expr) {
+if (expr == nullptr || !expr->can_evaluate_zonemap_filter()) {
+return false;
+}
+if (expr->op() == TExprOpcode::COMPOUND_AND) {
+return std::ranges::any_of(expr->children(),
can_evaluate_native_page_index);
+}
+if (expr->op() == TExprOpcode::COMPOUND_OR) {
+return !expr->children().empty() &&
+ std::ranges::all_of(expr->children(),
can_evaluate_native_page_index);
+}
+const auto probe =
expr_zonemap::extract_zonemap_filter_predicate_probe(expr);
+return probe.has_value() && probe->path.empty();
Review Comment:
Fixed in 214ba43510. RuntimeFilterExpr is now unwrapped only in the Page
Index capability and path classifier; the original wrapper remains in the
request for delegated evaluation. Added a regression covering both production
admission and range pruning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [improvement](parquet) Prune array_contains with row group statistics [doris]
hello-stephen commented on PR #67441: URL: https://github.com/apache/doris/pull/67441#issuecomment-5509721564 TPC-H: Total hot run time: 16925 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 6b293ea1648346842587eed02e3d4d78940ad3a9, data reload: false -- Round 1 -- q1 17606 307530613061 q2 2155269 229 229 q3 10189 845 506 506 q4 4673253 203 203 q5 7677569 386 386 q6 133 113 94 94 q7 525 501 380 380 q8 9235925 899 899 q9 3447238623622362 q10 6490844 694 694 q11 392 197 186 186 q12 615 263 201 201 q13 18122 154611571157 q14 164 151 140 140 q15 q16 436 394 369 369 q17 1362923 844 844 q18 3057224422592244 q19 1291879 799 799 q20 379 289 200 200 q21 5668174019191740 q22 341 270 231 231 Total cold run time: 93957 ms Total hot run time: 16925 ms - Round 2, with runtime_filter_mode=off - q1 3436333233393332 q2 499 389 370 370 q3 2253235522162216 q4 11811157894 894 q5 2191210320902090 q6 167 115 85 85 q7 1031894 837 837 q8 1593140214001400 q9 3134308230673067 q10 1857179016281628 q11 365 270 247 247 q12 452 444 351 351 q13 1470153911631163 q14 172 175 160 160 q15 q16 398 402 361 361 q17 3559330431623162 q18 4851442647654426 q19 893 910 891 891 q20 1016977 824 824 q21 3842319231533153 q22 396 340 335 335 Total cold run time: 34756 ms Total hot run time: 30992 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](parquet) Prune array_contains with row group statistics [doris]
hello-stephen commented on PR #67441: URL: https://github.com/apache/doris/pull/67441#issuecomment-5509860001 ClickBench: Total hot run time: 14.62 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 6b293ea1648346842587eed02e3d4d78940ad3a9, data reload: false query1 0.010.010.00 query2 0.080.030.03 query3 0.250.110.11 query4 1.610.100.09 query5 0.170.150.15 query6 1.260.710.67 query7 0.030.000.01 query8 0.050.030.03 query9 0.290.220.22 query10 0.350.350.35 query11 0.150.110.12 query12 0.140.130.12 query13 0.310.300.31 query14 0.470.460.46 query15 0.340.350.34 query16 0.220.210.23 query17 0.640.700.72 query18 0.180.160.17 query19 1.201.181.15 query20 0.010.010.02 query21 15.44 0.170.13 query22 5.040.040.05 query23 16.19 0.250.10 query24 3.050.320.26 query25 0.110.040.03 query26 0.780.160.12 query27 0.040.030.03 query28 3.680.580.27 query29 12.42 3.202.55 query30 0.250.120.13 query31 2.760.360.17 query32 3.530.320.24 query33 1.481.511.39 query34 15.41 2.151.76 query35 1.731.731.71 query36 0.450.280.30 query37 0.070.040.04 query38 0.040.030.02 query39 0.030.020.03 query40 0.120.080.07 query41 0.080.030.02 query42 0.030.030.02 query43 0.030.020.03 Total cold run time: 90.52 s Total hot run time: 14.62 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](parquet) Prune array_contains with row group statistics [doris]
hello-stephen commented on PR #67441: URL: https://github.com/apache/doris/pull/67441#issuecomment-5509802273 TPC-DS: Total hot run time: 81940 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 6b293ea1648346842587eed02e3d4d78940ad3a9, data reload: false query5 4270410 340 340 query6 393 136 123 123 query7 4939405 238 238 query8 294 120 115 115 query9 8674290628712871 query10 402 208 190 190 query11 53651038918 918 query12 115 69 69 69 query13 1186454 329 329 query14 6141219620562056 query14_11968194119411941 query15 167 120 112 112 query16 918 366 339 339 query17 777 436 345 345 query18 2319330 228 228 query19 160 135 102 102 query20 68 70 68 68 query21 196 102 86 86 query22 5356530652105210 query23 6614625859285928 query23_16065591560775915 query24 73241099766 766 query24_1798 788 776 776 query25 431 284 247 247 query26 1236236 137 137 query27 2767410 269 269 query28 4672150415211504 query29 938 455 366 366 query30 255 157 135 135 query31 820 398 337 337 query32 128 78 75 75 query33 467 222 180 180 query34 999 856 488 488 query35 385 402 351 351 query36 587 569 519 519 query37 122 83 71 71 query38 1012865 820 820 query39 508 475 474 474 query39_1449 446 447 446 query40 205 99 82 82 query41 59 56 57 56 query42 75 81 73 73 query43 240 243 208 208 query44 1015562 562 562 query45 109 111 107 107 query46 797 869 550 550 query47 782 777 725 725 query48 316 318 243 243 query49 551 272 190 190 query50 770 279 191 191 query51 8205818581738173 query52 66 66 61 61 query53 191 207 148 148 query54 255 189 176 176 query55 70 57 61 57 query56 188 203 188 188 query57 651 714 629 629 query58 206 166 156 156 query59 1234123811001100 query60 237 179 171 171 query61 128 118 126 118 query62 355 207 180 180 query63 173 143 139 139 query64 2659709 582 582 query65 1613161015921592 query66 1862249 234 234 query67 9673976696319631 query68 27561185810 810 query69 338 236 196 196 query70 651 628 615 615 query71 244 178 171 171 query72 2323148815591488 query73 656 583 340 340 query74 1574123811291129 query75 11701119960 960 query76 2298736 543 543 query77 257 258 210 210 query78 3933375932243224 query79 1312917 591 591 query80 1191316 283 283 query81 483 156 137 137 query82 613 124 98 98 query83 321 211 201 201 query84 303 109 91 91 query85 800 361 308 308 query86 391 179 176 176 query87 1006957 887 887 query88 2784211621232116 query89 272 193 174 174 query90 1921129 126 126 query91 129 116 103 103 query92 79 67 70 67 query93 14051056726 726 query94 659 259 193 193 query95 528 266 225 225 query96 792 554 272 272 query97 1052105910121012 query98 140 140 132 132 query99 413 342 312 312 Total cold run time: 175196 ms Total hot run time: 81940 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific
Re: [PR] [improvement](parquet) Prune array_contains with row group statistics [doris]
github-actions[bot] commented on code in PR #67441:
URL: https://github.com/apache/doris/pull/67441#discussion_r3912894122
##
be/src/exprs/vectorized_fn_call.cpp:
##
@@ -683,12 +683,13 @@ bool
VectorizedFnCall::is_safe_to_execute_on_selected_rows() const {
"is_null_pred",
"is_not_null_pred",
"element_at",
-
"struct_element"};
+
"struct_element",
+
"array_contains"};
Review Comment:
[P1] Keep unsupported array_contains types outside the safe prefix
This name-only opt-in is broader than the function's executable domain.
Nereids accepts matching complex element types (for example `ARRAY>`
with an `ARRAY` needle), but `_execute_dispatch()` only handles the
scalar/string families in `DispatchDataTypeMask::ALL` and returns a runtime
error for ARRAY/MAP/STRUCT elements. Marking all such calls safe lets
TableReader's metadata-pruning prefix cross that error-producing conjunct; a
later scalar ZoneMap predicate can then drop the Row Group before residual
execution and turn the error into an empty result. Please restrict safety to
supported prepared element types (or reject/implement the complex signatures
earlier) and add a safe-prefix regression that preserves this error.
##
be/src/format_v2/parquet/parquet_statistics.cpp:
##
@@ -863,6 +863,31 @@ bool has_expr_zonemap_filter(const
format::FileScanRequest& request, const Runti
return has_variant_shredded_filter(request);
}
+bool can_evaluate_native_page_index(const VExprSPtr& expr) {
+if (expr == nullptr || !expr->can_evaluate_zonemap_filter()) {
+return false;
+}
+if (expr->op() == TExprOpcode::COMPOUND_AND) {
+return std::ranges::any_of(expr->children(),
can_evaluate_native_page_index);
+}
+if (expr->op() == TExprOpcode::COMPOUND_OR) {
+return !expr->children().empty() &&
+ std::ranges::all_of(expr->children(),
can_evaluate_native_page_index);
+}
+const auto probe =
expr_zonemap::extract_zonemap_filter_predicate_probe(expr);
+return probe.has_value() && probe->path.empty();
Review Comment:
[P1] Preserve Page Index pruning for runtime-filter wrappers
This new gate rejects every `RuntimeFilterExpr`-only request even when its
wrapped scalar predicate is Page-Index evaluable. `RuntimeFilterExpr` keeps the
predicate in `_impl` and delegates `can_evaluate_zonemap_filter()`, evaluation,
and slot collection, but it does not populate the inherited `_children`; the
probe extractor uses non-virtual `get_num_children()`/`get_child()` and
therefore returns no probe for the wrapper. This makes the gate false and
`finalize_native_row_group_read_plan()` skips loading ColumnIndex/OffsetIndex,
although the downstream selector already supports the delegated runtime filter.
Please unwrap `get_impl()` for capability/path classification (while retaining
the wrapper for actual evaluation) and add an RF-only
production-gate/range-selection regression test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [improvement](parquet) Prune array_contains with row group statistics [doris]
Gabriel39 commented on code in PR #67441:
URL: https://github.com/apache/doris/pull/67441#discussion_r3912509796
##
be/src/exprs/function/array/function_array_index.h:
##
@@ -218,6 +220,21 @@ class FunctionArrayIndex : public IFunction {
return _execute_dispatch(block, arguments, result, input_rows_count);
}
+ZoneMapFilterResult evaluate_zonemap_filter(const ZoneMapEvalContext& ctx,
+const VExprSPtrs& arguments)
const override {
+if constexpr (!std::is_same_v) {
+return unsupported_zonemap_filter(ctx);
+}
+return expr_zonemap::evaluate_array_contains_zonemap(ctx, arguments);
Review Comment:
Fixed in 6b293ea164. `evaluate_array_contains_zonemap()` now returns
unsupported when the caller binds the logical ARRAY instead of the repeated
element type, avoiding the invariant failure. The regression test covers the
logical ARRAY context shared by Segment and legacy Parquet.
##
be/test/format_v2/parquet/parquet_statistics_test.cpp:
##
@@ -1820,6 +1853,105 @@ TEST(NativeParquetStatisticsTest,
TypeDefinedBoundsRequireSupportedColumnOrder)
EXPECT_TRUE(selected_ranges.empty());
}
+TEST(NativeParquetStatisticsTest,
ArrayContainsUsesRepeatedLeafFooterStatisticsOnly) {
+const auto encode_int32 = [](int32_t value) {
+std::string bytes(sizeof(value), '\0');
+memcpy(bytes.data(), &value, sizeof(value));
+return bytes;
+};
+
+const auto leaf_type = std::make_shared();
+const auto array_type = std::make_shared(leaf_type);
+auto root_schema =
std::make_unique();
+root_schema->kind = format::parquet::ParquetColumnSchemaKind::LIST;
+root_schema->local_id = 0;
+root_schema->name = "items";
+root_schema->type = array_type;
+root_schema->max_repetition_level = 1;
+auto leaf_schema =
std::make_unique();
+leaf_schema->kind = format::parquet::ParquetColumnSchemaKind::PRIMITIVE;
+leaf_schema->local_id = 0;
+leaf_schema->name = "element";
+leaf_schema->leaf_column_id = 0;
+leaf_schema->type = leaf_type;
+leaf_schema->type_descriptor.doris_type = leaf_type;
+leaf_schema->type_descriptor.physical_type = tparquet::Type::INT32;
+leaf_schema->max_repetition_level = 1;
+root_schema->children.push_back(std::move(leaf_schema));
+std::vector> schema;
+schema.push_back(std::move(root_schema));
+
+tparquet::Statistics statistics;
+statistics.__set_min_value(encode_int32(10));
+statistics.__set_max_value(encode_int32(20));
+statistics.__set_null_count(0);
+tparquet::ColumnMetaData column_metadata;
+column_metadata.__set_type(tparquet::Type::INT32);
+column_metadata.__set_num_values(2);
+column_metadata.__set_statistics(statistics);
+tparquet::ColumnChunk chunk;
+chunk.__set_meta_data(column_metadata);
+tparquet::RowGroup row_group;
+row_group.__set_columns({chunk});
+row_group.__set_num_rows(1);
+tparquet::ColumnOrder order;
+order.__set_TYPE_ORDER(tparquet::TypeDefinedOrder());
+tparquet::FileMetaData metadata;
+metadata.__set_column_orders({order});
+metadata.__set_row_groups({row_group});
+
+const auto select_for_value = [&](int32_t value,
+ format::parquet::ParquetPruningStats*
pruning_stats) {
+format::FileScanRequest request;
Review Comment:
Fixed in 6b293ea164. `array_contains` is now admitted as a total predicate
for selected-row execution, and a TableReader integration test verifies that a
prepared production expression reaches the metadata-safe request prefix. The
Page Index gate now requires an actually page-evaluable, non-repeated probe,
with a regression assertion that an array-only request does not trigger Page
Index I/O.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [improvement](parquet) Prune array_contains with row group statistics [doris]
github-actions[bot] commented on code in PR #67441:
URL: https://github.com/apache/doris/pull/67441#discussion_r3912305299
##
be/test/format_v2/parquet/parquet_statistics_test.cpp:
##
@@ -1820,6 +1853,105 @@ TEST(NativeParquetStatisticsTest,
TypeDefinedBoundsRequireSupportedColumnOrder)
EXPECT_TRUE(selected_ranges.empty());
}
+TEST(NativeParquetStatisticsTest,
ArrayContainsUsesRepeatedLeafFooterStatisticsOnly) {
+const auto encode_int32 = [](int32_t value) {
+std::string bytes(sizeof(value), '\0');
+memcpy(bytes.data(), &value, sizeof(value));
+return bytes;
+};
+
+const auto leaf_type = std::make_shared();
+const auto array_type = std::make_shared(leaf_type);
+auto root_schema =
std::make_unique();
+root_schema->kind = format::parquet::ParquetColumnSchemaKind::LIST;
+root_schema->local_id = 0;
+root_schema->name = "items";
+root_schema->type = array_type;
+root_schema->max_repetition_level = 1;
+auto leaf_schema =
std::make_unique();
+leaf_schema->kind = format::parquet::ParquetColumnSchemaKind::PRIMITIVE;
+leaf_schema->local_id = 0;
+leaf_schema->name = "element";
+leaf_schema->leaf_column_id = 0;
+leaf_schema->type = leaf_type;
+leaf_schema->type_descriptor.doris_type = leaf_type;
+leaf_schema->type_descriptor.physical_type = tparquet::Type::INT32;
+leaf_schema->max_repetition_level = 1;
+root_schema->children.push_back(std::move(leaf_schema));
+std::vector> schema;
+schema.push_back(std::move(root_schema));
+
+tparquet::Statistics statistics;
+statistics.__set_min_value(encode_int32(10));
+statistics.__set_max_value(encode_int32(20));
+statistics.__set_null_count(0);
+tparquet::ColumnMetaData column_metadata;
+column_metadata.__set_type(tparquet::Type::INT32);
+column_metadata.__set_num_values(2);
+column_metadata.__set_statistics(statistics);
+tparquet::ColumnChunk chunk;
+chunk.__set_meta_data(column_metadata);
+tparquet::RowGroup row_group;
+row_group.__set_columns({chunk});
+row_group.__set_num_rows(1);
+tparquet::ColumnOrder order;
+order.__set_TYPE_ORDER(tparquet::TypeDefinedOrder());
+tparquet::FileMetaData metadata;
+metadata.__set_column_orders({order});
+metadata.__set_row_groups({row_group});
+
+const auto select_for_value = [&](int32_t value,
+ format::parquet::ParquetPruningStats*
pruning_stats) {
+format::FileScanRequest request;
Review Comment:
[P1] Exercise and enable the production metadata-safe path
This hand-built request keeps `metadata_pruning_safe_conjunct_count` at its
unlimited default, but production TableReader classifies `array_contains` as
unsafe because `VectorizedFnCall::is_safe_to_execute_on_selected_rows()` does
not include it. ColumnMapper then leaves the safe count at zero, so
`metadata_pruning_conjuncts()` is empty and neither `has_expr_zonemap_filter()`
nor `check_native_statistics()` can reach this feature. Please make the
predicate safely reachable through the real mapper path and add a
TableReader/ColumnMapper integration test; when doing so, also keep the
PageIndex pre-I/O gate from loading indexes for the repeated leaf, since page
pruning intentionally rejects it.
##
be/src/exprs/function/array/function_array_index.h:
##
@@ -218,6 +220,21 @@ class FunctionArrayIndex : public IFunction {
return _execute_dispatch(block, arguments, result, input_rows_count);
}
+ZoneMapFilterResult evaluate_zonemap_filter(const ZoneMapEvalContext& ctx,
+const VExprSPtrs& arguments)
const override {
+if constexpr (!std::is_same_v) {
+return unsupported_zonemap_filter(ctx);
+}
+return expr_zonemap::evaluate_array_contains_zonemap(ctx, arguments);
Review Comment:
[P1] Keep this capability from failing non-V2 ZoneMap callers
This shared function capability is also consumed by Segment and the legacy
Parquet reader, not just FileScanner V2. Those callers bind the slot's
top-level `DataTypeArray` (and may have no ZoneMap for the complex wrapper),
whereas `evaluate_array_contains_zonemap()` extracts the element type and calls
`fetch_compatible_slot_type()` before checking `ctx.zone_map()`. The
ARRAY-versus-element mismatch therefore raises an always-on `DORIS_CHECK` and
the scanner reports a failed query for ordinary pushed-down
`array_contains(array_col, literal)` filters. Please scope repeated-element
capability to the V2 row-group context, or return unsupported for a non-element
binding before the invariant-checking helper, and cover Segment and legacy
Parquet callers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail:
Re: [PR] [improvement](parquet) Prune array_contains with row group statistics [doris]
Gabriel39 commented on PR #67441: URL: https://github.com/apache/doris/pull/67441#issuecomment-5507192366 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](parquet) Prune array_contains with row group statistics [doris]
Gabriel39 commented on PR #67441: URL: https://github.com/apache/doris/pull/67441#issuecomment-5507192808 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](parquet) Prune array_contains with row group statistics [doris]
hello-stephen commented on PR #67441: URL: https://github.com/apache/doris/pull/67441#issuecomment-5506287682 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](parquet) Prune array_contains with row group statistics [doris]
Gabriel39 commented on PR #67441: URL: https://github.com/apache/doris/pull/67441#issuecomment-5506292604 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](parquet) Prune array_contains with row group statistics [doris]
Gabriel39 commented on PR #67441: URL: https://github.com/apache/doris/pull/67441#issuecomment-5506289444 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
