Re: [PR] [fix](fe) Avoid OFFSET path and NULL path for complex datatype when its children are accessed. [doris]
924060929 commented on PR #63229: URL: https://github.com/apache/doris/pull/63229#issuecomment-4853008245 The path convention should be: each `*`/field segment descends exactly one level, and a trailing `OFFSET`/`NULL` applies to the level reached: - `[m, *, OFFSET]`= offset of the **value array** (one `*` → into the map value) - `[m, *, *, OFFSET]` = offset of the **array item** (two `*` → into the array element) So `[m, *, OFFSET]` is unambiguous, and **BE should honor it regardless of whether FE splits the path**. If BE routes that OFFSET down to the array *item* and fails, BE is violating the one-level-per-segment rule — a BE bug to fix in BE. And BE already implements this: #61888 (2026-04-03) gave the map reader's `ACCESS_ALL` branch "key fully read + OFFSET applies only to the value", with a regression test `MapAccessAllWithOffsetDoesNotPropagateOffsetToKey` — ~6 weeks before this PR. That leaves two possibilities, and this PR is questionable under both: 1. If BE already resolves `[m, *, OFFSET]` correctly, then the failure observed here was **not** "OFFSET routed to an array item child" — the stated root cause is wrong, and this PR masks a different, still-unidentified problem. 2. If value=array genuinely still misbehaves in BE (the regression test only covers value=string, `length(map['k'])`), then the fix belongs in BE — add value=array handling — not FE. To be clear: emitting KEYS+VALUES from FE is fine in itself (it's clearer and it helps the FE strip logic). The problem is doing it *as a workaround for BE not honoring `[m, *, OFFSET]`*. BE must satisfy the one-level-per-segment rule independently; a FE bypass premised on a BE limitation either hides the real root cause (case 1) or papers over a BE bug that should be fixed (case 2). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Avoid OFFSET path and NULL path for complex datatype when its children are accessed. [doris]
englefly merged PR #63229: URL: https://github.com/apache/doris/pull/63229 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Avoid OFFSET path and NULL path for complex datatype when its children are accessed. [doris]
github-actions[bot] commented on PR #63229: URL: https://github.com/apache/doris/pull/63229#issuecomment-4483824733 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Avoid OFFSET path and NULL path for complex datatype when its children are accessed. [doris]
englefly commented on code in PR #63229:
URL: https://github.com/apache/doris/pull/63229#discussion_r3263245764
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/NestedColumnPruning.java:
##
@@ -399,7 +375,10 @@ &&
containsDataSkippingOnlyAccessPath(collectAccessPathResults)) {
// third: build predicate access path
for (Entry kv :
slotIdToPredicateAccessTree.entrySet()) {
Slot slot = kv.getKey();
+stripCoveredOffsetSuffixPaths(slot, predicateAccessPaths,
allAccessPaths);
+stripCoveredArrayNullSuffixPaths(slot, predicateAccessPaths,
allAccessPaths);
stripNullSuffixPaths(slot, predicateAccessPaths);
+retainPredicatePathsInAllAccessPaths(slot, predicateAccessPaths,
allAccessPaths);
List predicatePaths =
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Avoid OFFSET path and NULL path for complex datatype when its children are accessed. [doris]
englefly commented on PR #63229: URL: https://github.com/apache/doris/pull/63229#issuecomment-4483260037 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Avoid OFFSET path and NULL path for complex datatype when its children are accessed. [doris]
hello-stephen commented on PR #63229: URL: https://github.com/apache/doris/pull/63229#issuecomment-4480207316 # FE Regression Coverage Report Increment line coverage ` 43.05% (127/295)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/63229_49845488c2e1053b2680c9692cb69cc3154956dc_merge_fe/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/63229_49845488c2e1053b2680c9692cb69cc3154956dc_merge_fe/report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Avoid OFFSET path and NULL path for complex datatype when its children are accessed. [doris]
hello-stephen commented on PR #63229: URL: https://github.com/apache/doris/pull/63229#issuecomment-4479292342 TPC-DS: Total hot run time: 168514 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 49845488c2e1053b2680c9692cb69cc3154956dc, data reload: false query5 4335665 505 505 query6 344 228 197 197 query7 4229575 309 309 query8 333 227 220 220 query9 8824397339533953 query10 449 324 299 299 query11 5846240221882188 query12 184 128 129 128 query13 1282565 406 406 query14 5878534350475047 query14_14344429642694269 query15 210 202 181 181 query16 1086442 413 413 query17 1166722 580 580 query18 2648494 344 344 query19 219 192 156 156 query20 137 130 128 128 query21 217 138 123 123 query22 13551 13471 13222 13222 query23 17239 16375 16124 16124 query23_116127 16146 16338 16146 query24 7523173612831283 query24_11287129512861286 query25 551 472 410 410 query26 1322323 172 172 query27 2704544 337 337 query28 4435194919451945 query29 1026647 525 525 query30 299 240 200 200 query31 11131061942 942 query32 98 76 77 76 query33 555 355 312 312 query34 11901126644 644 query35 775 784 678 678 query36 1334132812251225 query37 161 108 91 91 query38 3246314730373037 query39 927 918 900 900 query39_1893 876 884 876 query40 238 153 131 131 query41 70 69 68 68 query42 115 111 108 108 query43 316 324 284 284 query44 query45 217 204 194 194 query46 10591144709 709 query47 2353234822452245 query48 415 419 310 310 query49 647 508 436 436 query50 965 353 265 265 query51 4290429442294229 query52 107 110 94 94 query53 253 283 220 220 query54 382 288 274 274 query55 96 93 86 86 query56 317 335 315 315 query57 1430139613291329 query58 325 273 276 273 query59 1601160813691369 query60 329 341 330 330 query61 183 177 181 177 query62 685 630 572 572 query63 248 211 207 207 query64 2473811 647 647 query65 query66 1708471 354 354 query67 29844 29876 29696 29696 query68 query69 461 343 305 305 query70 1022964 949 949 query71 414 274 264 264 query72 2972267224472447 query73 875 752 400 400 query74 5016486447264726 query75 2659259322362236 query76 23341117767 767 query77 385 399 328 328 query78 12186 12105 11579 11579 query79 1491998 718 718 query80 1340530 466 466 query81 502 277 235 235 query82 1310158 120 120 query83 330 279 245 245 query84 262 136 111 111 query85 934 557 449 449 query86 456 355 310 310 query87 3393336732043204 query88 3522262226402622 query89 456 381 338 338 query90 1920180 180 180 query91 179 163 164 163 query92 75 76 75 75 query93 15281395820 820 query94 717 344 307 307 query95 676 395 348 348 query96 1103765 329 329 query97 2700270125662566 query98 232 228 231 228 query99 11221102955 955 Total cold run time: 254266 ms Total hot run time: 168514 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected]
Re: [PR] [fix](fe) Avoid OFFSET path and NULL path for complex datatype when its children are accessed. [doris]
hello-stephen commented on PR #63229: URL: https://github.com/apache/doris/pull/63229#issuecomment-4479181557 TPC-H: Total hot run time: 30764 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 49845488c2e1053b2680c9692cb69cc3154956dc, data reload: false -- Round 1 -- orders Doris NULLNULL0 0 0 NULL0 NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 17887 387239093872 q2 q3 10767 1345802 802 q4 4685462 344 344 q5 7584220620622062 q6 237 170 134 134 q7 957 781 642 642 q8 9382165415551555 q9 6371488248584858 q10 6423210518141814 q11 444 280 243 243 q12 683 412 289 289 q13 18288 349727592759 q14 256 254 235 235 q15 q16 823 770 697 697 q17 912 932 906 906 q18 6903578255005500 q19 1238132910281028 q20 507 402 262 262 q21 5633250624652465 q22 429 344 297 297 Total cold run time: 100409 ms Total hot run time: 30764 ms - Round 2, with runtime_filter_mode=off - orders Doris NULLNULL15000 42 6422171781 NULL22778155NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 4185408440724072 q2 q3 4410488242774277 q4 2089218513531353 q5 4358426742504250 q6 225 175 128 128 q7 1920186116291629 q8 2423211220432043 q9 7662771077607710 q10 4507448140104010 q11 542 411 367 367 q12 716 731 576 576 q13 3387364730003000 q14 302 319 287 287 q15 q16 699 745 656 656 q17 1336129112871287 q18 7772731470467046 q19 1084103411131034 q20 2211220219231923 q21 5331459144074407 q22 514 449 387 387 Total cold run time: 55673 ms Total hot run time: 50442 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Avoid OFFSET path and NULL path for complex datatype when its children are accessed. [doris]
englefly commented on PR #63229: URL: https://github.com/apache/doris/pull/63229#issuecomment-4478611787 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Avoid OFFSET path and NULL path for complex datatype when its children are accessed. [doris]
hello-stephen commented on PR #63229: URL: https://github.com/apache/doris/pull/63229#issuecomment-4476198882 TPC-DS: Total hot run time: 168900 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 55c8428da3e9f32545fccbdb71a5f95db3ae981e, data reload: false query5 4305661 516 516 query6 338 213 205 205 query7 4206589 296 296 query8 333 237 216 216 query9 8831398339883983 query10 439 337 301 301 query11 5773237422182218 query12 180 143 124 124 query13 1275621 447 447 query14 6072533750115011 query14_14340433843214321 query15 210 201 181 181 query16 996 456 421 421 query17 715 592 592 query18 2734485 350 350 query19 212 198 161 161 query20 141 133 137 133 query21 214 139 117 117 query22 13638 13526 13393 13393 query23 17257 16378 16018 16018 query23_116243 16121 16198 16121 query24 7579178112871287 query24_11288131512931293 query25 567 509 447 447 query26 1348324 174 174 query27 2664571 353 353 query28 4457196219641962 query29 1027627 535 535 query30 313 248 198 198 query31 11351077939 939 query32 94 78 76 76 query33 542 359 312 312 query34 11741161651 651 query35 773 787 687 687 query36 1335134611781178 query37 157 107 92 92 query38 3188313630783078 query39 949 932 911 911 query39_1873 895 880 880 query40 228 153 132 132 query41 71 67 67 67 query42 115 112 116 112 query43 328 327 289 289 query44 query45 217 208 194 194 query46 10871204731 731 query47 2361231521982198 query48 423 408 324 324 query49 647 502 398 398 query50 987 367 260 260 query51 4344431842274227 query52 112 108 100 100 query53 257 282 208 208 query54 331 290 269 269 query55 96 91 89 89 query56 325 320 319 319 query57 1412141013061306 query58 329 286 281 281 query59 1554162713741374 query60 342 344 329 329 query61 222 147 156 147 query62 669 636 550 550 query63 247 206 199 199 query64 2366816 623 623 query65 query66 1662478 346 346 query67 30101 29929 29729 29729 query68 query69 448 336 308 308 query70 10211001962 962 query71 308 276 267 267 query72 2953271224452445 query73 816 757 406 406 query74 5099487847004700 query75 2674258522752275 query76 23281133775 775 query77 400 403 316 316 query78 12141 12080 11745 11745 query79 14391070733 733 query80 676 535 446 446 query81 463 273 239 239 query82 1367158 124 124 query83 340 287 247 247 query84 263 140 119 119 query85 880 536 486 486 query86 395 350 308 308 query87 3409339631753175 query88 3528269326642664 query89 461 376 335 335 query90 1896176 177 176 query91 179 164 146 146 query92 80 78 75 75 query93 15141506852 852 query94 578 344 306 306 query95 646 383 437 383 query96 994 777 339 339 query97 2709270625672567 query98 236 233 225 225 query99 11231094993 993 Total cold run time: 253378 ms Total hot run time: 168900 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected]
Re: [PR] [fix](fe) Avoid OFFSET path and NULL path for complex datatype when its children are accessed. [doris]
hello-stephen commented on PR #63229: URL: https://github.com/apache/doris/pull/63229#issuecomment-4476102069 TPC-H: Total hot run time: 31357 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 55c8428da3e9f32545fccbdb71a5f95db3ae981e, data reload: false -- Round 1 -- orders Doris NULLNULL0 0 0 NULL0 NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 17670 399038943894 q2 q3 10780 1373821 821 q4 4681477 345 345 q5 7653222821272127 q6 236 172 137 137 q7 970 798 630 630 q8 9371177116141614 q9 5120490349214903 q10 6379208718041804 q11 434 272 246 246 q12 632 421 289 289 q13 18095 332127902790 q14 266 256 235 235 q15 q16 821 797 708 708 q17 948 866 963 866 q18 6982595956015601 q19 1354134511101110 q20 565 559 285 285 q21 6026282526402640 q22 457 379 312 312 Total cold run time: 99440 ms Total hot run time: 31357 ms - Round 2, with runtime_filter_mode=off - orders Doris NULLNULL15000 42 6422171781 NULL22778155NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 4876446645734466 q2 q3 4878522346754675 q4 2097219413901390 q5 4769476046184618 q6 224 174 127 127 q7 1920172015481548 q8 2398200619981998 q9 7740750571617161 q10 4465441940314031 q11 532 380 351 351 q12 706 716 510 510 q13 3042345428092809 q14 279 269 257 257 q15 q16 689 691 606 606 q17 1259122612221222 q18 7378679466566656 q19 1100107810671067 q20 2214225019491949 q21 5281454444394439 q22 522 448 405 405 Total cold run time: 56369 ms Total hot run time: 50285 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Avoid OFFSET path and NULL path for complex datatype when its children are accessed. [doris]
englefly commented on PR #63229: URL: https://github.com/apache/doris/pull/63229#issuecomment-4475143650 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Avoid OFFSET path and NULL path for complex datatype when its children are accessed. [doris]
hello-stephen commented on PR #63229: URL: https://github.com/apache/doris/pull/63229#issuecomment-4459377413 # FE Regression Coverage Report Increment line coverage ` 68.67% (114/166)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/63229_8c80c3a93f16e83d9acd754d5cd1c438e2906acf_merge_fe/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/63229_8c80c3a93f16e83d9acd754d5cd1c438e2906acf_merge_fe/report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Avoid OFFSET path and NULL path for complex datatype when its children are accessed. [doris]
hello-stephen commented on PR #63229: URL: https://github.com/apache/doris/pull/63229#issuecomment-4458559641 TPC-DS: Total hot run time: 169017 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 8c80c3a93f16e83d9acd754d5cd1c438e2906acf, data reload: false query5 4309660 527 527 query6 324 224 208 208 query7 4224614 292 292 query8 336 242 236 236 query9 8823407540514051 query10 456 350 300 300 query11 5818240721702170 query12 181 129 126 126 query13 1313597 436 436 query14 6139535350395039 query14_14394438243474347 query15 211 203 185 185 query16 985 493 450 450 query17 1185741 614 614 query18 2723510 362 362 query19 229 205 168 168 query20 138 132 133 132 query21 217 148 127 127 query22 13689 13509 13281 13281 query23 17042 16330 16054 16054 query23_116146 16118 16244 16118 query24 7453178713101310 query24_11311130513231305 query25 582 496 450 450 query26 1309324 177 177 query27 2665563 345 345 query28 4387198320271983 query29 1038650 503 503 query30 312 242 201 201 query31 11151072937 937 query32 92 78 75 75 query33 563 367 310 310 query34 11771155663 663 query35 792 766 676 676 query36 1322136111791179 query37 154 99 92 92 query38 3208311230493049 query39 929 925 886 886 query39_1885 884 881 881 query40 233 146 124 124 query41 66 62 67 62 query42 110 109 112 109 query43 335 330 296 296 query44 query45 208 202 200 200 query46 10701156714 714 query47 2275233421882188 query48 399 422 293 293 query49 637 488 377 377 query50 1035354 256 256 query51 4337425841574157 query52 104 107 96 96 query53 254 295 209 209 query54 321 272 253 253 query55 98 95 87 87 query56 327 322 311 311 query57 1428137513241324 query58 299 270 263 263 query59 1555162814501450 query60 337 330 319 319 query61 164 156 156 156 query62 664 631 567 567 query63 250 209 207 207 query64 2360789 641 641 query65 query66 1650474 357 357 query67 29988 29314 29921 29314 query68 query69 482 343 307 307 query70 1031923 969 923 query71 302 276 274 274 query72 3025292324002400 query73 814 758 418 418 query74 5057490747434743 query75 2677260122612261 query76 22961167774 774 query77 416 418 363 363 query78 12268 12113 11596 11596 query79 14411030714 714 query80 734 554 453 453 query81 485 272 238 238 query82 1405160 123 123 query83 336 281 246 246 query84 269 149 117 117 query85 909 543 448 448 query86 409 334 327 327 query87 3417338232033203 query88 3575267826552655 query89 451 380 337 337 query90 1890190 190 190 query91 179 167 136 136 query92 82 78 76 76 query93 15881500886 886 query94 569 364 324 324 query95 656 382 360 360 query96 998 798 324 324 query97 2710270225782578 query98 239 227 237 227 query99 11011082961 961 Total cold run time: 253290 ms Total hot run time: 169017 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected]
Re: [PR] [fix](fe) Avoid OFFSET path and NULL path for complex datatype when its children are accessed. [doris]
hello-stephen commented on PR #63229: URL: https://github.com/apache/doris/pull/63229#issuecomment-4458482596 TPC-H: Total hot run time: 31032 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 8c80c3a93f16e83d9acd754d5cd1c438e2906acf, data reload: false -- Round 1 -- orders Doris NULLNULL0 0 0 NULL0 NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 17737 388938823882 q2 q3 10756 1422859 859 q4 4690470 346 346 q5 7546227021352135 q6 246 182 137 137 q7 951 822 636 636 q8 9356173016011601 q9 5163495649034903 q10 6407206517821782 q11 439 277 249 249 q12 626 435 292 292 q13 18142 344927342734 q14 259 255 236 236 q15 q16 812 790 706 706 q17 879 898 940 898 q18 7072581655365536 q19 1303128510781078 q20 524 408 260 260 q21 5822249124632463 q22 438 362 299 299 Total cold run time: 99168 ms Total hot run time: 31032 ms - Round 2, with runtime_filter_mode=off - orders Doris NULLNULL15000 42 6422171781 NULL22778155NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 4189409141354091 q2 q3 4487487843204320 q4 2113225314051405 q5 4410431943224319 q6 235 179 131 131 q7 1965198817291729 q8 2711216721942167 q9 8053811478727872 q10 4513446240524052 q11 602 423 369 369 q12 731 773 534 534 q13 3292362629842984 q14 302 307 262 262 q15 q16 728 731 645 645 q17 1344131514141315 q18 7982742872007200 q19 1182108811141088 q20 2202220019361936 q21 5304464244754475 q22 506 467 414 414 Total cold run time: 56851 ms Total hot run time: 51308 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Avoid OFFSET path and NULL path for complex datatype when its children are accessed. [doris]
englefly commented on PR #63229: URL: https://github.com/apache/doris/pull/63229#issuecomment-4458256667 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Avoid OFFSET path and NULL path for complex datatype when its children are accessed. [doris]
hello-stephen commented on PR #63229: URL: https://github.com/apache/doris/pull/63229#issuecomment-4452006067 # FE Regression Coverage Report Increment line coverage ` 81.55% (84/103)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/63229_a05541e3bd5f76dbc442499938cc3e97c1fb53c9_merge_fe/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/63229_a05541e3bd5f76dbc442499938cc3e97c1fb53c9_merge_fe/report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Avoid OFFSET path and NULL path for complex datatype when its children are accessed. [doris]
hello-stephen commented on PR #63229: URL: https://github.com/apache/doris/pull/63229#issuecomment-4450864482 TPC-DS: Total hot run time: 170638 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit a05541e3bd5f76dbc442499938cc3e97c1fb53c9, data reload: false query5 4322661 494 494 query6 323 214 192 192 query7 4222556 324 324 query8 335 233 237 233 query9 8824397239713971 query10 448 340 296 296 query11 5766238721942194 query12 184 126 128 126 query13 1284606 459 459 query14 6480534150455045 query14_14342435643034303 query15 209 200 183 183 query16 1010445 434 434 query17 1112742 601 601 query18 2519479 345 345 query19 208 197 158 158 query20 142 136 133 133 query21 208 142 115 115 query22 13716 13971 14665 13971 query23 17453 16467 15989 15989 query23_116296 16199 16164 16164 query24 7363174013291329 query24_11352134613801346 query25 620 526 449 449 query26 1315315 175 175 query27 2703629 353 353 query28 4425194919491949 query29 1015657 534 534 query30 305 240 196 196 query31 11081082940 940 query32 85 78 73 73 query33 538 368 310 310 query34 11681103643 643 query35 765 774 681 681 query36 1361137212191219 query37 156 107 91 91 query38 3184312530363036 query39 923 916 878 878 query39_1856 873 881 873 query40 233 158 137 137 query41 70 66 66 66 query42 112 110 110 110 query43 317 326 281 281 query44 query45 213 210 201 201 query46 10611187710 710 query47 2349231621752175 query48 403 419 293 293 query49 665 541 433 433 query50 714 300 223 223 query51 4280419042904190 query52 105 110 96 96 query53 245 284 206 206 query54 353 281 248 248 query55 97 87 87 87 query56 302 306 300 300 query57 1408139212851285 query58 287 267 258 258 query59 1546162414071407 query60 341 332 312 312 query61 157 153 155 153 query62 683 620 564 564 query63 245 200 206 200 query64 2414812 680 680 query65 query66 1718528 392 392 query67 29981 29838 29803 29803 query68 query69 459 333 307 307 query70 1023974 933 933 query71 304 280 267 267 query72 2956271723912391 query73 849 798 403 403 query74 5062485946944694 query75 2763264123162316 query76 22671140757 757 query77 424 424 353 353 query78 12993 13030 12181 12181 query79 14771001721 721 query80 1344577 471 471 query81 524 284 238 238 query82 1029160 135 135 query83 381 278 250 250 query84 266 135 109 109 query85 910 501 439 439 query86 465 332 309 309 query87 3406333132263226 query88 3550267326512651 query89 437 382 332 332 query90 1906180 182 180 query91 180 163 134 134 query92 79 73 69 69 query93 1069968 558 558 query94 740 346 300 300 query95 651 471 337 337 query96 1044753 317 317 query97 2712269725412541 query98 238 231 230 230 query99 11201112994 994 Total cold run time: 254463 ms Total hot run time: 170638 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected]
Re: [PR] [fix](fe) Avoid OFFSET path and NULL path for complex datatype when its children are accessed. [doris]
hello-stephen commented on PR #63229: URL: https://github.com/apache/doris/pull/63229#issuecomment-4450791645 TPC-H: Total hot run time: 29392 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit a05541e3bd5f76dbc442499938cc3e97c1fb53c9, data reload: false -- Round 1 -- orders Doris NULLNULL0 0 0 NULL0 NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 17638 383838053805 q2 q3 10722 865 594 594 q4 4665466 347 347 q5 7472135911341134 q6 190 168 137 137 q7 917 931 757 757 q8 9299137213001300 q9 5579533152705270 q10 6280208517951795 q11 472 266 265 265 q12 635 417 292 292 q13 18127 325527072707 q14 289 285 264 264 q15 q16 895 855 784 784 q17 967 1067746 746 q18 6463577355925592 q19 1246128310131013 q20 514 387 265 265 q21 4612237419731973 q22 493 430 352 352 Total cold run time: 97475 ms Total hot run time: 29392 ms - Round 2, with runtime_filter_mode=off - orders Doris NULLNULL15000 42 6422171781 NULL22778155NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 4774470848544708 q2 q3 4655476641924192 q4 2134216714131413 q5 4981493351734933 q6 195 174 138 138 q7 2067179516361636 q8 3332306230763062 q9 8393840385608403 q10 4520450242334233 q11 593 407 399 399 q12 701 746 518 518 q13 3219359729842984 q14 311 302 277 277 q15 q16 924 784 702 702 q17 1336128912511251 q18 7919712171777121 q19 1119110811401108 q20 2236230119301930 q21 6000535048404840 q22 552 510 423 423 Total cold run time: 59961 ms Total hot run time: 54271 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Avoid OFFSET path and NULL path for complex datatype when its children are accessed. [doris]
github-actions[bot] commented on code in PR #63229:
URL: https://github.com/apache/doris/pull/63229#discussion_r3241351122
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/NestedColumnPruning.java:
##
@@ -399,7 +375,10 @@ &&
containsDataSkippingOnlyAccessPath(collectAccessPathResults)) {
// third: build predicate access path
for (Entry kv :
slotIdToPredicateAccessTree.entrySet()) {
Slot slot = kv.getKey();
+stripCoveredOffsetSuffixPaths(slot, predicateAccessPaths,
allAccessPaths);
+stripCoveredArrayNullSuffixPaths(slot, predicateAccessPaths,
allAccessPaths);
stripNullSuffixPaths(slot, predicateAccessPaths);
+retainPredicatePathsInAllAccessPaths(slot, predicateAccessPaths,
allAccessPaths);
List predicatePaths =
Review Comment:
This filtering still checks predicate paths against the raw collected
`allAccessPaths`, but some earlier branches emit normalized `AccessPathInfo`
paths without updating that multimap. For example, `where
cardinality(map_arr_col['a']) > 0` on `MAP>` takes the
`hasMapValueOffsetOnlyAccess()` branch and emits all paths `[map_arr_col.KEYS,
map_arr_col.VALUES.OFFSET]`, while the collected predicate path remains
`[map_arr_col.*.OFFSET]`. Because raw `allAccessPaths` also still contains
`[map_arr_col.*.OFFSET]`, `retainPredicatePathsInAllAccessPaths()` keeps it, so
BE receives a predicate path that is not a subset of the emitted all paths,
violating the invariant this PR adds in tests. Please normalize/filter
predicate paths against the actual `AccessPathInfo.getAllAccessPaths()` result
(or update `allAccessPaths` in the special map branch), and add a test for a
predicate-only `cardinality(map_arr_col['a'])` query.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Avoid OFFSET path and NULL path for complex datatype when its children are accessed. [doris]
englefly commented on PR #63229: URL: https://github.com/apache/doris/pull/63229#issuecomment-4450644150 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Avoid OFFSET path and NULL path for complex datatype when its children are accessed. [doris]
englefly commented on PR #63229: URL: https://github.com/apache/doris/pull/63229#issuecomment-4450643598 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
