Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
morrySnow merged PR #64080: URL: https://github.com/apache/doris/pull/64080 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
github-actions[bot] commented on PR #64080: URL: https://github.com/apache/doris/pull/64080#issuecomment-4714159124 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
hello-stephen commented on PR #64080: URL: https://github.com/apache/doris/pull/64080#issuecomment-4706769831 # FE Regression Coverage Report Increment line coverage ` 32.56% (28/86)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/64080_15223fcaa66fa2b3d167b8000c96e9a8d83098a6_merge_fe/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/64080_15223fcaa66fa2b3d167b8000c96e9a8d83098a6_merge_fe/report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
hello-stephen commented on PR #64080: URL: https://github.com/apache/doris/pull/64080#issuecomment-4706348305 TPC-DS: Total hot run time: 168871 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 15223fcaa66fa2b3d167b8000c96e9a8d83098a6, data reload: false query5 4316638 468 468 query6 432 197 183 183 query7 4825560 298 298 query8 361 212 194 194 query9 8768408240634063 query10 438 314 252 252 query11 5874238121542154 query12 159 104 98 98 query13 1249639 434 434 query14 6378544150885088 query14_14446444044304430 query15 203 199 179 179 query16 998 471 447 447 query17 1072711 595 595 query18 2492475 347 347 query19 199 192 143 143 query20 121 114 107 107 query21 220 141 121 121 query22 13775 13761 13348 13348 query23 17419 16479 16059 16059 query23_116355 16207 16320 16207 query24 7436180413091309 query24_11345133113091309 query25 571 461 398 398 query26 1307328 170 170 query27 2701580 355 355 query28 4474204120502041 query29 1081629 495 495 query30 313 236 195 195 query31 11231067976 976 query32 107 63 61 61 query33 521 330 255 255 query34 12241173639 639 query35 741 791 681 681 query36 1391139612461246 query37 152 106 85 85 query38 3195315930503050 query39 921 912 905 905 query39_1880 863 884 863 query40 220 125 99 99 query41 64 63 61 61 query42 93 93 93 93 query43 326 323 282 282 query44 query45 201 184 177 177 query46 11011239778 778 query47 2364234221812181 query48 395 404 289 289 query49 617 457 368 368 query50 1030355 258 258 query51 4267432643384326 query52 88 90 76 76 query53 249 265 189 189 query54 259 216 215 215 query55 78 82 70 70 query56 223 219 217 217 query57 1415136813551355 query58 243 214 207 207 query59 1600167214711471 query60 280 241 224 224 query61 146 152 151 151 query62 695 648 584 584 query63 232 186 185 185 query64 2525769 599 599 query65 query66 1786449 356 356 query67 29660 29654 29499 29499 query68 query69 424 296 273 273 query70 1010917 903 903 query71 295 228 210 210 query72 2953265123722372 query73 853 753 402 402 query74 5121497547934793 query75 2649256622292229 query76 23291161813 813 query77 394 366 294 294 query78 12594 12499 11901 11901 query79 14931119817 817 query80 577 452 387 387 query81 472 285 239 239 query82 580 162 128 128 query83 366 275 249 249 query84 query85 837 502 409 409 query86 360 304 274 274 query87 3383334631713171 query88 3661277726922692 query89 422 392 332 332 query90 2010185 187 185 query91 171 159 132 132 query92 59 59 57 57 query93 14391466908 908 query94 518 359 318 318 query95 697 379 432 379 query96 1048863 338 338 query97 2680270925772577 query98 216 206 202 202 query99 1143116210161016 Total cold run time: 250917 ms Total hot run time: 168871 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
hello-stephen commented on PR #64080: URL: https://github.com/apache/doris/pull/64080#issuecomment-4706256930 TPC-H: Total hot run time: 29083 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 15223fcaa66fa2b3d167b8000c96e9a8d83098a6, data reload: false -- Round 1 -- orders Doris NULLNULL0 0 0 NULL0 NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 17660 416241354135 q2 q3 10789 1375821 821 q4 4680472 346 346 q5 7570889 579 579 q6 185 175 139 139 q7 790 827 636 636 q8 9354156115331533 q9 5945446645174466 q10 6810182915321532 q11 427 280 253 253 q12 634 428 304 304 q13 18103 330228082808 q14 263 259 244 244 q15 q16 822 773 713 713 q17 1007987 946 946 q18 7120570756655665 q19 1323131110091009 q20 520 416 261 261 q21 6050260923952395 q22 439 368 298 298 Total cold run time: 100491 ms Total hot run time: 29083 ms - Round 2, with runtime_filter_mode=off - orders Doris NULLNULL15000 42 6422171781 NULL22778155NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 4468445343404340 q2 q3 4538496543104310 q4 2104218713931393 q5 4468432443614324 q6 226 173 129 129 q7 1739163921151639 q8 2808234822652265 q9 8226843080708070 q10 4822484343264326 q11 602 407 381 381 q12 746 775 553 553 q13 3278366630183018 q14 303 296 280 280 q15 q16 719 758 631 631 q17 1384135713491349 q18 8317733474067334 q19 1171110210651065 q20 2215225219341934 q21 5337468345474547 q22 542 488 426 426 Total cold run time: 58013 ms Total hot run time: 52314 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
hello-stephen commented on PR #64080: URL: https://github.com/apache/doris/pull/64080#issuecomment-4706135424 # FE UT Coverage Report Increment line coverage ` 62.35% (53/85)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/64080_15223fcaa66fa2b3d167b8000c96e9a8d83098a6/fe_increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/64080_15223fcaa66fa2b3d167b8000c96e9a8d83098a6/fe_report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
github-actions[bot] commented on code in PR #64080:
URL: https://github.com/apache/doris/pull/64080#discussion_r3411726764
##
fe/fe-core/src/main/java/org/apache/doris/nereids/types/MapType.java:
##
@@ -63,6 +64,15 @@ public DataType conversion() {
return MapType.of(keyType.conversion(), valueType.conversion());
}
+@Override
+public boolean isInjectiveCastTo(DataType target) {
+if (target instanceof MapType) {
+MapType mapType = (MapType) target;
+return keyType.isInjectiveCastTo(mapType.keyType) &&
valueType.isInjectiveCastTo(mapType.valueType);
+}
+return target instanceof CharacterType;
Review Comment:
This `MapType -> MapType` predicate is a separate false positive from the
existing complex-to-character thread. BE map-to-map casts convert the key/value
columns and then call `ColumnMap::deduplicate_keys()`, while map equality/hash
still include the raw entry count and key/value sequence before the cast. A map
value with duplicate keys can therefore be distinct before the cast but
normalize to the same value after it; for example, external/map-serde input
with entries `[(1,10),(1,20)]` and another row `[(1,20)]` are different source
maps, but `CAST(m AS MAP)` keeps only the last `1` entry in both
rows. If this cast is pushed below `UNION DISTINCT`, duplicate elimination
happens after that normalization and can collapse a row the original plan would
keep. Please keep map-to-map casts unsafe unless the rule can prove the source
map is already key-deduplicated, or account for the BE dedup behavior here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
hello-stephen commented on PR #64080: URL: https://github.com/apache/doris/pull/64080#issuecomment-4705475765 # FE Regression Coverage Report Increment line coverage ` 15.48% (26/168)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/64080_ce2c202816f108aa821e4afa1db3bf5c108647e7_merge_fe/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/64080_ce2c202816f108aa821e4afa1db3bf5c108647e7_merge_fe/report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
morrySnow commented on PR #64080: URL: https://github.com/apache/doris/pull/64080#issuecomment-4705472141 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
morrySnow commented on PR #64080: URL: https://github.com/apache/doris/pull/64080#issuecomment-4705470873 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
hello-stephen commented on PR #64080: URL: https://github.com/apache/doris/pull/64080#issuecomment-4704599735 TPC-DS: Total hot run time: 169979 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 9679bbf6c65f4a1a5b1a2d99d70ea769f3dad15b, data reload: false query5 4318644 466 466 query6 440 182 173 173 query7 4849551 312 312 query8 342 212 192 192 query9 8768404640494046 query10 439 296 256 256 query11 5939237021982198 query12 156 102 98 98 query13 1249607 384 384 query14 6361541150155015 query14_14368436443634363 query15 201 196 175 175 query16 1047424 417 417 query17 1101703 550 550 query18 2512471 336 336 query19 196 176 139 139 query20 112 109 106 106 query21 209 137 116 116 query22 13623 13628 13358 13358 query23 17374 16609 16190 16190 query23_116254 16248 16392 16248 query24 7443176112981298 query24_11317132213161316 query25 576 471 391 391 query26 1301299 173 173 query27 2678578 348 348 query28 4500207820572057 query29 1089625 518 518 query30 308 251 204 204 query31 1095962 962 query32 108 63 67 63 query33 538 327 267 267 query34 11861174641 641 query35 778 791 694 694 query36 1444138412871287 query37 157 107 94 94 query38 3185314530853085 query39 931 918 904 904 query39_1884 872 885 872 query40 219 128 104 104 query41 71 68 67 67 query42 98 95 96 95 query43 328 329 276 276 query44 query45 194 187 180 180 query46 11021228721 721 query47 2426232122562256 query48 407 409 290 290 query49 648 474 371 371 query50 1004354 265 265 query51 4381429943124299 query52 89 92 78 78 query53 248 271 190 190 query54 284 231 212 212 query55 79 78 79 78 query56 253 241 237 237 query57 1462142813271327 query58 258 209 218 209 query59 1556165514141414 query60 305 255 248 248 query61 182 179 199 179 query62 687 651 573 573 query63 235 188 189 188 query64 2523767 615 615 query65 query66 1793462 330 330 query67 29734 29739 29638 29638 query68 query69 419 305 262 262 query70 905 955 956 955 query71 311 225 216 216 query72 2973265123532353 query73 869 788 448 448 query74 5125494147634763 query75 2639260022452245 query76 23011162827 827 query77 352 379 290 290 query78 12380 12299 11770 11770 query79 12791056778 778 query80 529 473 399 399 query81 452 285 235 235 query82 240 162 122 122 query83 360 284 245 245 query84 query85 881 522 403 403 query86 356 317 291 291 query87 3361337432063206 query88 3638275927762759 query89 405 387 337 337 query90 1906183 175 175 query91 175 154 135 135 query92 65 62 57 57 query93 15341398867 867 query94 544 358 302 302 query95 691 374 341 341 query96 1061837 338 338 query97 2695271425612561 query98 210 202 200 200 query99 1153120910321032 Total cold run time: 250145 ms Total hot run time: 169979 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about thi
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
morrySnow commented on PR #64080: URL: https://github.com/apache/doris/pull/64080#issuecomment-4704561006 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
morrySnow commented on PR #64080: URL: https://github.com/apache/doris/pull/64080#issuecomment-4704560369 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
hello-stephen commented on PR #64080: URL: https://github.com/apache/doris/pull/64080#issuecomment-4704549094 TPC-H: Total hot run time: 29555 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 9679bbf6c65f4a1a5b1a2d99d70ea769f3dad15b, data reload: false -- Round 1 -- orders Doris NULLNULL0 0 0 NULL0 NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 17645 416341184118 q2 q3 10793 1524821 821 q4 4683479 339 339 q5 7558911 574 574 q6 184 175 135 135 q7 781 823 626 626 q8 9408165915621562 q9 5871460845194519 q10 6767181415291529 q11 434 276 249 249 q12 645 431 293 293 q13 18156 344727872787 q14 262 260 253 253 q15 q16 818 784 723 723 q17 1008975 1000975 q18 6990575255775577 q19 1186137111511151 q20 551 409 261 261 q21 6112284727472747 q22 463 392 316 316 Total cold run time: 100315 ms Total hot run time: 29555 ms - Round 2, with runtime_filter_mode=off - orders Doris NULLNULL15000 42 6422171781 NULL22778155NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 5309477248184772 q2 q3 4958525947824782 q4 2129222014041404 q5 4852502347294729 q6 237 176 131 131 q7 1894181015651565 q8 2489215321532153 q9 8000750274847484 q10 4745470642334233 q11 534 387 350 350 q12 726 756 524 524 q13 3109344228012801 q14 275 288 246 246 q15 q16 679 695 612 612 q17 1298128012721272 q18 7307688268086808 q19 1120108810961088 q20 2235224119521952 q21 5359467345024502 q22 508 448 400 400 Total cold run time: 57763 ms Total hot run time: 51808 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
morrySnow commented on PR #64080: URL: https://github.com/apache/doris/pull/64080#issuecomment-4703921261 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
hello-stephen commented on PR #64080: URL: https://github.com/apache/doris/pull/64080#issuecomment-4666874357 TPC-DS: Total hot run time: 169669 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 06d2fea080e873ea81629116944125d44366e55e, data reload: false query5 4320660 479 479 query6 467 200 186 186 query7 4871565 308 308 query8 361 219 198 198 query9 8754405340254025 query10 474 314 264 264 query11 5941235721962196 query12 164 106 102 102 query13 1282636 421 421 query14 6408541151305130 query14_14416445944264426 query15 208 207 182 182 query16 1020449 445 445 query17 1147720 589 589 query18 2616495 350 350 query19 206 192 148 148 query20 121 116 110 110 query21 219 145 117 117 query22 13686 13632 13425 13425 query23 17558 16566 16193 16193 query23_116263 16336 16533 16336 query24 7585180713101310 query24_11337132512901290 query25 595 491 455 455 query26 1297333 170 170 query27 2719576 342 342 query28 4456203119771977 query29 1043622 474 474 query30 299 239 198 198 query31 11351080956 956 query32 101 61 62 61 query33 500 316 251 251 query34 11951134643 643 query35 768 782 705 705 query36 1363136712561256 query37 161 108 88 88 query38 3228318930343034 query39 981 915 898 898 query39_1871 871 892 871 query40 222 123 104 104 query41 64 62 64 62 query42 97 98 96 96 query43 326 326 294 294 query44 query45 197 188 184 184 query46 10711187740 740 query47 2358235921862186 query48 407 412 289 289 query49 628 470 360 360 query50 1014347 254 254 query51 4439430142614261 query52 87 88 77 77 query53 249 275 193 193 query54 266 225 193 193 query55 80 75 68 68 query56 230 228 214 214 query57 1398140512761276 query58 258 214 220 214 query59 1594170514201420 query60 298 256 236 236 query61 158 157 155 155 query62 719 648 590 590 query63 226 187 188 187 query64 2567788 632 632 query65 query66 1770465 379 379 query67 29931 29795 29953 29795 query68 query69 444 326 272 272 query70 1037990 958 958 query71 297 231 211 211 query72 3011278724122412 query73 890 813 436 436 query74 5221499047954795 query75 2683261722452245 query76 23591136783 783 query77 360 382 298 298 query78 12408 12505 11971 11971 query79 14741079760 760 query80 1148458 390 390 query81 498 285 247 247 query82 578 161 122 122 query83 355 281 249 249 query84 query85 932 527 437 437 query86 414 305 275 275 query87 3498337732303230 query88 3684281227632763 query89 431 387 331 331 query90 1792186 182 182 query91 179 162 138 138 query92 66 65 60 60 query93 14101583886 886 query94 647 364 323 323 query95 687 384 454 384 query96 1076831 342 342 query97 2710269925662566 query98 215 208 204 204 query99 1155117210681068 Total cold run time: 252693 ms Total hot run time: 169669 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about th
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
hello-stephen commented on PR #64080: URL: https://github.com/apache/doris/pull/64080#issuecomment-4666821191 TPC-H: Total hot run time: 29594 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 06d2fea080e873ea81629116944125d44366e55e, data reload: false -- Round 1 -- orders Doris NULLNULL0 0 0 NULL0 NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 17779 411241094109 q2 q3 10756 1334791 791 q4 4686480 356 356 q5 7536860 586 586 q6 186 175 139 139 q7 797 845 650 650 q8 9775169616861686 q9 7029456445104510 q10 6871180815291529 q11 432 272 249 249 q12 629 423 302 302 q13 18125 348027652765 q14 267 261 243 243 q15 q16 824 774 706 706 q17 954 932 957 932 q18 6978579557035703 q19 1301129910671067 q20 531 418 263 263 q21 6152271526822682 q22 457 372 326 326 Total cold run time: 102065 ms Total hot run time: 29594 ms - Round 2, with runtime_filter_mode=off - orders Doris NULLNULL15000 42 6422171781 NULL22778155NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 5163482948554829 q2 q3 5121522446364636 q4 2135220514221422 q5 4849483046784678 q6 244 187 134 134 q7 1894175915641564 q8 2531213119261926 q9 7542749274317431 q10 4777472442394239 q11 536 389 355 355 q12 726 735 535 535 q13 3030344428292829 q14 275 284 266 266 q15 q16 681 708 615 615 q17 1298127712771277 q18 7540695568086808 q19 1127113311261126 q20 2224225319731973 q21 5349463944784478 q22 554 454 407 407 Total cold run time: 57596 ms Total hot run time: 51528 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
github-actions[bot] commented on code in PR #64080:
URL: https://github.com/apache/doris/pull/64080#discussion_r3385314788
##
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushProjectThroughUnion.java:
##
@@ -60,7 +61,12 @@ public static boolean canPushProject(List
projects, LogicalSetO
if (e instanceof SlotReference) {
return true;
} else {
-Expression expr =
ExpressionUtils.getExpressionCoveredByCast(e.child(0));
+Expression expr;
+if (logicalSetOperation.getQualifier().equals(Qualifier.ALL)) {
+expr =
ExpressionUtils.getExpressionCoveredByCast(e.child(0));
+} else {
Review Comment:
This still lets `UNION DISTINCT` run after a projection that is not
injective as a tuple. The check proves each expression is a slot or an
injective cast of one slot, but it never verifies that the referenced slots
form a permutation of the set-op output. For example:
```sql
select cast(a as bigint), cast(a as bigint)
from (select 1 a, 2 b union select 1, 3) t;
```
Before the rewrite, the inner union keeps two distinct `(a,b)` rows and the
outer project returns two `(1,1)` rows. After this rule pushes the project into
both children, the union distinct sees only `(1,1)` and collapses them to one
row. For non-`UNION ALL` set ops, please require the stripped slots to cover
every output slot exactly once (or otherwise prove the whole projection vector
is injective), not just each expression independently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
morrySnow commented on PR #64080: URL: https://github.com/apache/doris/pull/64080#issuecomment-4666154150 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
morrySnow commented on PR #64080: URL: https://github.com/apache/doris/pull/64080#issuecomment-4666153162 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
morrySnow commented on code in PR #64080:
URL: https://github.com/apache/doris/pull/64080#discussion_r3385279799
##
fe/fe-core/src/main/java/org/apache/doris/nereids/types/JsonType.java:
##
@@ -34,6 +35,11 @@ public class JsonType extends PrimitiveType {
private JsonType() {
}
+@Override
+public boolean isInjectiveCastTo(DataType target) {
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
github-actions[bot] commented on PR #64080: URL: https://github.com/apache/doris/pull/64080#issuecomment-4666058215 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
hello-stephen commented on PR #64080: URL: https://github.com/apache/doris/pull/64080#issuecomment-4659570864 # FE Regression Coverage Report Increment line coverage ` 7.19% (12/167)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/64080_b3c474ca2f33194267346d7bf418d79823f9b199_merge_fe/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/64080_b3c474ca2f33194267346d7bf418d79823f9b199_merge_fe/report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
hello-stephen commented on PR #64080: URL: https://github.com/apache/doris/pull/64080#issuecomment-4652995408 TPC-DS: Total hot run time: 170047 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit b3c474ca2f33194267346d7bf418d79823f9b199, data reload: false query5 4322629 514 514 query6 449 199 182 182 query7 4854562 304 304 query8 372 231 205 205 query9 8778411041034103 query10 443 310 271 271 query11 5973238422172217 query12 162 107 99 99 query13 1246611 441 441 query14 6630550051505150 query14_14493450744264426 query15 214 203 181 181 query16 1026424 460 424 query17 1110701 585 585 query18 2735484 342 342 query19 212 180 143 143 query20 119 118 108 108 query21 213 144 117 117 query22 13821 13698 13519 13519 query23 17411 16667 16216 16216 query23_116352 16281 16375 16281 query24 7523179813271327 query24_11326132713501327 query25 587 442 400 400 query26 1286315 173 173 query27 2620551 335 335 query28 4409203120172017 query29 1097646 490 490 query30 321 241 203 203 query31 11241073948 948 query32 137 63 59 59 query33 549 320 269 269 query34 12151165663 663 query35 759 784 700 700 query36 1387138912291229 query37 166 107 93 93 query38 3239318931013101 query39 954 932 912 912 query39_1900 873 861 861 query40 235 131 109 109 query41 73 72 69 69 query42 99 100 99 99 query43 332 338 297 297 query44 query45 203 201 199 199 query46 11101288734 734 query47 2323230922572257 query48 405 417 304 304 query49 656 523 349 349 query50 1040358 255 255 query51 4375432442624262 query52 89 90 78 78 query53 246 276 191 191 query54 286 223 198 198 query55 81 76 70 70 query56 259 221 212 212 query57 1450139812971297 query58 248 214 214 214 query59 1614175414661466 query60 282 252 236 236 query61 157 154 158 154 query62 702 654 586 586 query63 240 189 190 189 query64 2515814 623 623 query65 query66 1742470 344 344 query67 29696 29727 29512 29512 query68 query69 430 300 271 271 query70 10181002943 943 query71 306 230 213 213 query72 3107298624762476 query73 874 769 446 446 query74 5162500348004800 query75 2709257422572257 query76 23711169780 780 query77 377 395 297 297 query78 12361 12430 11897 11897 query79 14791092779 779 query80 610 470 406 406 query81 465 283 250 250 query82 923 165 119 119 query83 367 285 256 256 query84 query85 912 549 452 452 query86 418 302 284 284 query87 3463336531833183 query88 3691276627492749 query89 425 402 334 334 query90 1976181 191 181 query91 181 167 140 140 query92 62 63 62 62 query93 14751478902 902 query94 553 336 310 310 query95 690 402 448 402 query96 1006816 368 368 query97 2700270525562556 query98 218 224 200 200 query99 1187119210321032 Total cold run time: 252870 ms Total hot run time: 170047 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about thi
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
hello-stephen commented on PR #64080: URL: https://github.com/apache/doris/pull/64080#issuecomment-4652901980 TPC-H: Total hot run time: 29672 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit b3c474ca2f33194267346d7bf418d79823f9b199, data reload: false -- Round 1 -- orders Doris NULLNULL0 0 0 NULL0 NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 17624 403139893989 q2 q3 10809 1424832 832 q4 4690477 341 341 q5 7549874 612 612 q6 189 177 138 138 q7 793 841 630 630 q8 9342169516851685 q9 5800455045564550 q10 6774180215601560 q11 441 299 259 259 q12 629 451 301 301 q13 20720 353328362836 q14 278 267 244 244 q15 q16 839 791 755 755 q17 10241040923 923 q18 7070593556745674 q19 1386121010801080 q20 523 595 306 306 q21 6079285326412641 q22 457 376 316 316 Total cold run time: 103016 ms Total hot run time: 29672 ms - Round 2, with runtime_filter_mode=off - orders Doris NULLNULL15000 42 6422171781 NULL22778155NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 5010473048104730 q2 q3 4974533446824682 q4 2100223813931393 q5 5063472647034703 q6 235 179 127 127 q7 2043175015621562 q8 2426216921852169 q9 7827749274867486 q10 4794468442724272 q11 549 429 382 382 q12 738 745 535 535 q13 3052345127652765 q14 281 281 249 249 q15 q16 695 705 614 614 q17 1306127212731272 q18 7194686868586858 q19 1129108411271084 q20 2234222419721972 q21 5369468045374537 q22 538 473 428 428 Total cold run time: 57557 ms Total hot run time: 51820 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
hello-stephen commented on PR #64080: URL: https://github.com/apache/doris/pull/64080#issuecomment-4652120383 # FE Regression Coverage Report Increment line coverage ` 17.14% (12/70)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/64080_b3c474ca2f33194267346d7bf418d79823f9b199_merge_fe/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/64080_b3c474ca2f33194267346d7bf418d79823f9b199_merge_fe/report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
hello-stephen commented on PR #64080: URL: https://github.com/apache/doris/pull/64080#issuecomment-4651556927 # FE UT Coverage Report Increment line coverage ` 52.86% (37/70)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/64080_b3c474ca2f33194267346d7bf418d79823f9b199/fe_increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/64080_b3c474ca2f33194267346d7bf418d79823f9b199/fe_report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
github-actions[bot] commented on code in PR #64080:
URL: https://github.com/apache/doris/pull/64080#discussion_r3374932976
##
fe/fe-core/src/main/java/org/apache/doris/nereids/types/JsonType.java:
##
@@ -34,6 +35,11 @@ public class JsonType extends PrimitiveType {
private JsonType() {
}
+@Override
+public boolean isInjectiveCastTo(DataType target) {
Review Comment:
`JSONB -> STRING/VARCHAR/CHAR` is not injective, so this branch can still
make `PushProjectThroughUnion` change `UNION DISTINCT` results. In BE,
`DataTypeStringSerDeBase::deserialize_column_from_jsonb_vector` casts JSONB
strings by inserting the raw string payload, but non-string JSONB values
through `JsonbToJson`; therefore JSONB string `"1"` and JSONB number `1` are
distinct JSONB values before the cast but both become the same string `1`. With
this predicate, `select cast(j as string) from (select cast('\"1\"' as jsonb) j
union select cast('1' as jsonb) j) t` can have duplicate elimination moved
after the cast and collapse one row. Please keep JSONB-to-character unsafe
unless the cast output is proven to preserve JSONB type identity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
morrySnow commented on PR #64080: URL: https://github.com/apache/doris/pull/64080#issuecomment-4650537129 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
morrySnow commented on PR #64080: URL: https://github.com/apache/doris/pull/64080#issuecomment-4650535856 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
morrySnow commented on code in PR #64080:
URL: https://github.com/apache/doris/pull/64080#discussion_r3360073724
##
fe/fe-core/src/main/java/org/apache/doris/nereids/types/coercion/CharacterType.java:
##
@@ -42,6 +42,11 @@ public int getLen() {
return len;
}
+@Override
+public boolean canSafetyCastTo(DataType target) {
Review Comment:
好的
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
morrySnow commented on code in PR #64080:
URL: https://github.com/apache/doris/pull/64080#discussion_r3360073319
##
fe/fe-core/src/main/java/org/apache/doris/nereids/types/coercion/CharacterType.java:
##
@@ -42,6 +42,11 @@ public int getLen() {
return len;
}
+@Override
Review Comment:
be现在不处理这个，所以如果还有这种cast，他是安全的。cast之后也是100个字符
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
englefly commented on code in PR #64080:
URL: https://github.com/apache/doris/pull/64080#discussion_r3360058444
##
fe/fe-core/src/main/java/org/apache/doris/nereids/types/coercion/CharacterType.java:
##
@@ -42,6 +42,11 @@ public int getLen() {
return len;
}
+@Override
+public boolean canSafetyCastTo(DataType target) {
Review Comment:
canSafetyCastTo 的目的是: 把 cast 看成函数,要求这个是单射,对吗
如果是这样,建议改个名字 isInjectiveCastTo
##
fe/fe-core/src/main/java/org/apache/doris/nereids/types/coercion/CharacterType.java:
##
@@ -42,6 +42,11 @@ public int getLen() {
return len;
}
+@Override
Review Comment:
char[100] cast 成 char[10] 应该不能提升到 distinct 之上
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
hello-stephen commented on PR #64080: URL: https://github.com/apache/doris/pull/64080#issuecomment-4623171974 # FE UT Coverage Report Increment line coverage ` 52.86% (37/70)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/64080_0b76718bdd2b2371fc9eded250f9aeaff26aaa4f/fe_increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/64080_0b76718bdd2b2371fc9eded250f9aeaff26aaa4f/fe_report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
hello-stephen commented on PR #64080: URL: https://github.com/apache/doris/pull/64080#issuecomment-4623028350 # FE Regression Coverage Report Increment line coverage ` 17.14% (12/70)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/64080_0b76718bdd2b2371fc9eded250f9aeaff26aaa4f_merge_fe/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/64080_0b76718bdd2b2371fc9eded250f9aeaff26aaa4f_merge_fe/report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
hello-stephen commented on PR #64080: URL: https://github.com/apache/doris/pull/64080#issuecomment-4622516130 TPC-DS: Total hot run time: 169465 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 0b76718bdd2b2371fc9eded250f9aeaff26aaa4f, data reload: false query5 4327646 490 490 query6 436 204 179 179 query7 4809558 284 284 query8 360 218 213 213 query9 8745398939813981 query10 463 319 255 255 query11 5894234621322132 query12 150 101 101 101 query13 1260629 446 446 query14 6404538050775077 query14_14385439644194396 query15 215 199 175 175 query16 1046455 441 441 query17 1153689 564 564 query18 2565472 335 335 query19 193 176 136 136 query20 113 106 105 105 query21 224 132 125 125 query22 13623 13552 13377 13377 query23 17304 16604 16210 16210 query23_116229 16408 16246 16246 query24 7768176013021302 query24_11312132513341325 query25 557 477 372 372 query26 1284321 169 169 query27 2691561 346 346 query28 4481204520162016 query29 1085595 468 468 query30 312 235 199 199 query31 11031083937 937 query32 107 66 58 58 query33 544 329 243 243 query34 11611151661 661 query35 777 778 677 677 query36 1417142312571257 query37 152 102 91 91 query38 3207313530363036 query39 940 923 908 908 query39_1876 867 874 867 query40 215 125 102 102 query41 65 61 63 61 query42 93 90 92 90 query43 315 321 274 274 query44 query45 198 183 181 181 query46 11001264733 733 query47 2432233123182318 query48 404 421 303 303 query49 637 481 348 348 query50 1032352 249 249 query51 4348426242734262 query52 87 88 77 77 query53 245 263 186 186 query54 263 227 199 199 query55 80 75 69 69 query56 248 227 222 222 query57 1442141113251325 query58 242 215 207 207 query59 1577163014181418 query60 272 245 238 238 query61 161 159 157 157 query62 717 656 590 590 query63 233 189 184 184 query64 2535781 654 654 query65 query66 1771466 344 344 query67 29622 29569 29474 29474 query68 query69 415 298 268 268 query70 975 954 915 915 query71 309 223 210 210 query72 3003248523752375 query73 877 803 415 415 query74 5167494047604760 query75 2651257022302230 query76 23231136764 764 query77 369 380 293 293 query78 12374 12381 11827 11827 query79 12481008782 782 query80 563 516 419 419 query81 452 286 248 248 query82 248 160 121 121 query83 278 280 261 261 query84 270 150 115 115 query85 908 609 527 527 query86 339 302 289 289 query87 3360333632013201 query88 3639276127502750 query89 417 382 331 331 query90 2194185 177 177 query91 187 186 151 151 query92 62 61 61 61 query93 13711454851 851 query94 545 365 307 307 query95 733 375 445 375 query96 1115780 341 341 query97 2689270525452545 query98 209 206 205 205 query99 1179117610261026 Total cold run time: 250867 ms Total hot run time: 169465 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
hello-stephen commented on PR #64080: URL: https://github.com/apache/doris/pull/64080#issuecomment-4622433541 TPC-H: Total hot run time: 29503 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 0b76718bdd2b2371fc9eded250f9aeaff26aaa4f, data reload: false -- Round 1 -- orders Doris NULLNULL0 0 0 NULL0 NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 17633 404440854044 q2 q3 10744 1385814 814 q4 4686478 349 349 q5 7540885 588 588 q6 189 178 140 140 q7 788 855 631 631 q8 9433164616211621 q9 5999454945464546 q10 6749180015511551 q11 435 274 263 263 q12 635 432 290 290 q13 18108 333528122812 q14 270 262 245 245 q15 q16 824 780 713 713 q17 999 905 935 905 q18 6814577056795679 q19 1291121911181118 q20 519 408 262 262 q21 6272287426212621 q22 468 384 311 311 Total cold run time: 100396 ms Total hot run time: 29503 ms - Round 2, with runtime_filter_mode=off - orders Doris NULLNULL15000 42 6422171781 NULL22778155NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 5203474349704743 q2 q3 4944521846304630 q4 2119217614031403 q5 4864489347384738 q6 230 175 126 126 q7 1865176015841584 q8 2408213521442135 q9 7909785173547354 q10 4742468141994199 q11 557 387 349 349 q12 727 738 531 531 q13 3000335927792779 q14 274 277 267 267 q15 q16 673 690 635 635 q17 1287125512641255 q18 7294697368236823 q19 111211061106 q20 2208221119391939 q21 5272453744474447 q22 513 460 409 409 Total cold run time: 57201 ms Total hot run time: 51452 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
github-actions[bot] commented on code in PR #64080:
URL: https://github.com/apache/doris/pull/64080#discussion_r3355749053
##
fe/fe-core/src/main/java/org/apache/doris/nereids/types/ArrayType.java:
##
@@ -57,6 +58,14 @@ public DataType getItemType() {
return itemType;
}
+@Override
+public boolean canSafetyCastTo(DataType target) {
+if (target instanceof ArrayType) {
+return itemType.canSafetyCastTo(((ArrayType) target).itemType);
+}
+return target instanceof CharacterType;
Review Comment:
This marks `ARRAY -> STRING/VARCHAR/CHAR` as distinctness-preserving, but
the BE string form for arrays is not injective. `DataTypeArraySerDe::to_string`
joins elements with `, ` and nested `DataTypeStringSerDeBase::to_string` only
wraps string elements in quotes without escaping embedded quotes. For example,
`array('a', 'b')` and `array('a" , "b')`-style values can serialize to the same
text once delimiters/quotes are embedded in a string element, while they are
distinct as arrays. With this predicate, `Project(CAST(array_col AS STRING))`
can be pushed below `UNION DISTINCT`, so the distinct runs on the stringified
values and collapses a row that the original plan would keep. Please do not
treat complex-to-character casts as safe unless the serialization is proven
injective; the same concern applies to the `MapType` and `StructType` `target
instanceof CharacterType` branches added in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
github-actions[bot] commented on code in PR #64080:
URL: https://github.com/apache/doris/pull/64080#discussion_r3355740218
##
fe/fe-core/src/main/java/org/apache/doris/nereids/types/ArrayType.java:
##
@@ -57,6 +58,14 @@ public DataType getItemType() {
return itemType;
}
+@Override
+public boolean canSafetyCastTo(DataType target) {
+if (target instanceof ArrayType) {
+return itemType.canSafetyCastTo(((ArrayType) target).itemType);
+}
Review Comment:
`canSafetyCastTo` is now used to decide whether `UNION DISTINCT` duplicate
elimination may be moved after the cast, so `true` here must mean the cast is
injective. Complex-to-character casts are not always injective: for
arrays/maps/structs containing `FLOAT` or `DOUBLE`, direct
float/double-to-string is intentionally not marked safe, but complex
stringification delegates to the nested serde (for arrays,
`DataTypeArraySerDe::to_string` calls `nested_serde->to_string`) and BE formats
float/double with only `digits10 + 1` significant digits in `cast_to_string.h`.
Two distinct arrays whose float elements differ only beyond that formatted
precision can therefore stringify to the same value; after this rewrite the
pushed cast would make `UNION DISTINCT` collapse a row that the original plan
kept. The same unconditional `target instanceof CharacterType` pattern in
`MapType` and `StructType` has the same issue. Please either leave
complex-to-character casts unsafe or require the nested t
ypes themselves to be injective when cast to the character target.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
morrySnow commented on PR #64080: URL: https://github.com/apache/doris/pull/64080#issuecomment-4621618146 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
morrySnow commented on PR #64080: URL: https://github.com/apache/doris/pull/64080#issuecomment-4621616962 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
morrySnow commented on PR #64080: URL: https://github.com/apache/doris/pull/64080#issuecomment-4621607565 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
morrySnow commented on PR #64080: URL: https://github.com/apache/doris/pull/64080#issuecomment-4621606659 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
hello-stephen commented on PR #64080: URL: https://github.com/apache/doris/pull/64080#issuecomment-4613774169 TPC-DS: Total hot run time: 170262 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit f9d17b5358d804108f4fcef763d9dd5be1740aad, data reload: false query5 4313652 503 503 query6 456 198 181 181 query7 4849572 289 289 query8 375 226 209 209 query9 8738406440684064 query10 458 323 266 266 query11 5956236021522152 query12 155 104 99 99 query13 1256630 403 403 query14 6312539750745074 query14_14382437243444344 query15 210 198 176 176 query16 1011462 428 428 query17 1035687 550 550 query18 2443463 359 359 query19 200 178 140 140 query20 125 112 107 107 query21 214 136 117 117 query22 13624 13550 13391 13391 query23 17461 16602 16158 16158 query23_116229 16349 16408 16349 query24 7593178613181318 query24_11304132512951295 query25 596 475 427 427 query26 1289328 186 186 query27 2678585 366 366 query28 4465202420402024 query29 1137626 510 510 query30 326 243 203 203 query31 11261093967 967 query32 116 66 61 61 query33 547 336 260 260 query34 11871159662 662 query35 765 808 724 724 query36 1409139812661266 query37 158 110 92 92 query38 3189315530273027 query39 944 906 901 901 query39_1884 875 903 875 query40 224 128 106 106 query41 75 68 68 68 query42 100 97 95 95 query43 318 322 276 276 query44 query45 198 201 181 181 query46 10731224747 747 query47 2338239522862286 query48 433 430 301 301 query49 642 485 369 369 query50 1049354 262 262 query51 4339429042854285 query52 91 92 80 80 query53 248 271 189 189 query54 284 234 221 221 query55 82 80 74 74 query56 251 244 235 235 query57 1433140113181318 query58 259 226 219 219 query59 1611170514321432 query60 279 274 240 240 query61 159 164 162 162 query62 703 647 588 588 query63 232 187 188 187 query64 2551802 636 636 query65 query66 1813471 345 345 query67 29751 29871 29461 29461 query68 query69 422 306 277 277 query70 988 951 984 951 query71 307 221 213 213 query72 2966271122072207 query73 846 752 440 440 query74 5138492747854785 query75 2659259322622262 query76 23071155766 766 query77 359 372 305 305 query78 12394 12502 11852 11852 query79 1451996 746 746 query80 585 481 394 394 query81 448 281 247 247 query82 588 166 125 125 query83 369 285 251 251 query84 291 141 110 110 query85 926 558 443 443 query86 377 319 288 288 query87 3407332631833183 query88 3646274827202720 query89 417 388 330 330 query90 1957180 198 180 query91 178 181 138 138 query92 65 60 59 59 query93 14601414829 829 query94 526 358 325 325 query95 697 459 354 354 query96 1076785 336 336 query97 2670268725502550 query98 210 209 234 209 query99 1181118110771077 Total cold run time: 251355 ms Total hot run time: 170262 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
hello-stephen commented on PR #64080: URL: https://github.com/apache/doris/pull/64080#issuecomment-4613670617 TPC-H: Total hot run time: 29308 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit f9d17b5358d804108f4fcef763d9dd5be1740aad, data reload: false -- Round 1 -- orders Doris NULLNULL0 0 0 NULL0 NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 17591 410040374037 q2 q3 10784 1409829 829 q4 4693473 345 345 q5 7556892 580 580 q6 186 181 139 139 q7 780 844 659 659 q8 9363172916241624 q9 5889449144514451 q10 6761181715641564 q11 433 270 249 249 q12 634 437 287 287 q13 18141 338427882788 q14 264 259 246 246 q15 q16 793 772 705 705 q17 1000955 915 915 q18 7176571254685468 q19 1248125511031103 q20 506 405 260 260 q21 6007278027312731 q22 482 387 328 328 Total cold run time: 100287 ms Total hot run time: 29308 ms - Round 2, with runtime_filter_mode=off - orders Doris NULLNULL15000 42 6422171781 NULL22778155NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 5208492148764876 q2 q3 4922530447554755 q4 2188224314111411 q5 4892496247794779 q6 238 187 138 138 q7 1853181715981598 q8 2581226121962196 q9 8053780174647464 q10 4733467142614261 q11 537 411 377 377 q12 738 734 522 522 q13 3030354127662766 q14 285 279 247 247 q15 q16 686 716 614 614 q17 1321127812781278 q18 7274695169766951 q19 1125111011061106 q20 2228220919721972 q21 5424472148954721 q22 536 474 428 428 Total cold run time: 57852 ms Total hot run time: 52460 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
hello-stephen commented on PR #64080: URL: https://github.com/apache/doris/pull/64080#issuecomment-4613284305 # FE Regression Coverage Report Increment line coverage ` 100.00% (2/2)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/64080_f9d17b5358d804108f4fcef763d9dd5be1740aad_merge_fe/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/64080_f9d17b5358d804108f4fcef763d9dd5be1740aad_merge_fe/report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
hello-stephen commented on PR #64080: URL: https://github.com/apache/doris/pull/64080#issuecomment-4612353490 # FE UT Coverage Report Increment line coverage ` 100.00% (2/2)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/64080_f9d17b5358d804108f4fcef763d9dd5be1740aad/fe_increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/64080_f9d17b5358d804108f4fcef763d9dd5be1740aad/fe_report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
morrySnow commented on PR #64080: URL: https://github.com/apache/doris/pull/64080#issuecomment-4611601841 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
morrySnow commented on PR #64080: URL: https://github.com/apache/doris/pull/64080#issuecomment-4611602533 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
hello-stephen commented on PR #64080: URL: https://github.com/apache/doris/pull/64080#issuecomment-4611596158 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
[PR] [fix](fe) Prevent cast project pushdown through union distinct [doris]
morrySnow opened a new pull request, #64080: URL: https://github.com/apache/doris/pull/64080 ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: PushProjectThroughUnion allowed cast projections to be pushed below UNION DISTINCT. For non-injective casts, distinct was then evaluated after the cast instead of before the cast, which could incorrectly collapse rows that were distinct in the original union output. Restrict cast project pushdown to UNION ALL and add tests for the qualifier guard and the datetime-to-date regression case. ### Release note None ### Check List (For Author) - Test: Regression test / Unit Test - ./run-fe-ut.sh --run org.apache.doris.nereids.rules.rewrite.PushProjectThroughUnionTest - ./run-regression-test.sh --run -d nereids_syntax_p0 -s set_operation - mvn checkstyle:check -pl fe-core - Behavior changed: No - Does this need documentation: No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
