Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
morningman merged PR #44267: URL: https://github.com/apache/doris/pull/44267 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
github-actions[bot] commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2515905504 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
doris-robot commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2511525108 ClickBench: Total hot run time: 33.71 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 093f49e8380bd1193739fa32d9cf6332f1e61cec, data reload: false query1 0.030.030.03 query2 0.080.050.05 query3 0.230.060.05 query4 1.650.090.08 query5 0.410.390.41 query6 1.160.670.65 query7 0.020.020.02 query8 0.050.050.04 query9 0.540.510.49 query10 0.550.550.56 query11 0.170.120.12 query12 0.160.130.12 query13 0.620.600.59 query14 2.702.852.76 query15 0.930.840.84 query16 0.380.380.38 query17 1.081.061.08 query18 0.180.190.18 query19 1.981.772.03 query20 0.020.010.01 query21 15.36 0.680.68 query22 3.987.522.19 query23 18.28 1.411.33 query24 2.060.220.24 query25 0.150.090.08 query26 0.260.180.18 query27 0.090.080.08 query28 13.28 1.161.16 query29 12.62 3.433.41 query30 0.250.070.06 query31 2.860.420.41 query32 3.230.490.48 query33 3.033.033.14 query34 16.73 4.534.59 query35 4.604.594.61 query36 0.690.480.48 query37 0.190.160.17 query38 0.160.150.15 query39 0.060.040.04 query40 0.170.140.13 query41 0.100.050.05 query42 0.060.050.05 query43 0.050.040.05 Total cold run time: 111.2 s Total hot run time: 33.71 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
doris-robot commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2511511977 TPC-DS: Total hot run time: 196867 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 093f49e8380bd1193739fa32d9cf6332f1e61cec, data reload: false query1 1252991 962 962 query2 6264201520832015 query3 11081 457344144414 query4 66809 29165 23490 23490 query5 4998465 453 453 query6 419 198 182 182 query7 5517310 316 310 query8 333 243 248 243 query9 8840268526762676 query10 442 245 250 245 query11 17176 15209 15851 15209 query12 156 103 100 100 query13 1429418 406 406 query14 9831677079566770 query15 225 188 193 188 query16 7127507 493 493 query17 1065589 607 589 query18 1694311 324 311 query19 216 162 166 162 query20 119 114 111 111 query21 216 109 103 103 query22 4667459545774577 query23 34721 34215 34474 34215 query24 5560248625902486 query25 497 386 375 375 query26 666 148 154 148 query27 1872282 288 282 query28 4283249224462446 query29 711 413 411 411 query30 205 149 148 148 query31 1006813 877 813 query32 64 54 57 54 query33 431 284 289 284 query34 914 522 529 522 query35 888 761 780 761 query36 1109976 993 976 query37 131 70 71 70 query38 4596450344854485 query39 1518144815001448 query40 207 97 102 97 query41 44 41 44 41 query42 105 103 103 103 query43 538 483 502 483 query44 1211825 823 823 query45 198 167 169 167 query46 1172719 704 704 query47 2033201619251925 query48 406 306 333 306 query49 722 395 383 383 query50 878 407 408 407 query51 7455717471357135 query52 98 86 86 86 query53 246 182 177 177 query54 518 392 408 392 query55 86 73 80 73 query56 266 251 235 235 query57 1257113511041104 query58 214 207 213 207 query59 3108302329512951 query60 257 238 232 232 query61 102 110 103 103 query62 828 672 666 666 query63 213 215 198 198 query64 1334658 629 629 query65 3355318432293184 query66 699 302 305 302 query67 16074 15699 15776 15699 query68 3860565 551 551 query69 430 255 251 251 query70 1204107211441072 query71 372 250 248 248 query72 6366415140544054 query73 772 365 361 361 query74 10272 901089978997 query75 3372280227262726 query76 2010104311371043 query77 532 272 267 267 query78 10389 950694729472 query79 2137604 602 602 query80 1355415 424 415 query81 516 229 241 229 query82 1290120 128 120 query83 200 140 137 137 query84 282 72 73 72 query85 1008291 293 291 query86 417 296 297 296 query87 4686457145464546 query88 3828222921952195 query89 417 309 291 291 query90 1936189 188 188 query91 131 100 102 100 query92 68 48 58 48 query93 2753548 550 548 query94 841 283 331 283 query95 356 248 249 248 query96 625 278 279 278 query97 2871268826792679 query98 213 197 198 197 query99 1613131713491317 Total cold run time: 319794 ms Total hot run time: 196867 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
doris-robot commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2511480009 TPC-H: Total hot run time: 40078 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 093f49e8380bd1193739fa32d9cf6332f1e61cec, data reload: false -- Round 1 -- q1 17912 778573427342 q2 2044183 188 183 q3 10818 109311951093 q4 10468 738 781 738 q5 7606272127422721 q6 241 149 147 147 q7 1020617 614 614 q8 9241185019201850 q9 6658641565286415 q10 7040229423372294 q11 477 257 270 257 q12 431 233 235 233 q13 17770 304230683042 q14 238 212 209 209 q15 600 540 518 518 q16 664 588 600 588 q17 982 559 494 494 q18 7276675266536653 q19 1344909 964 909 q20 468 184 181 181 q21 4027332132863286 q22 385 311 324 311 Total cold run time: 107710 ms Total hot run time: 40078 ms - Round 2, with runtime_filter_mode=off - q1 7395729672897289 q2 324 239 225 225 q3 2925293329612933 q4 2167183818611838 q5 5653567856775677 q6 227 144 139 139 q7 2262181818281818 q8 3413354035153515 q9 8873894089748940 q10 3618354736263547 q11 586 513 518 513 q12 859 595 645 595 q13 12184 329932593259 q14 300 269 295 269 q15 577 519 538 519 q16 709 670 669 669 q17 1881161416151614 q18 8257779075727572 q19 1688153316221533 q20 2123190118651865 q21 5854552055975520 q22 639 594 567 567 Total cold run time: 72514 ms Total hot run time: 60416 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
doris-robot commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2511454692 TeamCity be ut coverage result: Function Coverage: 38.39% (9978/25989) Line Coverage: 29.46% (83688/284059) Region Coverage: 28.58% (43049/150617) Branch Coverage: 25.21% (21888/86834) Coverage Report: http://coverage.selectdb-in.cc/coverage/093f49e8380bd1193739fa32d9cf6332f1e61cec_093f49e8380bd1193739fa32d9cf6332f1e61cec/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
suxiaogang223 commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2511369124 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
doris-robot commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2511137654 TPC-DS: Total hot run time: 196512 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 45b432847a6f82cdd301080f845ef7fbb778b79e, data reload: false query1 1250933 945 933 query2 6238210320422042 query3 10987 423645064236 query4 66676 28539 23578 23578 query5 4969455 442 442 query6 430 189 196 189 query7 5473305 293 293 query8 327 244 234 234 query9 8454272226832683 query10 455 266 248 248 query11 17249 15294 15861 15294 query12 178 104 103 103 query13 1438435 413 413 query14 10177 713474447134 query15 223 190 191 190 query16 7067465 491 465 query17 1018583 563 563 query18 1847305 299 299 query19 196 152 153 152 query20 123 113 109 109 query21 210 108 102 102 query22 4864448044264426 query23 34759 34540 34102 34102 query24 5421246925282469 query25 519 383 377 377 query26 639 154 148 148 query27 1856283 276 276 query28 4751247724452445 query29 679 414 416 414 query30 205 153 153 153 query31 1036862 846 846 query32 66 56 55 55 query33 429 303 301 301 query34 913 500 539 500 query35 877 782 744 744 query36 1080983 965 965 query37 132 73 73 73 query38 4463439444294394 query39 1532150814591459 query40 201 97 101 97 query41 43 42 44 42 query42 112 96 98 96 query43 532 503 497 497 query44 1165815 803 803 query45 191 174 167 167 query46 1183727 724 724 query47 2022193119181918 query48 407 315 316 315 query49 733 394 370 370 query50 845 387 398 387 query51 7267722770277027 query52 107 89 89 89 query53 255 181 183 181 query54 521 414 407 407 query55 82 74 76 74 query56 255 242 249 242 query57 12761142 query58 234 216 217 216 query59 3226316130673067 query60 271 266 249 249 query61 111 104 104 104 query62 773 670 653 653 query63 207 182 195 182 query64 1375669 647 647 query65 3231320931653165 query66 719 304 303 303 query67 16074 15562 15476 15476 query68 3919557 538 538 query69 427 247 238 238 query70 1191112311401123 query71 327 245 253 245 query72 6411404541614045 query73 753 360 363 360 query74 10279 914590599059 query75 3408268126632663 query76 1888107110261026 query77 499 259 272 259 query78 10530 951794539453 query79 1482611 592 592 query80 897 448 438 438 query81 482 244 227 227 query82 1276120 116 116 query83 256 147 139 139 query84 290 71 71 71 query85 915 393 298 298 query86 351 291 301 291 query87 4857454145684541 query88 3760221621782178 query89 427 293 292 292 query90 1999195 184 184 query91 131 106 101 101 query92 62 49 52 49 query93 1995541 541 541 query94 770 283 276 276 query95 345 242 247 242 query96 628 276 282 276 query97 2870270226962696 query98 225 198 195 195 query99 1622132613021302 Total cold run time: 317896 ms Total hot run time: 196512 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
doris-robot commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2511107892 TPC-H: Total hot run time: 40571 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 45b432847a6f82cdd301080f845ef7fbb778b79e, data reload: false -- Round 1 -- q1 17577 773574357435 q2 2059182 163 163 q3 10661 113412471134 q4 10560 756 703 703 q5 7649279627362736 q6 251 152 152 152 q7 1006633 603 603 q8 9249188519631885 q9 6646654565526545 q10 7019232923792329 q11 468 264 267 264 q12 431 229 226 226 q13 17780 304430493044 q14 229 214 217 214 q15 580 541 532 532 q16 645 594 599 594 q17 987 599 573 573 q18 7543663167726631 q19 1331106310671063 q20 474 180 180 180 q21 4507333532523252 q22 462 313 319 313 Total cold run time: 108114 ms Total hot run time: 40571 ms - Round 2, with runtime_filter_mode=off - q1 7374737572967296 q2 323 231 228 228 q3 2953298530252985 q4 2112186018591859 q5 5663573256775677 q6 231 147 147 147 q7 2265181218361812 q8 3369356336203563 q9 8996906590439043 q10 3616362235643564 q11 618 502 494 494 q12 853 676 608 608 q13 12596 328332113211 q14 316 291 288 288 q15 580 533 541 533 q16 678 644 631 631 q17 1890166916111611 q18 8222784376347634 q19 1740166514551455 q20 2125190918771877 q21 5851561555525552 q22 637 591 572 572 Total cold run time: 73008 ms Total hot run time: 60640 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
suxiaogang223 commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2510870017 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
morningman commented on code in PR #44267: URL: https://github.com/apache/doris/pull/44267#discussion_r1865305204 ## fe/be-java-extensions/hudi-scanner/src/main/scala/org/apache/doris/hudi/BaseSplitReader.scala: ## @@ -497,8 +499,11 @@ abstract class BaseSplitReader(val split: HoodieSplit) { options: Map[String, String], hadoopConf: Configuration): PartitionedFile => Iterator[InternalRow] = { partitionedFile => { - val reader = new HoodieAvroHFileReader( -hadoopConf, partitionedFile.filePath.toPath, new CacheConfig(hadoopConf)) + var hadoopStorageConfiguration = new HadoopStorageConfiguration(hadoopConf); + var sotragePath = new StoragePath(partitionedFile.toPath.toUri.getPath); Review Comment: ```suggestion var storagePath = new StoragePath(partitionedFile.toPath.toUri.getPath); ``` ## fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java: ## @@ -142,6 +141,19 @@ public static String encodeObjectToString(T t) { @Override protected void setScanParams(TFileRangeDesc rangeDesc, Split split) { if (split instanceof PaimonSplit) { +PaimonSplit paimonSplit = (PaimonSplit) split; +if (rangeDesc.getFormatType() == TFileFormatType.FORMAT_JNI +&& paimonSplit.getSplit() == null) { +// fall back to JNI reader Review Comment: ```suggestion // fall back to native reader ``` ## be/src/vec/exec/scan/vfile_scanner.cpp: ## @@ -754,28 +750,10 @@ Status VFileScanner::_get_next_reader() { // create reader for specific format Status init_status; -TFileFormatType::type format_type = _params->format_type; +TFileFormatType::type format_type = range.format_type; // JNI reader can only push down column value range bool push_down_predicates = !_is_load && _params->format_type != TFileFormatType::FORMAT_JNI; -if (format_type == TFileFormatType::FORMAT_JNI && range.__isset.table_format_params) { Review Comment: This need to be reserved. Because when upgrading, the BE will be upgrade, and the old FE will send old format request to new BE. You can reserve this logic but mark it as deprecated in 3.1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
doris-robot commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2510519107 ClickBench: Total hot run time: 33.25 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 07cb8434938075bd10736b2672dbe6f3318b1e74, data reload: false query1 0.030.030.03 query2 0.080.050.05 query3 0.220.060.06 query4 1.660.080.08 query5 0.420.390.41 query6 1.150.660.67 query7 0.020.010.01 query8 0.050.050.04 query9 0.560.510.50 query10 0.550.550.55 query11 0.160.130.12 query12 0.150.140.13 query13 0.620.610.60 query14 2.862.732.73 query15 0.900.840.84 query16 0.390.370.37 query17 1.061.001.00 query18 0.200.180.19 query19 1.921.781.95 query20 0.020.020.01 query21 15.35 0.700.67 query22 4.827.451.80 query23 18.20 1.421.36 query24 2.140.220.22 query25 0.150.090.09 query26 0.260.180.18 query27 0.070.080.08 query28 13.25 1.161.14 query29 12.64 3.363.37 query30 0.250.060.06 query31 2.880.420.41 query32 3.230.490.49 query33 3.083.033.09 query34 16.88 4.574.56 query35 4.574.614.59 query36 0.660.470.48 query37 0.200.160.15 query38 0.160.150.15 query39 0.060.040.04 query40 0.160.130.12 query41 0.100.060.06 query42 0.060.050.05 query43 0.040.040.04 Total cold run time: 112.23 s Total hot run time: 33.25 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
doris-robot commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2510514535 TPC-DS: Total hot run time: 197569 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 07cb8434938075bd10736b2672dbe6f3318b1e74, data reload: false query1 1246976 955 955 query2 6232211020872087 query3 11053 446444754464 query4 67382 29089 23530 23530 query5 4887465 452 452 query6 434 189 178 178 query7 5506315 301 301 query8 324 233 224 224 query9 8645272527302725 query10 443 246 241 241 query11 17008 15317 15830 15317 query12 154 101 100 100 query13 1413434 418 418 query14 10719 712267486748 query15 222 214 201 201 query16 7252494 533 494 query17 1074605 594 594 query18 1922340 310 310 query19 230 159 178 159 query20 129 114 115 114 query21 202 110 104 104 query22 4576456846364568 query23 35462 35385 34557 34557 query24 5690253025312530 query25 503 375 399 375 query26 650 163 152 152 query27 1955282 292 282 query28 4559249225062492 query29 650 420 411 411 query30 211 158 146 146 query31 998 843 873 843 query32 62 53 97 53 query33 422 312 286 286 query34 944 519 526 519 query35 871 768 751 751 query36 1098949 934 934 query37 118 75 75 75 query38 4612446943114311 query39 1511149014601460 query40 207 105 104 104 query41 45 42 41 41 query42 110 100 99 99 query43 532 496 508 496 query44 1213828 820 820 query45 189 169 171 169 query46 1158741 752 741 query47 2059193719621937 query48 421 312 327 312 query49 744 383 400 383 query50 844 400 395 395 query51 7339722071937193 query52 99 85 84 84 query53 248 183 179 179 query54 500 410 398 398 query55 77 89 79 79 query56 246 251 236 236 query57 1246111311351113 query58 220 202 222 202 query59 3228309229572957 query60 267 253 239 239 query61 109 119 109 109 query62 795 682 660 660 query63 212 195 201 195 query64 1456659 637 637 query65 3305325532553255 query66 721 303 303 303 query67 16193 15680 15667 15667 query68 3740572 575 572 query69 471 247 248 247 query70 1203109111521091 query71 394 246 255 246 query72 6092415840554055 query73 772 360 371 360 query74 10188 908891049088 query75 3400267526672667 query76 1938111311061106 query77 493 274 275 274 query78 10543 944692779277 query79 2123595 613 595 query80 1351421 432 421 query81 524 248 231 231 query82 1283118 113 113 query83 259 144 140 140 query84 282 69 70 69 query85 1004343 314 314 query86 420 300 302 300 query87 4664460246194602 query88 3720222021942194 query89 419 301 285 285 query90 1936187 180 180 query91 135 102 102 102 query92 70 50 50 50 query93 3150533 549 533 query94 818 292 298 292 query95 358 249 240 240 query96 644 275 282 275 query97 2879271126892689 query98 221 204 202 202 query99 1610130512951295 Total cold run time: 322256 ms Total hot run time: 197569 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
doris-robot commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2510491661 TPC-H: Total hot run time: 39878 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 07cb8434938075bd10736b2672dbe6f3318b1e74, data reload: false -- Round 1 -- q1 17986 803572507250 q2 2056178 167 167 q3 10751 110411671104 q4 10533 701 761 701 q5 7612272526432643 q6 239 150 150 150 q7 1008639 602 602 q8 9246183819521838 q9 6705649965156499 q10 7011233023162316 q11 469 262 259 259 q12 425 228 231 228 q13 17782 301830773018 q14 249 214 209 209 q15 584 528 520 520 q16 667 587 575 575 q17 986 578 512 512 q18 7371678367666766 q19 13441043952 952 q20 457 201 178 178 q21 4042316930893089 q22 388 320 302 302 Total cold run time: 107911 ms Total hot run time: 39878 ms - Round 2, with runtime_filter_mode=off - q1 7372725972207220 q2 322 226 230 226 q3 2945296430472964 q4 2079187519621875 q5 5629567757295677 q6 220 143 141 141 q7 2331185418581854 q8 3353348235153482 q9 8978900990449009 q10 3627361535633563 q11 603 518 494 494 q12 816 624 647 624 q13 12102 331932413241 q14 305 272 280 272 q15 584 537 523 523 q16 677 665 640 640 q17 1848162615801580 q18 8438763476367634 q19 1693164316051605 q20 2164191418741874 q21 5769562454365436 q22 644 576 600 576 Total cold run time: 72499 ms Total hot run time: 60510 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
suxiaogang223 commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2510440109 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
doris-robot commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2510075041 ClickBench: Total hot run time: 33.4 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 68a9ea1d1a387cc158f225927fac2f422dfc6815, data reload: false query1 0.040.030.02 query2 0.070.030.03 query3 0.240.070.07 query4 1.620.100.11 query5 0.420.420.42 query6 1.140.680.67 query7 0.020.020.02 query8 0.040.040.03 query9 0.570.530.50 query10 0.540.570.54 query11 0.140.110.10 query12 0.140.110.12 query13 0.600.600.59 query14 2.702.782.83 query15 0.900.820.82 query16 0.380.380.38 query17 1.081.061.07 query18 0.230.210.20 query19 1.971.881.97 query20 0.010.010.01 query21 15.36 0.620.60 query22 2.832.492.32 query23 16.86 0.960.90 query24 3.121.121.39 query25 0.310.120.14 query26 0.380.150.13 query27 0.040.040.04 query28 10.30 1.091.07 query29 12.56 3.173.16 query30 0.250.060.06 query31 2.840.390.38 query32 3.270.460.47 query33 2.963.073.00 query34 17.12 4.444.46 query35 4.504.534.48 query36 0.670.480.48 query37 0.090.060.06 query38 0.050.030.03 query39 0.030.020.02 query40 0.170.130.13 query41 0.070.020.02 query42 0.030.020.02 query43 0.040.030.03 Total cold run time: 106.7 s Total hot run time: 33.4 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
doris-robot commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2510060776 TPC-DS: Total hot run time: 191399 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 68a9ea1d1a387cc158f225927fac2f422dfc6815, data reload: false query1 971 381 370 370 query2 6534212220542054 query3 6713205 213 205 query4 34205 23584 23592 23584 query5 4371463 442 442 query6 270 179 172 172 query7 4629301 304 301 query8 299 250 231 231 query9 9737270727192707 query10 478 253 257 253 query11 18395 15311 15223 15223 query12 147 101 101 101 query13 1646406 408 406 query14 9425717669676967 query15 258 171 179 171 query16 7972444 485 444 query17 1689581 562 562 query18 1928335 279 279 query19 357 150 141 141 query20 113 108 108 108 query21 203 101 98 98 query22 4605431342864286 query23 34715 35484 34808 34808 query24 11489 243423532353 query25 544 372 369 369 query26 1215155 151 151 query27 2803283 274 274 query28 8306243224372432 query29 657 401 396 396 query30 303 162 149 149 query31 1025801 818 801 query32 96 55 56 55 query33 786 291 281 281 query34 970 504 524 504 query35 872 735 720 720 query36 1107940 950 940 query37 191 78 72 72 query38 4412424943214249 query39 1481142014231420 query40 285 99 98 98 query41 51 47 45 45 query42 113 100 97 97 query43 526 488 491 488 query44 1226795 787 787 query45 180 164 161 161 query46 1170693 720 693 query47 1978184718811847 query48 395 313 316 313 query49 1262379 403 379 query50 802 392 368 368 query51 7164705470677054 query52 100 91 87 87 query53 256 180 183 180 query54 1104383 386 383 query55 77 76 81 76 query56 260 229 238 229 query57 1238113211031103 query58 219 225 211 211 query59 3178291929652919 query60 266 232 244 232 query61 110 106 107 106 query62 883 668 684 668 query63 213 184 178 178 query64 5009690 640 640 query65 3286326732343234 query66 1319309 304 304 query67 16031 15508 15695 15508 query68 5500540 541 540 query69 405 254 248 248 query70 1218114611241124 query71 351 254 247 247 query72 6063403840334033 query73 772 363 365 363 query74 10369 900890669008 query75 3370263926062606 query76 3285111410941094 query77 550 270 270 270 query78 10425 936493539353 query79 1699596 603 596 query80 1113432 423 423 query81 524 233 233 233 query82 927 113 117 113 query83 255 147 139 139 query84 229 121 69 69 query85 1379297 288 288 query86 369 305 312 305 query87 4812457545944575 query88 3390224021842184 query89 398 292 299 292 query90 2111183 180 180 query91 138 99 101 99 query92 57 50 50 50 query93 1093548 548 548 query94 1132285 287 285 query95 364 243 246 243 query96 612 274 282 274 query97 2862265927202659 query98 228 196 188 188 query99 1546130313091303 Total cold run time: 302164 ms Total hot run time: 191399 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to th
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
doris-robot commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2510030669 TPC-H: Total hot run time: 40361 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 68a9ea1d1a387cc158f225927fac2f422dfc6815, data reload: false -- Round 1 -- q1 17617 795572457245 q2 2054188 188 188 q3 10596 116312341163 q4 10246 755 669 669 q5 7600279826992699 q6 238 145 146 145 q7 1015628 602 602 q8 9254193019351930 q9 6715658365366536 q10 7003227323352273 q11 453 260 263 260 q12 414 229 229 229 q13 17769 299830082998 q14 254 217 210 210 q15 570 534 526 526 q16 628 589 620 589 q17 994 573 576 573 q18 7502676767876767 q19 1340995 1014995 q20 469 185 187 185 q21 4150327633563276 q22 375 303 314 303 Total cold run time: 107256 ms Total hot run time: 40361 ms - Round 2, with runtime_filter_mode=off - q1 7238725272257225 q2 328 227 235 227 q3 2925287430142874 q4 2075185118621851 q5 5656570056555655 q6 218 138 139 138 q7 2168176118511761 q8 3353352235083508 q9 8948907690679067 q10 3593354835703548 q11 606 504 510 504 q12 814 640 603 603 q13 10661 318332083183 q14 303 270 270 270 q15 570 547 528 528 q16 666 642 617 617 q17 1799159215701570 q18 7969750875377508 q19 1661155215761552 q20 2074182718161816 q21 5430518253365182 q22 661 559 571 559 Total cold run time: 69716 ms Total hot run time: 59746 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
suxiaogang223 commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2509816140 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
github-actions[bot] commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2508978514 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
github-actions[bot] commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2508978506 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
doris-robot commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2507123288 TeamCity be ut coverage result: Function Coverage: 38.33% (9976/26029) Line Coverage: 29.41% (83552/284058) Region Coverage: 28.53% (42996/150693) Branch Coverage: 25.15% (21840/86822) Coverage Report: http://coverage.selectdb-in.cc/coverage/b1d46693ee6d59f851f18ad679586e5bd558d2d5_b1d46693ee6d59f851f18ad679586e5bd558d2d5/report/index.html -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
doris-robot commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2507100810 ClickBench: Total hot run time: 32.94 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit b1d46693ee6d59f851f18ad679586e5bd558d2d5, data reload: false query1 0.030.030.03 query2 0.060.030.03 query3 0.230.060.06 query4 1.640.100.11 query5 0.420.400.39 query6 1.150.650.64 query7 0.020.010.01 query8 0.040.030.02 query9 0.560.520.51 query10 0.550.560.56 query11 0.140.100.11 query12 0.140.120.12 query13 0.610.590.60 query14 2.812.842.74 query15 0.890.820.82 query16 0.380.380.39 query17 1.001.051.01 query18 0.230.220.22 query19 1.851.811.88 query20 0.020.010.01 query21 15.36 0.630.57 query22 2.233.061.75 query23 17.01 0.860.89 query24 2.931.401.47 query25 0.160.030.21 query26 0.590.150.14 query27 0.070.040.04 query28 10.22 1.101.07 query29 12.52 3.343.34 query30 0.250.060.05 query31 2.880.370.38 query32 3.280.470.46 query33 2.972.993.01 query34 17.06 4.454.45 query35 4.534.504.49 query36 0.680.480.49 query37 0.090.060.06 query38 0.050.030.04 query39 0.030.030.02 query40 0.160.120.12 query41 0.080.030.02 query42 0.040.020.02 query43 0.040.030.04 Total cold run time: 106 s Total hot run time: 32.94 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
doris-robot commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2507096253 TPC-DS: Total hot run time: 191002 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit b1d46693ee6d59f851f18ad679586e5bd558d2d5, data reload: false query1 988 385 366 366 query2 6509220821242124 query3 6706208 210 208 query4 34292 23508 23824 23508 query5 4379455 440 440 query6 294 184 187 184 query7 4634303 300 300 query8 298 235 230 230 query9 9722274527272727 query10 479 261 259 259 query11 18122 15371 15245 15245 query12 155 105 99 99 query13 1677426 416 416 query14 10174 653467446534 query15 282 186 186 186 query16 8148455 491 455 query17 1682583 575 575 query18 2143290 303 290 query19 365 147 151 147 query20 115 113 108 108 query21 204 101 98 98 query22 4446417141934171 query23 35108 34254 34311 34254 query24 10772 249925302499 query25 658 408 378 378 query26 1639153 154 153 query27 2894277 285 277 query28 8248244424232423 query29 988 415 404 404 query30 296 151 143 143 query31 1043813 819 813 query32 122 56 63 56 query33 779 296 311 296 query34 956 509 556 509 query35 905 777 741 741 query36 1101936 972 936 query37 145 77 82 77 query38 4280421641564156 query39 1482144715301447 query40 286 103 100 100 query41 50 48 46 46 query42 115 98 100 98 query43 554 520 498 498 query44 1242845 804 804 query45 186 164 168 164 query46 1155706 688 688 query47 1952183418401834 query48 406 304 320 304 query49 1221382 409 382 query50 809 388 388 388 query51 7165697370946973 query52 97 92 91 91 query53 251 180 188 180 query54 1234401 411 401 query55 88 77 84 77 query56 261 226 237 226 query57 1245112011401120 query58 240 212 214 212 query59 3264317432873174 query60 293 262 238 238 query61 109 110 112 110 query62 886 722 693 693 query63 218 187 186 186 query64 5109655 635 635 query65 3313320132263201 query66 1435314 312 312 query67 16238 15574 15678 15574 query68 5340554 554 554 query69 415 250 249 249 query70 1212111711061106 query71 351 246 266 246 query72 6435414741934147 query73 774 361 369 361 query74 10535 914389268926 query75 3454266627792666 query76 3173112210601060 query77 501 266 285 266 query78 10406 933393239323 query79 2546608 632 608 query80 1187425 426 425 query81 524 236 238 236 query82 732 121 121 121 query83 244 151 143 143 query84 233 74 77 74 query85 1566310 303 303 query86 457 307 302 302 query87 4683457645244524 query88 4093222422112211 query89 408 303 299 299 query90 2167190 187 187 query91 138 107 104 104 query92 65 55 52 52 query93 1630550 538 538 query94 1078294 296 294 query95 363 252 257 252 query96 623 280 284 280 query97 2849267926902679 query98 226 192 193 192 query99 1538133313471333 Total cold run time: 306598 ms Total hot run time: 191002 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to t
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
doris-robot commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2507087683 TPC-H: Total hot run time: 40067 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit b1d46693ee6d59f851f18ad679586e5bd558d2d5, data reload: false -- Round 1 -- q1 17596 802173497349 q2 2049179 166 166 q3 10584 116711241124 q4 10214 724 780 724 q5 7612277227302730 q6 244 148 146 146 q7 990 621 595 595 q8 9231193119281928 q9 6663647966196479 q10 7031234923082308 q11 461 259 268 259 q12 429 222 224 222 q13 17754 298430102984 q14 250 207 212 207 q15 570 540 518 518 q16 682 581 603 581 q17 1009544 539 539 q18 7412670867876708 q19 1342974 973 973 q20 471 183 179 179 q21 4008319030283028 q22 388 328 320 320 Total cold run time: 106990 ms Total hot run time: 40067 ms - Round 2, with runtime_filter_mode=off - q1 7304729272667266 q2 333 242 241 241 q3 2942281428412814 q4 1971171016871687 q5 5414542854155415 q6 223 143 139 139 q7 2136170017041700 q8 3256345734763457 q9 8712863786798637 q10 3507350434613461 q11 590 490 476 476 q12 811 572 590 572 q13 12136 303830253025 q14 299 253 277 253 q15 567 510 514 510 q16 691 636 646 636 q17 1812161016111610 q18 7983759874977497 q19 1693159514441444 q20 2089182318071807 q21 5468530552205220 q22 661 596 566 566 Total cold run time: 70598 ms Total hot run time: 58433 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
suxiaogang223 commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2507041951 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
suxiaogang223 commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2507024751 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
morningman commented on code in PR #44267: URL: https://github.com/apache/doris/pull/44267#discussion_r1862327637 ## be/src/vec/exec/format/table/hudi_jni_reader.cpp: ## @@ -65,16 +65,21 @@ HudiJniReader::HudiJniReader(const TFileScanRangeParams& scan_params, {"input_format", _hudi_params.input_format}}; // Use compatible hadoop client to read data -for (auto& kv : _scan_params.properties) { +for (const auto& kv : _scan_params.properties) { if (kv.first.starts_with(HOODIE_CONF_PREFIX)) { params[kv.first] = kv.second; } else { params[HADOOP_CONF_PREFIX + kv.first] = kv.second; } } -_jni_connector = std::make_unique("org/apache/doris/hudi/HudiJniScanner", params, -required_fields); +if (config::use_old_hudi_jni_reader) [[unlikely]] { Review Comment: Better use a session variable set hudi_jni_scanner = "spark" or "hadoop". and default is hadoop -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
wuwenchi commented on code in PR #44267: URL: https://github.com/apache/doris/pull/44267#discussion_r1859641277 ## gensrc/thrift/PlanNodes.thrift: ## @@ -448,6 +447,8 @@ struct TFileScanRangeParams { 22: optional TTextSerdeType text_serde_type // used by flexible partial update 23: optional string sequence_map_col +// if set true, be will be forced to use jni reader +24: bool force_jni_reader; Review Comment: Should this behavior be specified in FE? Otherwise, when checking the plan, we won't know how many JNIs have been used. ## fe/be-java-extensions/hadoop-hudi-scanner/src/main/java/org/apache/doris/hudi/HadoopHudiJniScanner.java: ## @@ -0,0 +1,272 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.hudi; + +import org.apache.doris.common.classloader.ThreadClassLoaderContext; +import org.apache.doris.common.jni.JniScanner; +import org.apache.doris.common.jni.vec.ColumnType; + +import com.google.common.base.Joiner; +import com.google.common.base.Preconditions; +import com.google.common.base.Strings; +import com.google.common.collect.Maps; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hive.common.JavaUtils; +import org.apache.hadoop.hive.serde2.Deserializer; +import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector; +import org.apache.hadoop.hive.serde2.objectinspector.StructField; +import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector; +import org.apache.hadoop.io.ArrayWritable; +import org.apache.hadoop.io.NullWritable; +import org.apache.hadoop.mapred.FileSplit; +import org.apache.hadoop.mapred.InputFormat; +import org.apache.hadoop.mapred.JobConf; +import org.apache.hadoop.mapred.RecordReader; +import org.apache.hadoop.mapred.Reporter; +import org.apache.hadoop.util.ReflectionUtils; +import org.apache.hudi.common.model.HoodieLogFile; +import org.apache.hudi.common.util.Option; +import org.apache.hudi.hadoop.realtime.HoodieRealtimeFileSplit; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.IOException; +import java.time.ZoneId; +import java.util.Arrays; +import java.util.List; +import java.util.Map; +import java.util.Properties; +import java.util.stream.Collectors; +import java.util.stream.IntStream; + +/** + * HadoopHudiJniScanner is a JniScanner implementation that reads Hudi data using hudi-hadoop-mr. + */ +public class HadoopHudiJniScanner extends JniScanner { +private static final Logger LOG = LoggerFactory.getLogger(HadoopHudiJniScanner.class); + +private static final String HADOOP_CONF_PREFIX = "hadoop_conf."; + +// Hudi data info +private final String basePath; +private final String dataFilePath; +private final long dataFileLength; +private final String[] deltaFilePaths; +private final String instantTime; +private final String serde; +private final String inputFormat; + +// schema info +private final String hudiColumnNames; +private final String[] hudiColumnTypes; +private final String[] requiredFields; +private List requiredColumnIds; +private ColumnType[] requiredTypes; +// private final String[] nestedFields; Review Comment: delete useless variable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
github-actions[bot] commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2498388906 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
github-actions[bot] commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2497793065 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
github-actions[bot] commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2497730771 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
github-actions[bot] commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2497131793 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
github-actions[bot] commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2493362005 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
github-actions[bot] commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2492991571 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
doris-robot commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2492815038 TPC-H: Total hot run time: 40408 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 01413781a887846e4a866acc572e23fc4a456004, data reload: false -- Round 1 -- q1 17633 761772987298 q2 2059174 186 174 q3 10706 111611621116 q4 10447 818 743 743 q5 7605286427412741 q6 244 145 144 144 q7 992 637 624 624 q8 9251192019591920 q9 6639646464736464 q10 7053233123302330 q11 472 259 260 259 q12 436 219 211 211 q13 17790 306430523052 q14 241 208 211 208 q15 570 523 522 522 q16 674 587 589 587 q17 1002583 599 583 q18 7401673867266726 q19 13631058975 975 q20 469 182 182 182 q21 3966323332483233 q22 383 321 316 316 Total cold run time: 107396 ms Total hot run time: 40408 ms - Round 2, with runtime_filter_mode=off - q1 7313781772677267 q2 326 228 231 228 q3 2920294729942947 q4 2066187418441844 q5 5695571957405719 q6 231 140 141 140 q7 2265186318411841 q8 3449360935503550 q9 8957900090549000 q10 3618356435933564 q11 598 523 499 499 q12 827 614 598 598 q13 10428 324332653243 q14 307 263 292 263 q15 592 509 522 509 q16 721 661 662 661 q17 1887165416131613 q18 8537788477257725 q19 1809156916121569 q20 2106190219101902 q21 5719551853795379 q22 658 565 549 549 Total cold run time: 71029 ms Total hot run time: 60610 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
suxiaogang223 commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2492727481 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
suxiaogang223 commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2492720597 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
github-actions[bot] commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2490546504 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
github-actions[bot] commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2490486337 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
doris-robot commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2490417180 ClickBench: Total hot run time: 32.65 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit da2c6d4220c27610dc537441dfcc887ff06a08ed, data reload: false query1 0.030.040.03 query2 0.060.040.03 query3 0.240.070.07 query4 1.630.100.11 query5 0.410.410.41 query6 1.140.660.66 query7 0.020.020.01 query8 0.040.020.03 query9 0.580.510.51 query10 0.560.560.56 query11 0.150.100.11 query12 0.130.100.11 query13 0.610.600.60 query14 2.772.832.75 query15 0.910.820.83 query16 0.370.380.40 query17 1.071.051.06 query18 0.220.210.20 query19 1.961.781.96 query20 0.020.010.01 query21 15.37 0.590.58 query22 2.492.572.21 query23 17.11 0.930.75 query24 2.630.781.17 query25 0.270.060.06 query26 0.400.130.14 query27 0.040.040.05 query28 11.35 1.091.07 query29 12.52 3.243.24 query30 0.250.060.06 query31 2.870.380.37 query32 3.310.460.46 query33 2.993.033.06 query34 16.63 4.504.43 query35 4.564.464.55 query36 0.680.490.48 query37 0.090.060.07 query38 0.050.040.04 query39 0.030.020.03 query40 0.170.120.12 query41 0.090.020.02 query42 0.040.030.02 query43 0.040.030.03 Total cold run time: 106.9 s Total hot run time: 32.65 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
doris-robot commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2490405100 TPC-DS: Total hot run time: 197412 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit da2c6d4220c27610dc537441dfcc887ff06a08ed, data reload: false query1 1271944 928 928 query2 6247210820472047 query3 10775 404139703970 query4 67474 29435 23721 23721 query5 4933463 461 461 query6 421 192 180 180 query7 5542307 290 290 query8 316 229 229 229 query9 8625268626572657 query10 467 272 288 272 query11 17288 15341 15929 15341 query12 161 110 109 109 query13 1477414 430 414 query14 10837 744475507444 query15 226 198 197 197 query16 7099468 488 468 query17 1034563 574 563 query18 1833307 312 307 query19 213 160 149 149 query20 126 108 114 108 query21 205 108 114 108 query22 4710449246244492 query23 34812 34207 34442 34207 query24 5490249925832499 query25 494 400 415 400 query26 682 153 157 153 query27 1902287 296 287 query28 4332245724442444 query29 685 428 427 427 query30 223 152 150 150 query31 997 817 840 817 query32 65 59 64 59 query33 405 296 276 276 query34 928 528 511 511 query35 893 739 738 738 query36 1123988 997 988 query37 117 69 77 69 query38 4458445744224422 query39 1514148314511451 query40 202 100 105 100 query41 48 43 45 43 query42 113 99 102 99 query43 541 516 501 501 query44 1206855 832 832 query45 189 178 167 167 query46 1174717 701 701 query47 2027191519501915 query48 437 340 334 334 query49 748 419 422 419 query50 844 397 396 396 query51 7406722970287028 query52 103 89 92 89 query53 268 181 178 178 query54 539 410 405 405 query55 81 78 89 78 query56 263 260 264 260 query57 1337120011381138 query58 219 208 222 208 query59 3401306429612961 query60 266 249 239 239 query61 113 108 110 108 query62 802 669 664 664 query63 217 189 185 185 query64 1370681 724 681 query65 3309324232343234 query66 720 287 308 287 query67 16004 15678 15764 15678 query68 4018585 569 569 query69 422 255 262 255 query70 1270113210971097 query71 363 243 257 243 query72 6358404040464040 query73 758 359 364 359 query74 10389 916289038903 query75 3430272826882688 query76 1880996 1156996 query77 484 275 279 275 query78 10511 947693809380 query79 2152611 605 605 query80 1390427 440 427 query81 531 227 225 225 query82 1277187 117 117 query83 269 163 149 149 query84 285 79 72 72 query85 1014316 298 298 query86 420 312 303 303 query87 4781472146814681 query88 3563226723092267 query89 412 303 297 297 query90 1956191 189 189 query91 135 106 101 101 query92 68 49 51 49 query93 2695543 548 543 query94 826 301 286 286 query95 360 253 250 250 query96 634 285 276 276 query97 2869269827082698 query98 226 192 198 192 query99 1601132113011301 Total cold run time: 321694 ms Total hot run time: 197412 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
doris-robot commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2490379157 TPC-H: Total hot run time: 40093 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit da2c6d4220c27610dc537441dfcc887ff06a08ed, data reload: false -- Round 1 -- q1 17587 751473067306 q2 2049181 173 173 q3 10545 109712091097 q4 10451 755 834 755 q5 7597271827192718 q6 244 150 147 147 q7 998 635 609 609 q8 9452188719031887 q9 6609647365206473 q10 7055232923242324 q11 463 262 253 253 q12 413 229 232 229 q13 19417 307031043070 q14 247 213 218 213 q15 583 541 516 516 q16 656 598 598 598 q17 983 614 515 515 q18 7305683267116711 q19 13291034990 990 q20 471 186 183 183 q21 3987322230083008 q22 378 319 318 318 Total cold run time: 108819 ms Total hot run time: 40093 ms - Round 2, with runtime_filter_mode=off - q1 7362749073687368 q2 340 233 241 233 q3 3057313630413041 q4 2183204319501950 q5 5825583159005831 q6 242 161 153 153 q7 2326185718361836 q8 3471365435253525 q9 9015898889958988 q10 3607359435453545 q11 600 523 516 516 q12 831 615 599 599 q13 16572 337131943194 q14 326 268 288 268 q15 571 536 529 529 q16 688 647 639 639 q17 1858163416391634 q18 8514767878107678 q19 2404157115081508 q20 2131187618921876 q21 5593553853655365 q22 621 551 582 551 Total cold run time: 78137 ms Total hot run time: 60827 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
github-actions[bot] commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2490312876 clang-tidy review says "All clean, LGTM! :+1:" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
suxiaogang223 commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2490307865 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
doris-robot commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2489066742 ClickBench: Total hot run time: 33.32 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 550a2f47ed646fd15758baf12332be335ddfdbf8, data reload: false query1 0.040.040.03 query2 0.060.030.03 query3 0.230.080.07 query4 1.620.100.11 query5 0.440.380.41 query6 1.150.670.66 query7 0.020.020.01 query8 0.050.030.04 query9 0.570.520.50 query10 0.560.570.57 query11 0.150.100.10 query12 0.130.120.11 query13 0.620.610.61 query14 2.702.842.82 query15 0.890.830.84 query16 0.390.390.40 query17 1.061.071.04 query18 0.200.210.20 query19 1.961.842.04 query20 0.010.010.01 query21 15.38 0.590.58 query22 2.592.702.32 query23 17.04 0.900.93 query24 3.190.981.27 query25 0.190.260.11 query26 0.500.140.14 query27 0.040.050.04 query28 10.69 1.091.09 query29 12.56 3.203.21 query30 0.250.060.06 query31 2.870.380.38 query32 3.280.460.46 query33 3.003.083.00 query34 16.97 4.504.52 query35 4.524.454.50 query36 0.640.480.47 query37 0.100.060.06 query38 0.050.040.04 query39 0.040.030.03 query40 0.150.120.12 query41 0.080.020.02 query42 0.030.030.02 query43 0.040.030.02 Total cold run time: 107.05 s Total hot run time: 33.32 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
doris-robot commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2489053705 TPC-DS: Total hot run time: 196370 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 550a2f47ed646fd15758baf12332be335ddfdbf8, data reload: false query1 1242947 916 916 query2 6267227421692169 query3 10766 401239613961 query4 67925 29270 23590 23590 query5 4986457 445 445 query6 415 183 171 171 query7 5482296 284 284 query8 317 231 231 231 query9 8730268026692669 query10 455 252 254 252 query11 17173 15278 15848 15278 query12 154 103 102 102 query13 1434433 426 426 query14 10904 696870036968 query15 211 192 185 185 query16 7103473 445 445 query17 1033556 558 556 query18 1771282 284 282 query19 185 159 141 141 query20 115 109 110 109 query21 221 99 99 99 query22 4416448845234488 query23 34436 34043 34024 34024 query24 5305251024242424 query25 483 379 384 379 query26 638 156 146 146 query27 1692284 285 284 query28 4334243424182418 query29 654 396 400 396 query30 210 145 153 145 query31 1000783 841 783 query32 65 54 55 54 query33 427 280 283 280 query34 913 512 519 512 query35 828 712 727 712 query36 1067962 981 962 query37 125 76 78 76 query38 4378420243124202 query39 1458146114361436 query40 199 98 100 98 query41 48 48 46 46 query42 115 102 99 99 query43 553 515 502 502 query44 1165809 814 809 query45 188 165 168 165 query46 1127695 687 687 query47 1998187118891871 query48 411 325 326 325 query49 746 406 405 405 query50 860 384 400 384 query51 7306703671177036 query52 102 90 87 87 query53 247 174 179 174 query54 513 394 431 394 query55 76 74 77 74 query56 252 227 251 227 query57 1289120611491149 query58 221 215 217 215 query59 3378314233763142 query60 270 257 250 250 query61 109 109 107 107 query62 816 672 658 658 query63 218 198 189 189 query64 1401691 659 659 query65 3273324132213221 query66 701 306 321 306 query67 16059 15813 15820 15813 query68 3978561 568 561 query69 424 252 267 252 query70 1186116211501150 query71 344 253 255 253 query72 6378410140564056 query73 757 359 352 352 query74 10261 903290709032 query75 3409268326572657 query76 1762106210151015 query77 488 274 284 274 query78 10603 946194799461 query79 1153600 622 600 query80 786 442 441 441 query81 531 243 221 221 query82 240 120 119 119 query83 180 153 149 149 query84 284 72 69 69 query85 836 294 332 294 query86 354 304 294 294 query87 4628464646794646 query88 3712219721522152 query89 428 306 295 295 query90 2009188 188 188 query91 131 104 100 100 query92 61 48 54 48 query93 1409542 544 542 query94 775 294 289 289 query95 347 249 248 248 query96 606 281 278 278 query97 2848273327202720 query98 211 196 197 196 query99 1753136513261326 Total cold run time: 315831 ms Total hot run time: 196370 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
doris-robot commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2489019142 TPC-H: Total hot run time: 39897 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 550a2f47ed646fd15758baf12332be335ddfdbf8, data reload: false -- Round 1 -- q1 17600 746372667266 q2 2052167 164 164 q3 10616 110012381100 q4 10552 761 679 679 q5 7624273927452739 q6 248 151 150 150 q7 1000669 628 628 q8 9246186119111861 q9 6553640663776377 q10 7054230822882288 q11 464 256 278 256 q12 415 216 225 216 q13 17778 303430283028 q14 238 216 210 210 q15 583 525 528 525 q16 675 583 576 576 q17 983 548 550 548 q18 7307663367166633 q19 1332104010661040 q20 485 174 182 174 q21 4287339231263126 q22 371 315 313 313 Total cold run time: 107463 ms Total hot run time: 39897 ms - Round 2, with runtime_filter_mode=off - q1 7224728272687268 q2 339 228 223 223 q3 2958286730062867 q4 2055179517971795 q5 5664572257065706 q6 236 141 140 140 q7 2270186218581858 q8 3451350535513505 q9 8879892188998899 q10 3624357635743574 q11 598 501 483 483 q12 838 634 612 612 q13 10980 324532323232 q14 311 270 286 270 q15 582 515 531 515 q16 697 653 652 652 q17 1863164916231623 q18 8216773376397639 q19 1729161715291529 q20 2113188618671867 q21 5518558154225422 q22 648 587 609 587 Total cold run time: 70793 ms Total hot run time: 60266 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
suxiaogang223 commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2488918977 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
Re: [PR] [fix](hudi) upgrade hudi to 0.15.0 [doris]
doris-robot commented on PR #44267: URL: https://github.com/apache/doris/pull/44267#issuecomment-2485050955 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org