Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
eldenmoon merged PR #57702: URL: https://github.com/apache/doris/pull/57702 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
gavinchou commented on code in PR #57702:
URL: https://github.com/apache/doris/pull/57702#discussion_r2567311170
##
gensrc/proto/olap_file.proto:
##
@@ -484,6 +487,9 @@ message TabletSchemaCloudPB {
optional int64 storage_page_size = 31 [default=65536];
optional int64 storage_dict_page_size = 32 [default=262144];
+// Whether new segments of this tablet use externalized ColumnMetaPB
layout (CMO)
+optional bool is_external_segment_meta_used_default = 33 [default = false];
Review Comment:
naming
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
gavinchou commented on code in PR #57702:
URL: https://github.com/apache/doris/pull/57702#discussion_r2567310464
##
be/src/olap/rowset/segment_v2/segment.h:
##
@@ -65,12 +65,19 @@ struct RowLocation;
namespace segment_v2 {
+// SegmentFooterPB versioning:
+// 1 = V2 baseline
+// 2 = V3 (externalized ColumnMetaPB region + CMO present)
+static constexpr uint32_t kSegmentFooterVersionV2 = 1;
+static constexpr uint32_t kSegmentFooterVersionV3_ExtColMeta = 2;
Review Comment:
make it pb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
github-actions[bot] commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3584329474 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
github-actions[bot] commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3584329396 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
hello-stephen commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3584190866 # FE Regression Coverage Report Increment line coverage ` 44.44% (8/18)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/57702_cc33be800f367dd823c6773f6272fec6bb00115d_merge_fe/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/57702_cc33be800f367dd823c6773f6272fec6bb00115d_merge_fe/report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
hello-stephen commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3584179744 # BE Regression && UT Coverage Report Increment line coverage `90.74% (823/907)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/57702_cc33be800f367dd823c6773f6272fec6bb00115d_merge/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/57702_cc33be800f367dd823c6773f6272fec6bb00115d_merge/report/index.html) | Category | Coverage | |---|| | Function Coverage | 71.74% (24584/34270) | | Line Coverage | 58.25% (256281/439988) | | Region Coverage | 53.26% (213218/400300) | | Branch Coverage | 54.83% (91250/166420) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
hello-stephen commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3584173029 # BE UT Coverage Report Increment line coverage `84.45% (766/907)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/cc33be800f367dd823c6773f6272fec6bb00115d_cc33be800f367dd823c6773f6272fec6bb00115d/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/cc33be800f367dd823c6773f6272fec6bb00115d_cc33be800f367dd823c6773f6272fec6bb00115d/report/index.html) | Category | Coverage | |---|| | Function Coverage | 52.90% (18500/34969) | | Line Coverage | 38.46% (169504/440731) | | Region Coverage | 33.24% (131559/395776) | | Branch Coverage | 34.21% (56675/165649) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
doris-robot commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3582701034 ClickBench: Total hot run time: 28.36 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit cc33be800f367dd823c6773f6272fec6bb00115d, data reload: false query1 0.050.050.05 query2 0.130.070.07 query3 0.320.070.07 query4 1.600.080.08 query5 0.260.250.25 query6 1.170.650.66 query7 0.050.030.02 query8 0.070.060.06 query9 0.640.540.54 query10 0.580.580.59 query11 0.250.140.13 query12 0.260.140.15 query13 0.650.620.62 query14 1.021.041.02 query15 0.920.860.86 query16 0.400.390.38 query17 1.041.031.05 query18 0.220.210.21 query19 1.971.811.83 query20 0.020.020.01 query21 15.40 0.290.24 query22 4.980.100.10 query23 15.36 0.390.23 query24 2.850.520.33 query25 0.100.090.10 query26 0.190.180.18 query27 0.100.100.09 query28 3.701.251.07 query29 12.58 4.003.31 query30 0.330.130.10 query31 2.840.630.44 query32 3.240.600.50 query33 3.143.043.12 query34 16.35 5.234.53 query35 4.544.494.55 query36 0.640.530.51 query37 0.220.090.09 query38 0.200.050.05 query39 0.060.050.05 query40 0.200.190.18 query41 0.110.060.06 query42 0.070.050.05 query43 0.060.050.05 Total cold run time: 98.88 s Total hot run time: 28.36 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
doris-robot commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3582681517 TPC-DS: Total hot run time: 183913 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit cc33be800f367dd823c6773f6272fec6bb00115d, data reload: false query1 1064411 403 403 query2 6567153916261539 query3 6756222 221 221 query4 25994 22513 22876 22513 query5 4371607 481 481 query6 340 237 216 216 query7 4650492 297 297 query8 317 250 241 241 query9 8692259325962593 query10 534 364 297 297 query11 15190 14810 14490 14490 query12 173 122 116 116 query13 1694588 447 447 query14 10227 900289458945 query15 228 205 184 184 query16 7185704 513 513 query17 1265773 650 650 query18 2009431 329 329 query19 215 214 184 184 query20 132 131 126 126 query21 218 143 115 115 query22 3943393438933893 query23 32937 31748 31766 31748 query24 8354243423892389 query25 621 542 476 476 query26 1246276 166 166 query27 2758501 345 345 query28 4290215421472147 query29 854 640 516 516 query30 309 244 217 217 query31 862 716 631 631 query32 86 74 74 74 query33 625 421 342 342 query34 808 862 580 580 query35 811 802 747 747 query36 919 965 827 827 query37 120 109 86 86 query38 3340332632903290 query39 1517142214211421 query40 222 131 119 119 query41 64 63 62 62 query42 127 111 113 111 query43 445 455 425 425 query44 1219740 751 740 query45 200 188 183 183 query46 874 992 640 640 query47 1657172116451645 query48 392 424 340 340 query49 777 508 443 443 query50 641 678 417 417 query51 3821393738523852 query52 106 117 108 108 query53 243 273 208 208 query54 303 288 283 283 query55 96 94 99 94 query56 333 336 323 323 query57 1157116811151115 query58 294 275 278 275 query59 2348252323302330 query60 386 352 345 345 query61 168 151 160 151 query62 797 718 656 656 query63 236 200 194 194 query64 44241195886 886 query65 4038399139423942 query66 1177449 338 338 query67 15068 14967 14895 14895 query68 8504889 612 612 query69 513 347 318 318 query70 1342121812511218 query71 484 331 322 322 query72 6005494451974944 query73 735 620 349 349 query74 8789891585128512 query75 3903335828342834 query76 37631108716 716 query77 827 419 317 317 query78 9561954189188918 query79 2548793 586 586 query80 634 585 492 492 query81 508 268 239 239 query82 420 164 132 132 query83 300 259 249 249 query84 312 113 97 97 query85 933 481 447 447 query86 382 302 282 282 query87 3441352034253425 query88 3483225222512251 query89 385 325 297 297 query90 1861229 230 229 query91 172 174 140 140 query92 83 70 66 66 query93 2070958 643 643 query94 725 460 352 352 query95 503 401 402 401 query96 497 569 278 278 query97 2931296428712871 query98 243 217 211 211 query99 1404136312501250 Total cold run time: 272095 ms Total hot run time: 183913 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
doris-robot commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3582621561 TPC-H: Total hot run time: 33969 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit cc33be800f367dd823c6773f6272fec6bb00115d, data reload: false -- Round 1 -- q1 16448 505948954895 q2 2070301 211 211 q3 97741291701 701 q4 9912895 367 367 q5 7445241023042304 q6 187 167 135 135 q7 919 773 627 627 q8 9156132711361136 q9 7046522352395223 q10 6796222418351835 q11 498 318 290 290 q12 327 367 221 221 q13 17608 365930023002 q14 238 242 218 218 q15 595 520 505 505 q16 10211022959 959 q17 585 857 362 362 q18 7647726070387038 q19 955 931 526 526 q20 351 342 234 234 q21 3598256122382238 q22 996 1006942 942 Total cold run time: 104172 ms Total hot run time: 33969 ms - Round 2, with runtime_filter_mode=off - q1 5002496049414941 q2 337 411 313 313 q3 2203262222332233 q4 1398174613191319 q5 4220410544954105 q6 214 177 135 135 q7 2113198618051805 q8 2642262326452623 q9 7655741173277327 q10 3152331427432743 q11 626 522 495 495 q12 689 766 646 646 q13 3561394633093309 q14 278 309 290 290 q15 548 510 518 510 q16 1095114010711071 q17 1165151213721372 q18 7881786777227722 q19 772 782 878 782 q20 1992208120182018 q21 4838436341694169 q22 10751030984 984 Total cold run time: 53456 ms Total hot run time: 50912 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
doris-robot commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3582555344 # Cloud UT Coverage Report Increment line coverage ` ` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/cc33be800f367dd823c6773f6272fec6bb00115d_cc33be800f367dd823c6773f6272fec6bb00115d_cloud/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/cc33be800f367dd823c6773f6272fec6bb00115d_cc33be800f367dd823c6773f6272fec6bb00115d_cloud/report/index.html) | Category | Coverage | |---|| | Function Coverage | 80.86% (1707/2111) | | Line Coverage | 66.90% (29908/44708) | | Region Coverage | 67.41% (14907/22114) | | Branch Coverage | 57.60% (7930/13768) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
eldenmoon commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3582383587 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
doris-robot commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3582095218 # BE UT Coverage Report Increment line coverage `77.62% (732/943)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/cd6e986b049504864f4a0b3022a7448188e95c8a_cd6e986b049504864f4a0b3022a7448188e95c8a/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/cd6e986b049504864f4a0b3022a7448188e95c8a_cd6e986b049504864f4a0b3022a7448188e95c8a/report/index.html) | Category | Coverage | |---|| | Function Coverage | 52.90% (18492/34954) | | Line Coverage | 38.44% (169298/440411) | | Region Coverage | 33.22% (131423/395588) | | Branch Coverage | 34.21% (56633/165554) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
eldenmoon commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3582070271 run p0 10 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
doris-robot commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3581659706 ClickBench: Total hot run time: 28.26 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit cd6e986b049504864f4a0b3022a7448188e95c8a, data reload: false query1 0.050.050.05 query2 0.110.060.06 query3 0.310.080.07 query4 1.600.090.09 query5 0.260.250.25 query6 1.180.650.64 query7 0.030.030.03 query8 0.070.060.07 query9 0.630.540.54 query10 0.590.580.58 query11 0.250.140.13 query12 0.250.140.14 query13 0.660.630.62 query14 1.031.041.02 query15 0.920.850.86 query16 0.400.400.38 query17 1.091.071.04 query18 0.220.210.21 query19 1.981.841.82 query20 0.020.010.01 query21 15.39 0.280.24 query22 5.010.100.10 query23 15.40 0.380.23 query24 2.860.490.31 query25 0.100.090.09 query26 0.180.180.18 query27 0.090.080.09 query28 3.591.281.08 query29 12.62 3.953.28 query30 0.350.120.11 query31 2.810.640.44 query32 3.240.600.49 query33 3.063.043.08 query34 16.57 5.204.49 query35 4.484.584.50 query36 0.640.520.50 query37 0.210.090.09 query38 0.190.060.05 query39 0.060.050.05 query40 0.210.170.18 query41 0.110.060.07 query42 0.070.040.04 query43 0.060.050.05 Total cold run time: 98.95 s Total hot run time: 28.26 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
doris-robot commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3581636255 TPC-DS: Total hot run time: 184290 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit cd6e986b049504864f4a0b3022a7448188e95c8a, data reload: false query1 1083420 388 388 query2 6598161216051605 query3 6751221 227 221 query4 26211 22956 23113 22956 query5 4426596 449 449 query6 338 242 226 226 query7 4655500 292 292 query8 301 252 247 247 query9 8694257125212521 query10 496 345 299 299 query11 15264 14992 14546 14546 query12 183 122 111 111 query13 1688570 452 452 query14 10427 887787658765 query15 216 207 183 183 query16 7231684 530 530 query17 1259778 619 619 query18 1991426 315 315 query19 208 204 167 167 query20 128 124 123 123 query21 215 134 111 111 query22 3838396038443844 query23 33050 31914 32023 31914 query24 8419235423962354 query25 610 531 464 464 query26 1231270 162 162 query27 2766494 338 338 query28 4367214021142114 query29 844 661 480 480 query30 307 243 207 207 query31 818 707 632 632 query32 83 72 73 72 query33 589 382 330 330 query34 801 904 553 553 query35 794 831 744 744 query36 911 923 836 836 query37 119 107 85 85 query38 3381335932453245 query39 1503139314141393 query40 221 125 116 116 query41 65 62 61 61 query42 129 111 113 111 query43 449 482 422 422 query44 1232749 748 748 query45 200 190 181 181 query46 889 990 644 644 query47 1713171416691669 query48 387 428 331 331 query49 781 490 398 398 query50 652 683 411 411 query51 4028392039153915 query52 115 110 105 105 query53 239 264 196 196 query54 307 291 280 280 query55 95 93 93 93 query56 327 320 329 320 query57 1140118710881088 query58 285 273 263 263 query59 2388255524312431 query60 355 351 335 335 query61 162 183 160 160 query62 790 708 643 643 query63 223 191 189 189 query64 45901179869 869 query65 4060397239833972 query66 1190424 343 343 query67 15179 15091 14915 14915 query68 4632931 626 626 query69 527 336 305 305 query70 1230124611201120 query71 415 330 313 313 query72 6014512251545122 query73 645 589 347 347 query74 8468890484558455 query75 3320334528182818 query76 33061121730 730 query77 515 413 323 323 query78 9554984490389038 query79 2387810 602 602 query80 1698628 584 584 query81 585 265 234 234 query82 422 166 128 128 query83 358 266 247 247 query84 262 114 93 93 query85 959 501 446 446 query86 462 284 285 284 query87 3514345033683368 query88 2875229222392239 query89 371 328 300 300 query90 1786231 226 226 query91 184 171 154 154 query92 70 72 65 65 query93 1306997 654 654 query94 739 435 332 332 query95 495 397 403 397 query96 495 568 283 283 query97 2877297328612861 query98 240 213 207 207 query99 1292140312871287 Total cold run time: 266988 ms Total hot run time: 184290 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
doris-robot commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3581586407 TPC-H: Total hot run time: 34306 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit cd6e986b049504864f4a0b3022a7448188e95c8a, data reload: false -- Round 1 -- q1 17597 506549374937 q2 2041321 206 206 q3 10194 1298729 729 q4 10256 980 387 387 q5 7519236823792368 q6 218 179 138 138 q7 918 798 630 630 q8 9361133310831083 q9 7334532853165316 q10 6917224018061806 q11 495 303 292 292 q12 368 371 222 222 q13 17786 369830163016 q14 248 241 210 210 q15 580 519 504 504 q16 10271012970 970 q17 596 869 357 357 q18 7384726370837083 q19 1101943 573 573 q20 349 337 223 223 q21 3751319523102310 q22 1014991 946 946 Total cold run time: 107054 ms Total hot run time: 34306 ms - Round 2, with runtime_filter_mode=off - q1 4973512449584958 q2 334 401 313 313 q3 2195272922912291 q4 1358177513391339 q5 4255450244594459 q6 240 181 131 131 q7 2039198319351935 q8 2700266225762576 q9 7621764275067506 q10 3035322829082908 q11 572 520 490 490 q12 681 805 826 805 q13 3545381833363336 q14 296 293 285 285 q15 564 536 511 511 q16 1143113210871087 q17 1155158313541354 q18 8061791374507450 q19 873 847 981 847 q20 2047222018741874 q21 4916445743294329 q22 1116103710371037 Total cold run time: 53719 ms Total hot run time: 51821 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
hello-stephen commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3581388663 # FE UT Coverage Report Increment line coverage ` 27.78% (5/18)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/57702_cd6e986b049504864f4a0b3022a7448188e95c8a/fe_increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/57702_cd6e986b049504864f4a0b3022a7448188e95c8a/fe_report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
hello-stephen commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3581252082 # Cloud UT Coverage Report Increment line coverage ` ` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/cd6e986b049504864f4a0b3022a7448188e95c8a_cd6e986b049504864f4a0b3022a7448188e95c8a_cloud/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/cd6e986b049504864f4a0b3022a7448188e95c8a_cd6e986b049504864f4a0b3022a7448188e95c8a_cloud/report/index.html) | Category | Coverage | |---|| | Function Coverage | 80.86% (1707/2111) | | Line Coverage | 66.86% (29892/44708) | | Region Coverage | 67.34% (14892/22114) | | Branch Coverage | 57.55% (7923/13768) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
eldenmoon commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3581146547 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
doris-robot commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3580477479 ClickBench: Total hot run time: 28.88 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 1c2b666189b0b34281c6dfe390dddf97b4e72219, data reload: false query1 0.050.050.05 query2 0.130.070.06 query3 0.310.070.08 query4 1.610.080.08 query5 0.260.250.26 query6 1.190.650.64 query7 0.030.030.02 query8 0.070.060.06 query9 0.640.520.53 query10 0.590.580.57 query11 0.260.130.14 query12 0.250.140.14 query13 0.660.640.63 query14 1.041.021.04 query15 0.930.870.86 query16 0.380.390.39 query17 1.031.051.04 query18 0.230.250.22 query19 1.941.781.83 query20 0.010.010.02 query21 15.41 0.290.24 query22 5.000.110.10 query23 15.38 0.370.22 query24 2.850.500.32 query25 0.100.090.09 query26 0.180.180.18 query27 0.090.090.09 query28 3.671.261.06 query29 12.59 4.163.51 query30 0.310.120.09 query31 2.860.630.43 query32 3.240.600.51 query33 3.023.143.11 query34 16.27 5.154.43 query35 4.574.904.98 query36 0.690.580.52 query37 0.250.090.10 query38 0.210.060.06 query39 0.060.050.05 query40 0.210.200.18 query41 0.120.070.06 query42 0.070.050.05 query43 0.070.050.05 Total cold run time: 98.83 s Total hot run time: 28.88 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
doris-robot commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3580456521 TPC-DS: Total hot run time: 184171 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 1c2b666189b0b34281c6dfe390dddf97b4e72219, data reload: false query1 1035409 405 405 query2 6567159816011598 query3 6756228 225 225 query4 26226 22868 22418 22418 query5 4455658 502 502 query6 353 256 229 229 query7 4653504 302 302 query8 309 278 262 262 query9 8739259326052593 query10 486 376 323 323 query11 15416 14760 14648 14648 query12 194 123 114 114 query13 1693583 465 465 query14 11945 922291119111 query15 217 202 192 192 query16 7691720 536 536 query17 1462794 646 646 query18 2042440 340 340 query19 219 207 185 185 query20 139 128 126 126 query21 213 135 116 116 query22 3855409039203920 query23 33108 31843 32181 31843 query24 8292242624392426 query25 624 559 519 519 query26 1243278 161 161 query27 2734482 351 351 query28 4337213321262126 query29 784 609 501 501 query30 301 255 210 210 query31 882 719 629 629 query32 82 75 77 75 query33 596 384 338 338 query34 810 867 522 522 query35 816 836 749 749 query36 901 953 874 874 query37 130 111 94 94 query38 3355332133113311 query39 1485144114131413 query40 220 130 117 117 query41 66 62 62 62 query42 126 114 114 114 query43 470 463 444 444 query44 1232750 762 750 query45 197 193 178 178 query46 876 990 634 634 query47 1688171216221622 query48 415 429 315 315 query49 788 528 431 431 query50 653 722 404 404 query51 3867392839113911 query52 114 115 102 102 query53 257 283 191 191 query54 309 301 269 269 query55 94 93 90 90 query56 312 323 317 317 query57 1179116310961096 query58 285 277 283 277 query59 2358249723752375 query60 354 355 346 346 query61 164 157 150 150 query62 789 719 643 643 query63 229 191 196 191 query64 44391211886 886 query65 4071398739813981 query66 1081432 325 325 query67 15244 15128 14802 14802 query68 8532948 618 618 query69 510 333 302 302 query70 1317120611931193 query71 499 335 315 315 query72 6007495448404840 query73 677 593 351 351 query74 8600878583588358 query75 4051331128712871 query76 38001125717 717 query77 816 407 314 314 query78 9626966888578857 query79 2408853 591 591 query80 677 589 490 490 query81 486 273 239 239 query82 437 161 132 132 query83 308 259 256 256 query84 308 124 97 97 query85 914 485 442 442 query86 336 312 287 287 query87 3574356334653465 query88 3302225922382238 query89 400 337 299 299 query90 1968230 221 221 query91 174 172 142 142 query92 83 71 67 67 query93 1513996 669 669 query94 730 447 345 345 query95 485 397 391 391 query96 518 583 283 283 query97 3006295528732873 query98 253 219 217 217 query99 1428139712701270 Total cold run time: 274765 ms Total hot run time: 184171 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
doris-robot commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3580410421 TPC-H: Total hot run time: 34931 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 1c2b666189b0b34281c6dfe390dddf97b4e72219, data reload: false -- Round 1 -- q1 17610 527149184918 q2 2041352 211 211 q3 10196 1334730 730 q4 10221 873 370 370 q5 7494240623302330 q6 178 166 134 134 q7 951 783 653 653 q8 9353131810791079 q9 7105536853155315 q10 6887221518161816 q11 514 300 292 292 q12 349 362 229 229 q13 17776 366630253025 q14 232 237 211 211 q15 583 509 511 509 q16 10401012980 980 q17 608 868 374 374 q18 7448752480177524 q19 1121951 556 556 q20 404 367 241 241 q21 4187335224162416 q22 1125112410181018 Total cold run time: 107423 ms Total hot run time: 34931 ms - Round 2, with runtime_filter_mode=off - q1 5211537451425142 q2 318 410 329 329 q3 2360293424962496 q4 1449195714431443 q5 4640457445584558 q6 213 167 128 128 q7 2042193618541854 q8 2636262426962624 q9 7636756474947494 q10 3187322527982798 q11 574 506 477 477 q12 693 760 604 604 q13 3297367830223022 q14 264 290 253 253 q15 534 491 501 491 q16 1028106310171017 q17 1104145213041304 q18 7358711772337117 q19 748 765 895 765 q20 1914197917971797 q21 4717436541804180 q22 10971039990 990 Total cold run time: 53020 ms Total hot run time: 50883 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
doris-robot commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3580060442 # Cloud UT Coverage Report Increment line coverage ` ` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/1c2b666189b0b34281c6dfe390dddf97b4e72219_1c2b666189b0b34281c6dfe390dddf97b4e72219_cloud/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/1c2b666189b0b34281c6dfe390dddf97b4e72219_1c2b666189b0b34281c6dfe390dddf97b4e72219_cloud/report/index.html) | Category | Coverage | |---|| | Function Coverage | 80.86% (1707/2111) | | Line Coverage | 66.89% (29906/44708) | | Region Coverage | 67.36% (14895/22114) | | Branch Coverage | 57.60% (7930/13768) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
hello-stephen commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-358006 # FE Regression Coverage Report Increment line coverage ` 44.44% (8/18)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/57702_d3845d1c83423ecb203afaae91b10c0f950d78f8_merge_fe/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/57702_d3845d1c83423ecb203afaae91b10c0f950d78f8_merge_fe/report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
eldenmoon commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3579961145 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
eldenmoon commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3579955992 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
doris-robot commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3579309187 ClickBench: Total hot run time: 28.32 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit d3845d1c83423ecb203afaae91b10c0f950d78f8, data reload: false query1 0.050.040.04 query2 0.120.060.06 query3 0.310.070.07 query4 1.610.080.08 query5 0.260.250.24 query6 1.170.660.65 query7 0.030.030.02 query8 0.060.060.08 query9 0.640.540.52 query10 0.570.580.58 query11 0.240.130.14 query12 0.260.130.14 query13 0.640.630.62 query14 1.011.031.02 query15 0.920.860.86 query16 0.400.390.38 query17 1.041.071.03 query18 0.230.210.22 query19 1.981.821.81 query20 0.020.010.02 query21 15.39 0.280.23 query22 4.990.100.10 query23 15.43 0.380.23 query24 2.870.520.32 query25 0.090.090.09 query26 0.180.170.18 query27 0.090.090.09 query28 3.691.251.08 query29 12.64 4.013.36 query30 0.310.120.10 query31 2.810.620.45 query32 3.230.610.51 query33 3.023.073.16 query34 16.42 5.184.47 query35 4.584.524.55 query36 0.640.510.51 query37 0.230.090.09 query38 0.190.050.06 query39 0.060.040.05 query40 0.210.180.17 query41 0.110.060.05 query42 0.070.050.05 query43 0.050.050.05 Total cold run time: 98.86 s Total hot run time: 28.32 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
doris-robot commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3579293091 TPC-DS: Total hot run time: 183747 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit d3845d1c83423ecb203afaae91b10c0f950d78f8, data reload: false query1 1066399 397 397 query2 6561158616181586 query3 6759225 221 221 query4 26004 22915 22378 22378 query5 4360617 460 460 query6 337 236 211 211 query7 4653500 297 297 query8 292 253 246 246 query9 8725255825982558 query10 516 374 299 299 query11 15199 14792 14580 14580 query12 174 114 109 109 query13 1693577 454 454 query14 10476 905290269026 query15 226 204 201 201 query16 7387707 556 556 query17 1267811 633 633 query18 1989424 320 320 query19 202 202 169 169 query20 131 125 124 124 query21 218 133 112 112 query22 4032412138823882 query23 33213 32212 31855 31855 query24 8507240824282408 query25 588 517 455 455 query26 1234275 160 160 query27 2741498 352 352 query28 4366213321062106 query29 787 644 477 477 query30 313 240 219 219 query31 843 688 629 629 query32 84 71 76 71 query33 589 371 330 330 query34 790 881 536 536 query35 795 843 743 743 query36 914 986 853 853 query37 124 113 89 89 query38 3347335733053305 query39 1465142814171417 query40 222 131 118 118 query41 64 67 63 63 query42 125 114 111 111 query43 462 467 434 434 query44 1237746 764 746 query45 209 196 185 185 query46 884 1007639 639 query47 1683173916571657 query48 391 415 320 320 query49 790 501 423 423 query50 641 688 411 411 query51 3883391139323911 query52 112 112 109 109 query53 242 261 191 191 query54 323 291 272 272 query55 91 84 88 84 query56 325 312 320 312 query57 1154117410931093 query58 281 279 273 273 query59 2402253424322432 query60 357 359 328 328 query61 161 156 157 156 query62 805 717 676 676 query63 225 193 195 193 query64 46471314997 997 query65 4090401039903990 query66 1206456 355 355 query67 15422 14979 14842 14842 query68 8351945 627 627 query69 526 360 311 311 query70 1209123512721235 query71 459 341 326 326 query72 5927492848804880 query73 646 588 367 367 query74 8703849683368336 query75 3642329627852785 query76 34821126733 733 query77 756 420 316 316 query78 9520967788398839 query79 2575831 603 603 query80 681 560 492 492 query81 511 272 233 233 query82 469 162 136 136 query83 284 258 262 258 query84 265 119 100 100 query85 944 482 430 430 query86 387 299 299 299 query87 3434346434343434 query88 3885221322192213 query89 381 334 290 290 query90 1919223 208 208 query91 168 167 139 139 query92 81 68 67 67 query93 2248987 649 649 query94 728 460 327 327 query95 487 391 387 387 query96 487 585 285 285 query97 2954296228602860 query98 242 204 218 204 query99 1346137412331233 Total cold run time: 273098 ms Total hot run time: 183747 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
doris-robot commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3579251646 TPC-H: Total hot run time: 34486 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit d3845d1c83423ecb203afaae91b10c0f950d78f8, data reload: false -- Round 1 -- q1 17599 508049134913 q2 2040310 214 214 q3 10247 1294730 730 q4 10240 950 367 367 q5 7530253223082308 q6 182 166 135 135 q7 951 782 641 641 q8 9345138711231123 q9 7206525153525251 q10 6831226118521852 q11 494 306 284 284 q12 327 373 214 214 q13 17755 369630783078 q14 232 241 218 218 q15 569 528 516 516 q16 10091026968 968 q17 587 944 382 382 q18 7362722872137213 q19 993 948 562 562 q20 343 338 233 233 q21 3940258623342334 q22 10371022950 950 Total cold run time: 106819 ms Total hot run time: 34486 ms - Round 2, with runtime_filter_mode=off - q1 4917490949574909 q2 339 388 313 313 q3 2212267723312331 q4 1354178213081308 q5 4211445045484450 q6 216 178 133 133 q7 2145201118241824 q8 2707266625942594 q9 7733752375277523 q10 3072335928072807 q11 594 528 529 528 q12 689 802 645 645 q13 3585400232553255 q14 286 314 273 273 q15 565 501 513 501 q16 1088116610611061 q17 1198154113741374 q18 7921784275697569 q19 781 848 1030848 q20 1988195318201820 q21 4744428243714282 q22 1103108910011001 Total cold run time: 53448 ms Total hot run time: 51349 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
doris-robot commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3579041858 # Cloud UT Coverage Report Increment line coverage ` ` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/d3845d1c83423ecb203afaae91b10c0f950d78f8_d3845d1c83423ecb203afaae91b10c0f950d78f8_cloud/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/d3845d1c83423ecb203afaae91b10c0f950d78f8_d3845d1c83423ecb203afaae91b10c0f950d78f8_cloud/report/index.html) | Category | Coverage | |---|| | Function Coverage | 80.86% (1707/2111) | | Line Coverage | 66.85% (29887/44708) | | Region Coverage | 67.35% (14894/22114) | | Branch Coverage | 57.57% (7926/13768) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
eldenmoon commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3578882695 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
hello-stephen commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3577142294 # FE Regression Coverage Report Increment line coverage ` 28.57% (4/14)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/57702_839366f500bab532082fcf3f97cf445bdb1e85b2_merge_fe/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/57702_839366f500bab532082fcf3f97cf445bdb1e85b2_merge_fe/report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
doris-robot commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3576449174 ClickBench: Total hot run time: 27.68 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 839366f500bab532082fcf3f97cf445bdb1e85b2, data reload: false query1 0.060.050.06 query2 0.090.040.05 query3 0.260.090.08 query4 1.610.110.10 query5 0.270.260.25 query6 1.170.660.64 query7 0.030.030.03 query8 0.060.040.04 query9 0.600.530.51 query10 0.580.580.58 query11 0.150.110.12 query12 0.150.120.12 query13 0.620.610.61 query14 1.011.010.99 query15 0.850.820.86 query16 0.420.380.40 query17 1.011.061.00 query18 0.210.190.20 query19 1.921.801.83 query20 0.020.010.01 query21 15.43 0.180.14 query22 5.140.070.05 query23 15.69 0.260.11 query24 2.620.850.43 query25 0.090.060.06 query26 0.140.130.13 query27 0.070.050.06 query28 4.091.140.95 query29 12.66 3.943.27 query30 0.270.140.11 query31 2.820.590.38 query32 3.220.540.48 query33 3.033.053.03 query34 15.80 5.144.53 query35 4.574.694.98 query36 0.690.550.56 query37 0.100.070.06 query38 0.060.040.04 query39 0.030.030.03 query40 0.170.160.15 query41 0.080.030.03 query42 0.040.030.03 query43 0.040.040.03 Total cold run time: 97.94 s Total hot run time: 27.68 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
doris-robot commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3576419212 TPC-DS: Total hot run time: 183829 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 839366f500bab532082fcf3f97cf445bdb1e85b2, data reload: false query1 1063446 401 401 query2 6574162316121612 query3 6750225 223 223 query4 25927 22693 22262 22262 query5 4425630 475 475 query6 345 234 225 225 query7 4659501 296 296 query8 310 270 242 242 query9 8718258325882583 query10 520 383 320 320 query11 15137 14859 14534 14534 query12 174 118 116 116 query13 1693577 452 452 query14 10133 900789218921 query15 221 200 191 191 query16 7526704 527 527 query17 1263782 627 627 query18 2023433 343 343 query19 218 209 187 187 query20 135 131 126 126 query21 218 137 122 122 query22 3948397238313831 query23 33109 32127 31925 31925 query24 8465241424802414 query25 623 543 476 476 query26 1249282 167 167 query27 2727502 358 358 query28 4394215321212121 query29 852 661 521 521 query30 306 241 212 212 query31 850 735 640 640 query32 87 75 78 75 query33 615 383 377 377 query34 785 868 547 547 query35 824 838 757 757 query36 928 974 880 880 query37 121 105 94 94 query38 3339337732653265 query39 1490142114141414 query40 225 132 117 117 query41 68 63 64 63 query42 126 113 112 112 query43 473 475 445 445 query44 1219753 737 737 query45 202 187 188 187 query46 896 1009635 635 query47 1687171216471647 query48 389 425 318 318 query49 765 511 430 430 query50 658 684 412 412 query51 3957393939123912 query52 107 113 107 107 query53 247 269 195 195 query54 312 303 279 279 query55 92 85 90 85 query56 338 323 299 299 query57 1180117510861086 query58 286 275 272 272 query59 2351252024052405 query60 364 349 334 334 query61 157 161 164 161 query62 821 724 670 670 query63 241 197 191 191 query64 45121210886 886 query65 4047400639563956 query66 1161433 338 338 query67 15588 14991 14817 14817 query68 8365887 615 615 query69 519 348 306 306 query70 1315124111821182 query71 481 323 312 312 query72 5613493048484848 query73 712 574 363 363 query74 8842880684798479 query75 3997335428192819 query76 37941126712 712 query77 833 407 314 314 query78 9484982188968896 query79 1696852 610 610 query80 641 590 494 494 query81 496 277 240 240 query82 418 165 129 129 query83 270 270 252 252 query84 259 119 94 94 query85 916 492 459 459 query86 338 309 286 286 query87 3575346534083408 query88 2838228822942288 query89 384 319 297 297 query90 1933233 223 223 query91 175 174 144 144 query92 78 71 65 65 query93 1072972 651 651 query94 733 448 348 348 query95 498 406 410 406 query96 498 578 285 285 query97 2976300528702870 query98 233 217 210 210 query99 1431137212691269 Total cold run time: 270626 ms Total hot run time: 183829 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
doris-robot commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3576368094 TPC-H: Total hot run time: 34776 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 839366f500bab532082fcf3f97cf445bdb1e85b2, data reload: false -- Round 1 -- q1 17640 509849574957 q2 2015328 221 221 q3 10238 1337729 729 q4 10227 922 368 368 q5 7534246023512351 q6 191 173 142 142 q7 956 790 642 642 q8 9366144511591159 q9 7382532254005322 q10 6924227418301830 q11 503 295 283 283 q12 381 376 235 235 q13 17856 377330893089 q14 241 233 222 222 q15 612 528 514 514 q16 10421066982 982 q17 605 862 398 398 q18 7638725172227222 q19 1093988 559 559 q20 373 367 243 243 q21 4168337023732373 q22 10531010935 935 Total cold run time: 108038 ms Total hot run time: 34776 ms - Round 2, with runtime_filter_mode=off - q1 5014499049724972 q2 334 396 299 299 q3 2251272523332333 q4 1367182013581358 q5 4385462746564627 q6 223 179 134 134 q7 2144200818571857 q8 2676264725662566 q9 7504766373007300 q10 3092337531323132 q11 609 518 495 495 q12 665 773 621 621 q13 3664403833163316 q14 312 301 281 281 q15 594 544 549 544 q16 1143114411371137 q17 1201157314761476 q18 7972769176747674 q19 799 837 887 837 q20 2155223420482048 q21 5197468643054305 q22 1073108110191019 Total cold run time: 54374 ms Total hot run time: 52331 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
doris-robot commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3575950098 # Cloud UT Coverage Report Increment line coverage ` ` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/839366f500bab532082fcf3f97cf445bdb1e85b2_839366f500bab532082fcf3f97cf445bdb1e85b2_cloud/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/839366f500bab532082fcf3f97cf445bdb1e85b2_839366f500bab532082fcf3f97cf445bdb1e85b2_cloud/report/index.html) | Category | Coverage | |---|| | Function Coverage | 80.86% (1707/2111) | | Line Coverage | 66.83% (29880/44708) | | Region Coverage | 67.37% (14898/22114) | | Branch Coverage | 57.52% (7920/13768) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
eldenmoon commented on code in PR #57702:
URL: https://github.com/apache/doris/pull/57702#discussion_r2560101444
##
gensrc/proto/segment_v2.proto:
##
@@ -246,6 +256,17 @@ message SegmentFooterPB {
// Primary key index meta
optional PrimaryKeyIndexMetaPB primary_key_index_meta = 10;
+
+// Externalized ColumnMetaPB layout (optional; presence indicates v2 meta
layout available)
+// Start offset of the contiguous per-column ColumnMetaPB region (absolute
file offset).
+optional uint64 col_meta_region_start = 11;
+// Per-column external meta entries, in col_id order. Each entry contains
the top-level
+// unique id and the serialized length (in bytes) of its ColumnMetaPB in
the external
+// Column Meta Region.
+repeated ColumnMetaEntryPB column_meta_entries = 12;
+// Number of top-level columns in this segment for external meta
addressing (col_id in [0, N)).
+// Must equal column_meta_entries.size().
+optional uint32 num_columns = 13;
Review Comment:
done
##
be/src/olap/rowset/segment_v2/column_meta_accessor.cpp:
##
@@ -0,0 +1,222 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
eldenmoon commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3575799480 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
eldenmoon commented on code in PR #57702:
URL: https://github.com/apache/doris/pull/57702#discussion_r2560102341
##
be/src/olap/rowset/segment_v2/column_meta_accessor.h:
##
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include
+#include
+#include
+
+#include "common/status.h"
+#include "gen_cpp/segment_v2.pb.h"
+#include "io/fs/file_reader.h"
+
+namespace doris::segment_v2 {
+
+// Centralized accessor for column metadata layout of a Segment.
+// It selects V2 (inline footer.columns) or V3 (external Column Meta Region)
+// implementation internally and exposes a unified API to callers.
+class ColumnMetaAccessor {
+public:
+class Impl;
+ColumnMetaAccessor() = default;
+~ColumnMetaAccessor();
+
+// Initialize accessor and choose V2 / V3 implementation based on footer.
+Status init(const SegmentFooterPB& footer, io::FileReaderSPtr file_reader);
+
+// Build uid -> col_id mapping.
+Status build_uid_to_colid_map(std::unordered_map*
uid_to_colid) const;
+
+// Get ColumnMetaPB for the given column_id.
+Status get_column_meta(uint32_t column_id, ColumnMetaPB* out) const;
Review Comment:
done
##
be/src/olap/rowset/segment_v2/column_meta_accessor.cpp:
##
@@ -0,0 +1,222 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/rowset/segment_v2/column_meta_accessor.h"
+
+#include
+#include
+
+#include "io/io_common.h"
+#include "olap/rowset/segment_v2/external_col_meta_util.h"
+#include "olap/rowset/segment_v2/segment.h" //
kSegmentFooterVersionV3_ExtColMeta
+
+namespace doris::segment_v2 {
+
+// Abstract base implementation shared by both V2 and V3 layouts.
+class ColumnMetaAccessor::Impl {
+public:
+virtual ~Impl() = default;
+virtual Status build_uid_to_colid_map(
+std::unordered_map* uid_to_colid) const = 0;
+virtual Status get_column_meta(uint32_t column_id, ColumnMetaPB* out)
const = 0;
+virtual Status traverse_metas(
+const std::function& visitor) const = 0;
+};
+
+// V2: inline layout using only footer.columns().
+class ColumnMetaAccessorV2 : public ColumnMetaAccessor::Impl {
+public:
+ColumnMetaAccessorV2(const SegmentFooterPB* footer, io::FileReaderSPtr
file_reader)
+: _footer(footer), _file_reader(std::move(file_reader)) {}
+
+Status build_uid_to_colid_map(
+std::unordered_map* uid_to_colid) const override {
+uid_to_colid->clear();
+uint32_t ordinal = 0;
+for (const auto& column_meta : _footer->columns()) {
+// 跳过 variant 子列（unique_id == -1）
+if (column_meta.unique_id() == -1) {
+ordinal++;
+continue;
+}
+uid_to_colid->try_emplace(column_meta.unique_id(), ordinal++);
+}
+return Status::OK();
+}
+
+Status get_column_meta(uint32_t column_id, ColumnMetaPB* out) const
override {
+if (_footer->columns_size() == 0 ||
+column_id >= static_cast(_footer->columns_size())) {
+return Status::Corruption(
+"no inline column meta available for column_id={}
(columns_size={})", column_id,
+_footer->columns_size());
+}
+*out = _footer->columns(static_cast(column_id));
+return Status::OK();
+}
+
+Status traverse_metas(const std::function&
visitor) const override {
+if (_foot
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
eldenmoon commented on code in PR #57702:
URL: https://github.com/apache/doris/pull/57702#discussion_r2560100457
##
be/src/olap/rowset/segment_v2/segment.cpp:
##
@@ -681,16 +635,12 @@ Status
Segment::_create_column_meta_once(OlapReaderStatistics* stats) {
}
Status Segment::_create_column_meta(const SegmentFooterPB& footer) {
-// unique_id -> idx in footer.columns()
-uint32_t ordinal = 0;
-for (const auto& column_meta : footer.columns()) {
-// no need to create column reader for variant's subcolumn
-if (column_meta.unique_id() == -1) {
-ordinal++;
-continue;
-}
-_column_uid_to_footer_ordinal.try_emplace(column_meta.unique_id(),
ordinal++);
-}
+// Build uid -> col_id mapping used to locate per-column meta.
+// The concrete layout (inline vs external CMO) is handled by
ColumnMetaAccessor.
+_column_uid_to_footer_ordinal.clear();
+ColumnMetaAccessor accessor;
+RETURN_IF_ERROR(accessor.init(footer, _file_reader));
+
RETURN_IF_ERROR(accessor.build_uid_to_colid_map(&_column_uid_to_footer_ordinal));
Review Comment:
done
##
be/src/olap/rowset/segment_v2/external_col_meta_util.cpp:
##
@@ -0,0 +1,204 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/rowset/segment_v2/external_col_meta_util.h"
+
+#include
+#include
+#include
+#include
+
+#include "io/io_common.h"
+#include "olap/rowset/segment_v2/variant/variant_ext_meta_writer.h"
+#include "util/coding.h"
+#include "util/faststring.h"
+
+namespace doris::segment_v2 {
+
+bool ExternalColMetaUtil::parse_external_meta_pointers(
+const SegmentFooterPB& footer,
ExternalColMetaUtil::ExternalMetaPointers* out) {
+out->region_start = footer.col_meta_region_start();
+out->num_columns = footer.num_columns();
+// Validate pointers and basic consistency to ensure external meta is
actually present.
+if (out->region_start == 0) {
+return false;
+}
+if (out->num_columns == 0) {
+return false;
+}
+if (footer.column_meta_entries_size() != out->num_columns) {
+return false;
+}
+// Compute region_end = region_start + sum(length_i) with overflow guards.
+uint64_t region_size = 0;
+for (int i = 0; i < footer.column_meta_entries_size(); ++i) {
+const auto& entry = footer.column_meta_entries(i);
+const uint64_t len = static_cast(entry.length());
+if (len == 0) {
+return false;
+}
+if (region_size + len < region_size) {
+return false;
+} // overflow guard
+region_size += len;
+}
+out->region_end = out->region_start + region_size;
+if (out->region_end <= out->region_start) {
+return false;
+}
+return true;
+}
+
+bool ExternalColMetaUtil::parse_uid_to_colid_map(
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
yiguolei commented on code in PR #57702:
URL: https://github.com/apache/doris/pull/57702#discussion_r2559313109
##
be/src/olap/rowset/segment_v2/column_meta_accessor.cpp:
##
@@ -0,0 +1,222 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/rowset/segment_v2/column_meta_accessor.h"
+
+#include
+#include
+
+#include "io/io_common.h"
+#include "olap/rowset/segment_v2/external_col_meta_util.h"
+#include "olap/rowset/segment_v2/segment.h" //
kSegmentFooterVersionV3_ExtColMeta
+
+namespace doris::segment_v2 {
+
+// Abstract base implementation shared by both V2 and V3 layouts.
+class ColumnMetaAccessor::Impl {
+public:
+virtual ~Impl() = default;
+virtual Status build_uid_to_colid_map(
+std::unordered_map* uid_to_colid) const = 0;
+virtual Status get_column_meta(uint32_t column_id, ColumnMetaPB* out)
const = 0;
+virtual Status traverse_metas(
+const std::function& visitor) const = 0;
+};
+
+// V2: inline layout using only footer.columns().
+class ColumnMetaAccessorV2 : public ColumnMetaAccessor::Impl {
+public:
+ColumnMetaAccessorV2(const SegmentFooterPB* footer, io::FileReaderSPtr
file_reader)
+: _footer(footer), _file_reader(std::move(file_reader)) {}
+
+Status build_uid_to_colid_map(
+std::unordered_map* uid_to_colid) const override {
+uid_to_colid->clear();
+uint32_t ordinal = 0;
+for (const auto& column_meta : _footer->columns()) {
+// 跳过 variant 子列（unique_id == -1）
+if (column_meta.unique_id() == -1) {
+ordinal++;
+continue;
+}
+uid_to_colid->try_emplace(column_meta.unique_id(), ordinal++);
+}
+return Status::OK();
+}
+
+Status get_column_meta(uint32_t column_id, ColumnMetaPB* out) const
override {
+if (_footer->columns_size() == 0 ||
+column_id >= static_cast(_footer->columns_size())) {
+return Status::Corruption(
+"no inline column meta available for column_id={}
(columns_size={})", column_id,
+_footer->columns_size());
+}
+*out = _footer->columns(static_cast(column_id));
+return Status::OK();
+}
+
+Status traverse_metas(const std::function&
visitor) const override {
+if (_footer->columns_size() == 0) {
+return Status::Corruption("no column meta found in footer (inline
V2)");
+}
+for (const auto& column : _footer->columns()) {
+visitor(column);
+}
+return Status::OK();
+}
+
+private:
+const SegmentFooterPB* _footer = nullptr; // non-owning
+io::FileReaderSPtr _file_reader;
+};
+
+// V3: use external Column Meta Region + column_meta_entries layout.
+class ColumnMetaAccessorV3 : public ColumnMetaAccessor::Impl {
+public:
+ColumnMetaAccessorV3(const SegmentFooterPB* footer, io::FileReaderSPtr
file_reader,
+ const ExternalColMetaUtil::ExternalMetaPointers& ptrs)
+: _footer(footer), _file_reader(std::move(file_reader)),
_ptrs(ptrs) {}
+
+Status build_uid_to_colid_map(
+std::unordered_map* uid_to_colid) const override {
+uid_to_colid->clear();
+
+bool loaded = false;
+// Prefer external uid->col_id map when available.
+loaded = ExternalColMetaUtil::parse_uid_to_colid_map(*_footer, _ptrs,
uid_to_colid);
+
+// Compatibility: if external map is broken, fall back to inline
columns.
+if (!loaded && _footer->columns_size() > 0) {
+uint32_t ordinal = 0;
+for (const auto& column_meta : _footer->columns()) {
+if (column_meta.unique_id() == -1) {
+ordinal++;
+continue;
+}
+uid_to_colid->try_emplace(column_meta.unique_id(), ordinal++);
+}
+loaded = true;
+}
+
+if (!loaded) {
+return Status::Corruption(
+"segment external meta mapping missing or corrupted and no
inline columns; "
+"uid->col_id cannot be resolved. path={}",
+
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
yiguolei commented on code in PR #57702:
URL: https://github.com/apache/doris/pull/57702#discussion_r2558773209
##
be/src/olap/rowset/segment_v2/column_meta_accessor.h:
##
@@ -0,0 +1,55 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include
+#include
+#include
+
+#include "common/status.h"
+#include "gen_cpp/segment_v2.pb.h"
+#include "io/fs/file_reader.h"
+
+namespace doris::segment_v2 {
+
+// Centralized accessor for column metadata layout of a Segment.
+// It selects V2 (inline footer.columns) or V3 (external Column Meta Region)
+// implementation internally and exposes a unified API to callers.
+class ColumnMetaAccessor {
+public:
+class Impl;
+ColumnMetaAccessor() = default;
+~ColumnMetaAccessor();
+
+// Initialize accessor and choose V2 / V3 implementation based on footer.
+Status init(const SegmentFooterPB& footer, io::FileReaderSPtr file_reader);
+
+// Build uid -> col_id mapping.
+Status build_uid_to_colid_map(std::unordered_map*
uid_to_colid) const;
+
+// Get ColumnMetaPB for the given column_id.
+Status get_column_meta(uint32_t column_id, ColumnMetaPB* out) const;
Review Comment:
Status get_column_meta_by_uniqueid(uint32_t column_unique_id, ColumnMetaPB*
out) const;
Status get_column_meta_by_id(uint32_t column_ordinal, ColumnMetaPB* out)
const;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
eldenmoon commented on code in PR #57702:
URL: https://github.com/apache/doris/pull/57702#discussion_r2558696382
##
be/src/olap/rowset/segment_v2/segment.cpp:
##
@@ -681,16 +635,12 @@ Status
Segment::_create_column_meta_once(OlapReaderStatistics* stats) {
}
Status Segment::_create_column_meta(const SegmentFooterPB& footer) {
-// unique_id -> idx in footer.columns()
-uint32_t ordinal = 0;
-for (const auto& column_meta : footer.columns()) {
-// no need to create column reader for variant's subcolumn
-if (column_meta.unique_id() == -1) {
-ordinal++;
-continue;
-}
-_column_uid_to_footer_ordinal.try_emplace(column_meta.unique_id(),
ordinal++);
-}
+// Build uid -> col_id mapping used to locate per-column meta.
+// The concrete layout (inline vs external CMO) is handled by
ColumnMetaAccessor.
+_column_uid_to_footer_ordinal.clear();
+ColumnMetaAccessor accessor;
+RETURN_IF_ERROR(accessor.init(footer, _file_reader));
+
RETURN_IF_ERROR(accessor.build_uid_to_colid_map(&_column_uid_to_footer_ordinal));
Review Comment:
统一成_column_uid_to_column_id
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
eldenmoon commented on code in PR #57702:
URL: https://github.com/apache/doris/pull/57702#discussion_r2558695791
##
gensrc/proto/segment_v2.proto:
##
@@ -246,6 +256,17 @@ message SegmentFooterPB {
// Primary key index meta
optional PrimaryKeyIndexMetaPB primary_key_index_meta = 10;
+
+// Externalized ColumnMetaPB layout (optional; presence indicates v2 meta
layout available)
+// Start offset of the contiguous per-column ColumnMetaPB region (absolute
file offset).
+optional uint64 col_meta_region_start = 11;
+// Per-column external meta entries, in col_id order. Each entry contains
the top-level
+// unique id and the serialized length (in bytes) of its ColumnMetaPB in
the external
+// Column Meta Region.
+repeated ColumnMetaEntryPB column_meta_entries = 12;
+// Number of top-level columns in this segment for external meta
addressing (col_id in [0, N)).
+// Must equal column_meta_entries.size().
+optional uint32 num_columns = 13;
Review Comment:
本来是用来校验的， 实际用不上，我删了
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
yiguolei commented on code in PR #57702:
URL: https://github.com/apache/doris/pull/57702#discussion_r2558498513
##
be/src/olap/rowset/segment_v2/segment.cpp:
##
@@ -681,16 +635,12 @@ Status
Segment::_create_column_meta_once(OlapReaderStatistics* stats) {
}
Status Segment::_create_column_meta(const SegmentFooterPB& footer) {
-// unique_id -> idx in footer.columns()
-uint32_t ordinal = 0;
-for (const auto& column_meta : footer.columns()) {
-// no need to create column reader for variant's subcolumn
-if (column_meta.unique_id() == -1) {
-ordinal++;
-continue;
-}
-_column_uid_to_footer_ordinal.try_emplace(column_meta.unique_id(),
ordinal++);
-}
+// Build uid -> col_id mapping used to locate per-column meta.
+// The concrete layout (inline vs external CMO) is handled by
ColumnMetaAccessor.
+_column_uid_to_footer_ordinal.clear();
+ColumnMetaAccessor accessor;
+RETURN_IF_ERROR(accessor.init(footer, _file_reader));
+
RETURN_IF_ERROR(accessor.build_uid_to_colid_map(&_column_uid_to_footer_ordinal));
Review Comment:
我们到底是叫colid还是ordinal 得统一了_column_uid_to_footer_ordinal
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
yiguolei commented on code in PR #57702:
URL: https://github.com/apache/doris/pull/57702#discussion_r2558478975
##
be/src/olap/rowset/segment_v2/external_col_meta_util.cpp:
##
@@ -0,0 +1,204 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/rowset/segment_v2/external_col_meta_util.h"
+
+#include
+#include
+#include
+#include
+
+#include "io/io_common.h"
+#include "olap/rowset/segment_v2/variant/variant_ext_meta_writer.h"
+#include "util/coding.h"
+#include "util/faststring.h"
+
+namespace doris::segment_v2 {
+
+bool ExternalColMetaUtil::parse_external_meta_pointers(
+const SegmentFooterPB& footer,
ExternalColMetaUtil::ExternalMetaPointers* out) {
+out->region_start = footer.col_meta_region_start();
+out->num_columns = footer.num_columns();
+// Validate pointers and basic consistency to ensure external meta is
actually present.
+if (out->region_start == 0) {
+return false;
+}
+if (out->num_columns == 0) {
+return false;
+}
+if (footer.column_meta_entries_size() != out->num_columns) {
+return false;
+}
+// Compute region_end = region_start + sum(length_i) with overflow guards.
+uint64_t region_size = 0;
+for (int i = 0; i < footer.column_meta_entries_size(); ++i) {
+const auto& entry = footer.column_meta_entries(i);
+const uint64_t len = static_cast(entry.length());
+if (len == 0) {
+return false;
+}
+if (region_size + len < region_size) {
+return false;
+} // overflow guard
+region_size += len;
+}
+out->region_end = out->region_start + region_size;
+if (out->region_end <= out->region_start) {
+return false;
+}
+return true;
+}
+
+bool ExternalColMetaUtil::parse_uid_to_colid_map(
Review Comment:
这里也不要返回true or false
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
yiguolei commented on code in PR #57702:
URL: https://github.com/apache/doris/pull/57702#discussion_r2558464710
##
gensrc/proto/segment_v2.proto:
##
@@ -246,6 +256,17 @@ message SegmentFooterPB {
// Primary key index meta
optional PrimaryKeyIndexMetaPB primary_key_index_meta = 10;
+
+// Externalized ColumnMetaPB layout (optional; presence indicates v2 meta
layout available)
+// Start offset of the contiguous per-column ColumnMetaPB region (absolute
file offset).
+optional uint64 col_meta_region_start = 11;
+// Per-column external meta entries, in col_id order. Each entry contains
the top-level
+// unique id and the serialized length (in bytes) of its ColumnMetaPB in
the external
+// Column Meta Region.
+repeated ColumnMetaEntryPB column_meta_entries = 12;
+// Number of top-level columns in this segment for external meta
addressing (col_id in [0, N)).
+// Must equal column_meta_entries.size().
+optional uint32 num_columns = 13;
Review Comment:
看注释num columns 一定跟column_meta_entries.SIZE 相等，我们保存他的意义是什么？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
yiguolei commented on code in PR #57702: URL: https://github.com/apache/doris/pull/57702#discussion_r2558458675 ## be/src/olap/rowset/segment_v2/column_meta_accessor.cpp: ## @@ -0,0 +1,222 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file Review Comment: 我们需要为v2和v3 建立单测，否则后续的bugfix 和改进，都不会有单测去保证改动合理并且正确。 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
yiguolei commented on code in PR #57702:
URL: https://github.com/apache/doris/pull/57702#discussion_r2558449585
##
be/src/olap/rowset/segment_v2/external_col_meta_util.cpp:
##
@@ -0,0 +1,204 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/rowset/segment_v2/external_col_meta_util.h"
+
+#include
+#include
+#include
+#include
+
+#include "io/io_common.h"
+#include "olap/rowset/segment_v2/variant/variant_ext_meta_writer.h"
+#include "util/coding.h"
+#include "util/faststring.h"
+
+namespace doris::segment_v2 {
+
+bool ExternalColMetaUtil::parse_external_meta_pointers(
+const SegmentFooterPB& footer,
ExternalColMetaUtil::ExternalMetaPointers* out) {
+out->region_start = footer.col_meta_region_start();
+out->num_columns = footer.num_columns();
+// Validate pointers and basic consistency to ensure external meta is
actually present.
+if (out->region_start == 0) {
+return false;
Review Comment:
不要返回true or false，返回error status，因为error status 里我们可以详细记录错误原因。 返回true or
false，这部分信息就丢了，我们如果真的有bug，定位起来就很难了。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
hello-stephen commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3572409032 # FE Regression Coverage Report Increment line coverage ` 28.57% (4/14)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/57702_946eb74ee832f0fe4402a686dc53037cea686521_merge_fe/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/57702_946eb74ee832f0fe4402a686dc53037cea686521_merge_fe/report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
doris-robot commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3571883169 ClickBench: Total hot run time: 28.54 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 946eb74ee832f0fe4402a686dc53037cea686521, data reload: false query1 0.050.050.05 query2 0.120.060.06 query3 0.310.080.08 query4 1.600.080.08 query5 0.260.240.25 query6 1.170.650.67 query7 0.030.020.03 query8 0.080.060.06 query9 0.650.550.53 query10 0.590.590.58 query11 0.250.130.13 query12 0.260.140.14 query13 0.660.630.62 query14 1.021.021.02 query15 0.950.860.86 query16 0.380.380.38 query17 1.061.051.08 query18 0.230.230.22 query19 1.971.871.88 query20 0.010.020.02 query21 15.40 0.290.24 query22 4.960.100.10 query23 15.39 0.380.23 query24 2.940.490.30 query25 0.100.090.10 query26 0.180.170.18 query27 0.100.090.09 query28 3.641.261.07 query29 12.55 4.083.36 query30 0.340.130.12 query31 2.830.630.46 query32 3.250.600.50 query33 3.223.063.14 query34 16.27 5.184.50 query35 4.524.554.63 query36 0.640.520.52 query37 0.220.090.08 query38 0.200.060.06 query39 0.060.050.05 query40 0.200.180.18 query41 0.130.060.06 query42 0.070.050.05 query43 0.060.040.05 Total cold run time: 98.92 s Total hot run time: 28.54 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
doris-robot commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3571859310 TPC-DS: Total hot run time: 184990 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 946eb74ee832f0fe4402a686dc53037cea686521, data reload: false query1 1028433 420 420 query2 6549160415961596 query3 6759222 224 222 query4 25820 22786 22853 22786 query5 4423629 470 470 query6 316 235 247 235 query7 4653494 302 302 query8 290 249 252 249 query9 8698258026012580 query10 518 352 305 305 query11 14957 14816 14602 14602 query12 180 118 116 116 query13 1688592 472 472 query14 10227 883688038803 query15 220 202 187 187 query16 7626700 543 543 query17 1234740 608 608 query18 2015424 321 321 query19 210 192 170 170 query20 128 124 118 118 query21 214 135 115 115 query22 3948395238423842 query23 33126 32361 32257 32257 query24 8466241223592359 query25 583 537 482 482 query26 1247272 170 170 query27 2749526 361 361 query28 4334224622082208 query29 833 644 517 517 query30 310 248 208 208 query31 834 710 646 646 query32 82 92 86 86 query33 621 394 349 349 query34 796 860 536 536 query35 802 825 737 737 query36 930 933 851 851 query37 131 113 90 90 query38 3282334432743274 query39 1493145514261426 query40 226 139 123 123 query41 71 68 69 68 query42 132 118 115 115 query43 459 485 445 445 query44 1295789 781 781 query45 201 195 194 194 query46 900 1017666 666 query47 1712176416491649 query48 396 461 330 330 query49 792 514 433 433 query50 682 691 410 410 query51 3902396438743874 query52 113 117 103 103 query53 253 281 203 203 query54 328 321 301 301 query55 89 86 88 86 query56 335 354 333 333 query57 1163117711301130 query58 296 285 275 275 query59 2453257624172417 query60 360 359 368 359 query61 197 224 158 158 query62 791 692 678 678 query63 222 183 190 183 query64 44941188886 886 query65 4101398639693969 query66 1124443 369 369 query67 15341 15080 14819 14819 query68 8416923 638 638 query69 525 331 297 297 query70 1281118612041186 query71 479 344 312 312 query72 5633493848474847 query73 700 574 370 370 query74 8562886486648664 query75 3981334527802780 query76 38071116703 703 query77 812 422 309 309 query78 9564980789228922 query79 1982815 606 606 query80 643 593 500 500 query81 503 272 233 233 query82 431 160 131 131 query83 284 260 256 256 query84 267 116 104 104 query85 910 483 442 442 query86 344 292 319 292 query87 3561348534303430 query88 3299226822442244 query89 394 331 291 291 query90 1938230 229 229 query91 174 189 144 144 query92 87 64 61 61 query93 1122999 685 685 query94 728 458 316 316 query95 504 412 396 396 query96 496 557 286 286 query97 2954301028732873 query98 227 208 206 206 query99 1407139112611261 Total cold run time: 270763 ms Total hot run time: 184990 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
doris-robot commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3571811179 TPC-H: Total hot run time: 34924 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 946eb74ee832f0fe4402a686dc53037cea686521, data reload: false -- Round 1 -- q1 17425 512750455045 q2 2052321 211 211 q3 10140 1342739 739 q4 10226 919 366 366 q5 7529246823292329 q6 188 170 135 135 q7 984 796 655 655 q8 9350141311351135 q9 7180540253965396 q10 6883227018381838 q11 502 316 291 291 q12 343 371 223 223 q13 17779 377630863086 q14 240 241 229 229 q15 614 534 535 534 q16 10751047973 973 q17 599 895 416 416 q18 7693741672077207 q19 979 963 565 565 q20 358 345 230 230 q21 3958319823532353 q22 998 988 968 968 Total cold run time: 107095 ms Total hot run time: 34924 ms - Round 2, with runtime_filter_mode=off - q1 4958498650084986 q2 338 405 291 291 q3 2238269322642264 q4 1384178613201320 q5 4282459145444544 q6 230 203 150 150 q7 2113198819561956 q8 2690257626572576 q9 7600743575867435 q10 3205336727942794 q11 614 532 498 498 q12 699 780 647 647 q13 3646398633943394 q14 311 314 276 276 q15 558 530 521 521 q16 1117115111161116 q17 1195164115241524 q18 8048790578517851 q19 824 836 900 836 q20 1948200018421842 q21 4870432143484321 q22 11041043994 994 Total cold run time: 53972 ms Total hot run time: 52136 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
hello-stephen commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3571427214 # Cloud UT Coverage Report Increment line coverage ` ` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/946eb74ee832f0fe4402a686dc53037cea686521_946eb74ee832f0fe4402a686dc53037cea686521_cloud/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/946eb74ee832f0fe4402a686dc53037cea686521_946eb74ee832f0fe4402a686dc53037cea686521_cloud/report/index.html) | Category | Coverage | |---|| | Function Coverage | 80.86% (1707/2111) | | Line Coverage | 66.86% (29892/44708) | | Region Coverage | 67.37% (14899/22114) | | Branch Coverage | 57.60% (7931/13768) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
eldenmoon commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3571206315 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
yiguolei commented on code in PR #57702:
URL: https://github.com/apache/doris/pull/57702#discussion_r2554833314
##
be/src/olap/rowset/segment_v2/column_reader_cache.cpp:
##
@@ -155,31 +170,58 @@ Status ColumnReaderCache::get_path_column_reader(uint32_t
col_uid,
->get_subcolumn_meta_by_path(relative_path)
: nullptr;
}
-if (node != nullptr) {
-// lazy create column reader from footer
-DCHECK_GE(node->data.footer_ordinal, 0);
-std::shared_ptr footer_pb_shared;
-{
-std::lock_guard lock(_cache_mutex);
-// keep the lock until the footer is loaded, since
_get_segment_footer is not thread safe
-RETURN_IF_ERROR(_segment->_get_segment_footer(footer_pb_shared,
stats));
-}
-const auto& col_footer_pb =
footer_pb_shared->columns(node->data.footer_ordinal);
+
Review Comment:
这里直接把get_path_column_reader 这个函数分成2个函数吧，一个 get_path_column_reader_v2
另外一个get_path_column_reader_v3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
yiguolei commented on code in PR #57702:
URL: https://github.com/apache/doris/pull/57702#discussion_r2544267001
##
be/src/olap/rowset/segment_v2/segment.cpp:
##
@@ -681,15 +692,39 @@ Status
Segment::_create_column_meta_once(OlapReaderStatistics* stats) {
}
Status Segment::_create_column_meta(const SegmentFooterPB& footer) {
-// unique_id -> idx in footer.columns()
-uint32_t ordinal = 0;
-for (const auto& column_meta : footer.columns()) {
-// no need to create column reader for variant's subcolumn
-if (column_meta.unique_id() == -1) {
-ordinal++;
-continue;
+// Build uid -> col_id mapping used to locate per-column meta.
+// Priority:
+// 1) uid2colid blob in footer.file_meta_datas (external meta mode, raw
uid array)
+// 2) fallback: scan footer.columns() (old or mixed format)
+// 3) both missing -> treat as no columns resolvable in this segment
(return NOT_FOUND on lookup)
+_column_uid_to_footer_ordinal.clear();
+// Decide path by footer version: V3 and above uses externalized
ColumnMeta region
+const bool use_external_meta = footer.version() >=
kSegmentFooterVersionV3_ExtColMeta;
+// 1) Prefer uid->col_id blob (external meta mode). If corrupted or
missing, fall back.
+ExternalColMetaUtil::ExternalMetaPointers ptrs;
+bool loaded = false;
+if (use_external_meta) {
+loaded = ExternalColMetaUtil::parse_external_meta_pointers(footer,
&ptrs) &&
+ ExternalColMetaUtil::parse_uid_to_colid_map(footer, ptrs,
+
&_column_uid_to_footer_ordinal);
+}
+// 2) Fallback: inline footer columns (old or mixed format)
+if (!loaded && footer.columns_size() > 0) {
+uint32_t ordinal = 0;
+for (const auto& column_meta : footer.columns()) {
+if (column_meta.unique_id() == -1) {
+ordinal++;
+continue;
+}
+_column_uid_to_footer_ordinal.try_emplace(column_meta.unique_id(),
ordinal++);
}
-_column_uid_to_footer_ordinal.try_emplace(column_meta.unique_id(),
ordinal++);
+loaded = true;
+}
+// 3) If still not loaded, warn once to aid troubleshooting (damaged
mapping and empty inline columns).
+if (!loaded) {
+LOG(WARNING) << "segment external meta mapping missing or corrupted
and no inline columns; "
+"uid->col_id cannot be resolved. path="
Review Comment:
此时，为什么不直接报错？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
hello-stephen commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3552362164 # FE Regression Coverage Report Increment line coverage ` 25.00% (3/12)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/57702_099689b8fa5843d33df45eb0d51b13041762f49f_merge_fe/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/57702_099689b8fa5843d33df45eb0d51b13041762f49f_merge_fe/report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
yiguolei commented on code in PR #57702: URL: https://github.com/apache/doris/pull/57702#discussion_r2541142215 ## be/src/olap/rowset/segment_v2/column_reader.cpp: ## @@ -248,12 +250,31 @@ Status ColumnReader::create(const ColumnReaderOptions& opts, const SegmentFooter uint32_t column_id, uint64_t num_rows, Review Comment: 我们之前这个接口有点失误，我们不应该传递footer，应该在外部把variant的subcolumns 这些解析出来，然后构建reader。 这里如果列真的很多，我们parse_external_meta_pointers 这个函数的计算复杂度还是很高的。 如果是在segment 里，我们就可以一把计算好，每个column metapb 在segment 文件里的offset和length，下次查询就只要在map 里找就可以了。 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
hello-stephen commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3548776920 # FE Regression Coverage Report Increment line coverage ` 25.00% (3/12)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/57702_099689b8fa5843d33df45eb0d51b13041762f49f_merge_fe/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/57702_099689b8fa5843d33df45eb0d51b13041762f49f_merge_fe/report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
hello-stephen commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3548227530 # FE UT Coverage Report Increment line coverage ` 16.67% (2/12)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/57702_099689b8fa5843d33df45eb0d51b13041762f49f/fe_increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/57702_099689b8fa5843d33df45eb0d51b13041762f49f/fe_report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
doris-robot commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3548077649 ClickBench: Total hot run time: 27.47 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 099689b8fa5843d33df45eb0d51b13041762f49f, data reload: false query1 0.060.050.05 query2 0.090.050.05 query3 0.260.080.08 query4 1.600.110.12 query5 0.270.260.24 query6 1.180.650.64 query7 0.030.030.02 query8 0.050.040.04 query9 0.570.500.51 query10 0.570.570.56 query11 0.160.110.11 query12 0.150.110.12 query13 0.610.600.60 query14 1.011.000.98 query15 0.840.820.82 query16 0.380.390.39 query17 1.031.021.05 query18 0.210.200.20 query19 1.891.761.86 query20 0.050.010.02 query21 15.43 0.200.13 query22 5.110.070.04 query23 15.67 0.250.09 query24 2.621.330.30 query25 0.060.060.07 query26 0.140.130.13 query27 0.070.050.05 query28 4.351.160.95 query29 12.60 3.843.21 query30 0.280.140.11 query31 2.810.570.39 query32 3.230.550.48 query33 3.052.983.09 query34 15.67 5.224.59 query35 4.564.795.04 query36 0.710.560.53 query37 0.100.070.06 query38 0.060.050.03 query39 0.040.020.02 query40 0.180.160.15 query41 0.090.030.03 query42 0.040.030.03 query43 0.040.030.03 Total cold run time: 97.92 s Total hot run time: 27.47 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
doris-robot commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3548045120 TPC-DS: Total hot run time: 188206 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 099689b8fa5843d33df45eb0d51b13041762f49f, data reload: false query1 1054396 390 390 query2 6591167617351676 query3 6764234 230 230 query4 26542 22913 22771 22771 query5 4438671 520 520 query6 369 262 253 253 query7 4664496 326 326 query8 309 278 271 271 query9 8679295329462946 query10 510 369 341 341 query11 15408 15093 15265 15093 query12 181 127 122 122 query13 1684586 433 433 query14 11016 926391649164 query15 200 181 167 167 query16 7458659 481 481 query17 1224756 643 643 query18 2009416 326 326 query19 211 206 181 181 query20 135 127 123 123 query21 213 129 113 113 query22 3922408439543954 query23 34038 32963 32908 32908 query24 7932242224142414 query25 663 567 493 493 query26 1252286 170 170 query27 2737512 359 359 query28 4259224122282228 query29 843 672 507 507 query30 320 225 197 197 query31 918 803 713 713 query32 92 84 85 84 query33 587 407 351 351 query34 807 841 519 519 query35 838 841 758 758 query36 945 977 918 918 query37 134 121 109 109 query38 3511360034553455 query39 1465140514101405 query40 221 140 129 129 query41 78 59 58 58 query42 134 122 113 113 query43 488 483 448 448 query44 1279818 800 800 query45 184 181 172 172 query46 896 982 662 662 query47 1762174817061706 query48 403 423 321 321 query49 776 505 432 432 query50 669 678 418 418 query51 3957394539293929 query52 115 117 112 112 query53 252 274 202 202 query54 341 326 306 306 query55 95 92 92 92 query56 365 351 355 351 query57 1183119311011101 query58 301 294 292 292 query59 2446267625272527 query60 374 402 354 354 query61 165 163 164 163 query62 783 703 674 674 query63 234 204 198 198 query64 42211178888 888 query65 4012394439433943 query66 1131451 345 345 query67 15143 14976 14590 14590 query68 5769957 626 626 query69 498 337 304 304 query70 1369132212721272 query71 438 353 342 342 query72 5724499050024990 query73 664 570 368 368 query74 9154906389108910 query75 3275325628062806 query76 33541116742 742 query77 517 416 351 351 query78 9461963589098909 query79 2551838 606 606 query80 751 586 539 539 query81 507 257 230 230 query82 229 180 144 144 query83 278 264 256 256 query84 259 109 98 98 query85 869 481 448 448 query86 375 341 304 304 query87 3681364135883588 query88 3724223222362232 query89 389 339 293 293 query90 1947238 232 232 query91 164 172 140 140 query92 92 81 71 71 query93 2132984 683 683 query94 742 411 334 334 query95 434 349 322 322 query96 494 576 284 284 query97 2888296828662866 query98 248 222 216 216 query99 1342138912621262 Total cold run time: 271249 ms Total hot run time: 188206 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
doris-robot commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3547991458 TPC-H: Total hot run time: 34545 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 099689b8fa5843d33df45eb0d51b13041762f49f, data reload: false -- Round 1 -- q1 16467 510949694969 q2 2024315 208 208 q3 98081300736 736 q4 9844964 362 362 q5 7307241523962396 q6 184 170 136 136 q7 935 753 636 636 q8 9047138111991199 q9 6759515252185152 q10 6840223617911791 q11 514 295 280 280 q12 347 383 240 240 q13 17569 362130393039 q14 250 233 214 214 q15 553 523 509 509 q16 981 995 929 929 q17 580 880 368 368 q18 7478708979157089 q19 11711002580 580 q20 395 347 237 237 q21 4299357024202420 q22 1176107010551055 Total cold run time: 104528 ms Total hot run time: 34545 ms - Round 2, with runtime_filter_mode=off - q1 5288519752515197 q2 247 330 230 230 q3 2384294124422442 q4 1458190914271427 q5 4610454544304430 q6 208 165 130 130 q7 2051198417321732 q8 2712253826932538 q9 7298726172967261 q10 3035326827922792 q11 597 507 492 492 q12 689 797 628 628 q13 3262366530573057 q14 263 278 252 252 q15 512 485 480 480 q16 991 1041985 985 q17 1126146012991299 q18 7281719870337033 q19 760 754 850 754 q20 1892195418031803 q21 4774426843314268 q22 10791036984 984 Total cold run time: 52517 ms Total hot run time: 50214 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
doris-robot commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3547961957 # Cloud UT Coverage Report Increment line coverage ` ` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/099689b8fa5843d33df45eb0d51b13041762f49f_099689b8fa5843d33df45eb0d51b13041762f49f_cloud/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/099689b8fa5843d33df45eb0d51b13041762f49f_099689b8fa5843d33df45eb0d51b13041762f49f_cloud/report/index.html) | Category | Coverage | |---|| | Function Coverage | 80.61% (1700/2109) | | Line Coverage | 66.68% (29778/44660) | | Region Coverage | 67.33% (14880/22101) | | Branch Coverage | 57.56% (7914/13750) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
eldenmoon commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3547837648 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
doris-robot commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3547470535 ClickBench: Total hot run time: 28.41 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 22c56acb6fdbb080ac3ef44a829c31195df11696, data reload: false query1 0.050.050.04 query2 0.130.070.07 query3 0.320.070.07 query4 1.610.080.08 query5 0.280.250.24 query6 1.180.650.64 query7 0.030.020.03 query8 0.080.060.06 query9 0.670.550.52 query10 0.590.590.57 query11 0.260.130.13 query12 0.260.130.15 query13 0.650.630.62 query14 1.031.021.02 query15 0.950.870.85 query16 0.380.380.38 query17 1.041.041.03 query18 0.240.220.23 query19 1.921.821.80 query20 0.020.010.02 query21 15.40 0.300.24 query22 4.960.100.10 query23 15.35 0.390.23 query24 2.830.520.34 query25 0.100.100.09 query26 0.190.180.18 query27 0.090.090.09 query28 3.591.261.09 query29 12.56 3.963.32 query30 0.340.120.10 query31 2.810.620.45 query32 3.240.600.52 query33 3.093.183.12 query34 16.19 5.224.47 query35 4.534.554.57 query36 0.630.520.51 query37 0.220.080.08 query38 0.190.060.05 query39 0.060.050.05 query40 0.210.180.17 query41 0.110.060.06 query42 0.070.050.05 query43 0.060.050.06 Total cold run time: 98.51 s Total hot run time: 28.41 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
doris-robot commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3547450447 TPC-DS: Total hot run time: 187712 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 22c56acb6fdbb080ac3ef44a829c31195df11696, data reload: false query1 1016400 389 389 query2 6552164917081649 query3 6756228 220 220 query4 25783 23253 22770 22770 query5 4440660 515 515 query6 352 246 248 246 query7 4660501 308 308 query8 313 264 263 263 query9 8708293529462935 query10 522 349 297 297 query11 15939 14990 14953 14953 query12 195 133 116 116 query13 1679567 446 446 query14 11148 916690449044 query15 198 194 170 170 query16 7668658 505 505 query17 1223753 619 619 query18 2029417 323 323 query19 209 198 181 181 query20 135 125 121 121 query21 217 130 116 116 query22 4014405639103910 query23 33785 33047 33064 33047 query24 8217241424332414 query25 635 518 460 460 query26 1243271 170 170 query27 2732498 356 356 query28 4413223322062206 query29 831 604 486 486 query30 292 215 188 188 query31 904 835 694 694 query32 90 83 79 79 query33 564 383 353 353 query34 793 830 527 527 query35 843 841 757 757 query36 941 971 898 898 query37 128 119 95 95 query38 3493348834303430 query39 1483142314041404 query40 222 132 123 123 query41 79 73 69 69 query42 127 117 113 113 query43 479 486 461 461 query44 1273798 807 798 query45 187 180 173 173 query46 874 984 659 659 query47 1793177317151715 query48 413 428 329 329 query49 776 500 403 403 query50 659 695 410 410 query51 3905384639403846 query52 120 118 112 112 query53 241 272 206 206 query54 315 315 318 315 query55 90 94 96 94 query56 388 341 354 341 query57 11971180 query58 300 285 294 285 query59 2550270824962496 query60 361 370 349 349 query61 166 164 161 161 query62 775 705 664 664 query63 231 202 196 196 query64 44011196894 894 query65 3996392439263924 query66 1109445 354 354 query67 15148 15094 14963 14963 query68 4610965 656 656 query69 494 349 313 313 query70 1270132912181218 query71 415 354 331 331 query72 6003518550165016 query73 641 578 365 365 query74 9004919487418741 query75 3249326127422742 query76 33521138738 738 query77 504 410 353 353 query78 9608982788998899 query79 2158836 607 607 query80 727 601 528 528 query81 532 260 224 224 query82 219 163 132 132 query83 264 267 248 248 query84 257 114 97 97 query85 892 478 448 448 query86 378 310 312 310 query87 3704368936163616 query88 2838227822372237 query89 419 345 301 301 query90 1938239 229 229 query91 199 166 137 137 query92 94 72 77 72 query93 2022997 695 695 query94 700 426 326 326 query95 435 342 333 333 query96 479 593 277 277 query97 2872296328432843 query98 236 216 211 211 query99 1280136112451245 Total cold run time: 269111 ms Total hot run time: 187712 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
doris-robot commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3547378812 TPC-H: Total hot run time: 35455 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 22c56acb6fdbb080ac3ef44a829c31195df11696, data reload: false -- Round 1 -- q1 17458 512249474947 q2 1995311 204 204 q3 10188 1289755 755 q4 10239 951 376 376 q5 7521237123982371 q6 196 178 139 139 q7 920 786 632 632 q8 9376136511311131 q9 7045515251085108 q10 6869219918031803 q11 517 312 284 284 q12 370 371 233 233 q13 17774 365230363036 q14 232 234 221 221 q15 584 507 503 503 q16 1050998 944 944 q17 605 880 359 359 q18 7615738878387388 q19 1229988 573 573 q20 378 353 231 231 q21 4445340331413141 q22 1175107611071076 Total cold run time: 107781 ms Total hot run time: 35455 ms - Round 2, with runtime_filter_mode=off - q1 5337525052785250 q2 243 327 219 219 q3 2367291224782478 q4 1442188114501450 q5 4572456744564456 q6 212 168 127 127 q7 2031191718181818 q8 2628261626042604 q9 7254727567226722 q10 2896311626672667 q11 563 505 502 502 q12 666 727 554 554 q13 3284368530553055 q14 288 279 265 265 q15 545 497 493 493 q16 1046105610021002 q17 1123150313411341 q18 7344719871197119 q19 815 823 877 823 q20 1936204118131813 q21 4768443143204320 q22 10751029996 996 Total cold run time: 52435 ms Total hot run time: 50074 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
hello-stephen commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3547106107 # FE UT Coverage Report Increment line coverage ` 16.67% (2/12)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/57702_22c56acb6fdbb080ac3ef44a829c31195df11696/fe_increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/57702_22c56acb6fdbb080ac3ef44a829c31195df11696/fe_report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
doris-robot commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3546918214 # Cloud UT Coverage Report Increment line coverage ` ` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/22c56acb6fdbb080ac3ef44a829c31195df11696_22c56acb6fdbb080ac3ef44a829c31195df11696_cloud/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/22c56acb6fdbb080ac3ef44a829c31195df11696_22c56acb6fdbb080ac3ef44a829c31195df11696_cloud/report/index.html) | Category | Coverage | |---|| | Function Coverage | 80.61% (1700/2109) | | Line Coverage | 66.69% (29785/44660) | | Region Coverage | 67.29% (14872/22101) | | Branch Coverage | 57.56% (7915/13750) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
eldenmoon commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3546746933 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
eldenmoon commented on code in PR #57702:
URL: https://github.com/apache/doris/pull/57702#discussion_r2536508069
##
be/src/olap/tablet_meta.cpp:
##
@@ -149,10 +152,12 @@ TabletMeta::TabletMeta(int64_t table_id, int64_t
partition_id, int64_t tablet_id
int64_t time_series_compaction_empty_rowsets_threshold,
int64_t time_series_compaction_level_threshold,
TInvertedIndexFileStorageFormat::type
inverted_index_file_storage_format,
- TEncryptionAlgorithm::type tde_algorithm)
+ TEncryptionAlgorithm::type tde_algorithm,
+ bool is_external_segment_meta_used_default)
: _tablet_uid(0, 0),
_schema(new TabletSchema),
- _delete_bitmap(new DeleteBitmap(tablet_id)) {
+ _delete_bitmap(new DeleteBitmap(tablet_id)),
+
_is_external_segment_meta_used_default(is_external_segment_meta_used_default) {
Review Comment:
"storage_format" = "V3" 开启后 _is_external_segment_meta_used_default
在创建tablet的时候会设置成true
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
yiguolei commented on code in PR #57702:
URL: https://github.com/apache/doris/pull/57702#discussion_r2536500919
##
be/src/olap/tablet_meta.cpp:
##
@@ -149,10 +152,12 @@ TabletMeta::TabletMeta(int64_t table_id, int64_t
partition_id, int64_t tablet_id
int64_t time_series_compaction_empty_rowsets_threshold,
int64_t time_series_compaction_level_threshold,
TInvertedIndexFileStorageFormat::type
inverted_index_file_storage_format,
- TEncryptionAlgorithm::type tde_algorithm)
+ TEncryptionAlgorithm::type tde_algorithm,
+ bool is_external_segment_meta_used_default)
: _tablet_uid(0, 0),
_schema(new TabletSchema),
- _delete_bitmap(new DeleteBitmap(tablet_id)) {
+ _delete_bitmap(new DeleteBitmap(tablet_id)),
+
_is_external_segment_meta_used_default(is_external_segment_meta_used_default) {
Review Comment:
_is_external_segment_meta_used_default 这个字段，跟我们的segmentv3 字段是什么关系？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
eldenmoon commented on code in PR #57702:
URL: https://github.com/apache/doris/pull/57702#discussion_r2536490777
##
be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:
##
@@ -1344,6 +1345,17 @@ Status VerticalSegmentWriter::_write_primary_key_index()
{
Status VerticalSegmentWriter::_write_footer() {
_footer.set_num_rows(_row_count);
+// Decide whether to externalize ColumnMetaPB by tablet default, and stamp
footer version
+DCHECK(_tablet != nullptr);
+const bool enable_external_col_meta =
+_tablet->tablet_meta()->is_external_segment_meta_used_default();
Review Comment:
或者放到tablet schema中
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
yiguolei commented on code in PR #57702:
URL: https://github.com/apache/doris/pull/57702#discussion_r2536488196
##
be/src/olap/rowset/segment_v2/vertical_segment_writer.cpp:
##
@@ -1344,6 +1345,17 @@ Status VerticalSegmentWriter::_write_primary_key_index()
{
Status VerticalSegmentWriter::_write_footer() {
_footer.set_num_rows(_row_count);
+// Decide whether to externalize ColumnMetaPB by tablet default, and stamp
footer version
+DCHECK(_tablet != nullptr);
+const bool enable_external_col_meta =
+_tablet->tablet_meta()->is_external_segment_meta_used_default();
Review Comment:
我们现在关于适用segmentv2还是v3 需要把整个tablet传递下来吗？
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
hello-stephen commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3542548149 # FE Regression Coverage Report Increment line coverage ` 23.08% (3/13)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/57702_7183ab2a5f9754ff4071d3d38bb04ce574d7e839_merge_fe/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/57702_7183ab2a5f9754ff4071d3d38bb04ce574d7e839_merge_fe/report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
doris-robot commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3541742607 ClickBench: Total hot run time: 28.64 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 7183ab2a5f9754ff4071d3d38bb04ce574d7e839, data reload: false query1 0.040.050.05 query2 0.120.070.07 query3 0.290.080.08 query4 1.600.080.08 query5 0.270.250.26 query6 1.180.690.66 query7 0.030.020.02 query8 0.080.060.07 query9 0.630.540.54 query10 0.620.580.58 query11 0.240.130.14 query12 0.260.140.14 query13 0.650.630.62 query14 1.041.031.04 query15 0.980.890.88 query16 0.390.390.38 query17 1.041.051.08 query18 0.240.220.23 query19 2.031.811.83 query20 0.020.010.01 query21 15.40 0.290.25 query22 5.020.110.10 query23 15.38 0.390.23 query24 2.550.560.35 query25 0.100.090.10 query26 0.190.180.17 query27 0.100.090.10 query28 3.601.401.13 query29 12.57 4.083.30 query30 0.330.120.10 query31 2.810.670.46 query32 3.230.630.53 query33 3.103.103.12 query34 16.18 5.334.49 query35 4.544.544.57 query36 0.630.530.52 query37 0.230.100.09 query38 0.200.060.06 query39 0.060.050.05 query40 0.210.180.18 query41 0.110.060.06 query42 0.060.050.04 query43 0.060.060.05 Total cold run time: 98.41 s Total hot run time: 28.64 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
doris-robot commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3541722047 TPC-DS: Total hot run time: 188523 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 7183ab2a5f9754ff4071d3d38bb04ce574d7e839, data reload: false query1 1017397 389 389 query2 6569172016901690 query3 6749240 232 232 query4 26292 23375 22950 22950 query5 4436668 560 560 query6 344 260 239 239 query7 4649497 314 314 query8 305 266 248 248 query9 8669294829672948 query10 497 375 307 307 query11 15454 15038 14810 14810 query12 182 125 117 117 query13 1674563 452 452 query14 10756 906090869060 query15 203 185 171 171 query16 7673689 486 486 query17 1230755 626 626 query18 2030424 326 326 query19 209 200 179 179 query20 137 124 130 124 query21 219 133 115 115 query22 3973403740024002 query23 33938 33098 33114 33098 query24 8165242424102410 query25 598 535 466 466 query26 1230281 171 171 query27 2721498 361 361 query28 4372225622432243 query29 794 643 498 498 query30 292 228 196 196 query31 905 811 732 732 query32 94 83 86 83 query33 610 392 354 354 query34 800 848 528 528 query35 824 828 771 771 query36 978 987 906 906 query37 132 114 91 91 query38 3562362134923492 query39 1470140214321402 query40 230 135 125 125 query41 62 64 62 62 query42 135 118 118 118 query43 497 473 469 469 query44 1291807 796 796 query45 181 185 169 169 query46 938 981 658 658 query47 1801179917841784 query48 405 444 334 334 query49 781 494 414 414 query50 670 673 413 413 query51 3867393538733873 query52 122 116 108 108 query53 254 288 223 223 query54 338 331 303 303 query55 96 95 88 88 query56 349 351 339 339 query57 1149118310891089 query58 302 280 287 280 query59 2485264224772477 query60 372 386 348 348 query61 183 171 158 158 query62 784 693 650 650 query63 232 208 203 203 query64 43891174917 917 query65 4043393639113911 query66 1141485 370 370 query67 15313 15070 14797 14797 query68 4612998 659 659 query69 502 346 311 311 query70 1346130913031303 query71 430 359 350 350 query72 6258509049834983 query73 664 612 373 373 query74 9230914989678967 query75 3262325527722772 query76 33121142743 743 query77 500 408 343 343 query78 9590972889078907 query79 2269863 607 607 query80 831 590 541 541 query81 542 258 220 220 query82 399 159 135 135 query83 267 271 249 249 query84 259 113 91 91 query85 887 482 454 454 query86 413 334 303 303 query87 3730366435673567 query88 2873230122492249 query89 392 332 296 296 query90 1770250 229 229 query91 169 169 133 133 query92 83 75 75 75 query93 12771044697 697 query94 726 431 336 336 query95 439 338 332 332 query96 489 593 292 292 query97 2894298328562856 query98 241 221 224 221 query99 1325140413061306 Total cold run time: 269143 ms Total hot run time: 188523 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go t
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
doris-robot commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3541678764 TPC-H: Total hot run time: 35161 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 7183ab2a5f9754ff4071d3d38bb04ce574d7e839, data reload: false -- Round 1 -- q1 17621 516851085108 q2 1988329 220 220 q3 10201 1289730 730 q4 10237 896 375 375 q5 7510240523582358 q6 203 172 140 140 q7 927 783 645 645 q8 9362134911431143 q9 7195515651825156 q10 6908224118471847 q11 509 304 293 293 q12 381 373 243 243 q13 17798 367130393039 q14 241 235 220 220 q15 586 515 502 502 q16 1054999 928 928 q17 594 899 362 362 q18 7474748276527482 q19 11991045588 588 q20 375 356 230 230 q21 4474358524932493 q22 1158113210591059 Total cold run time: 107995 ms Total hot run time: 35161 ms - Round 2, with runtime_filter_mode=off - q1 5422534154275341 q2 234 326 242 242 q3 2395291124702470 q4 1487186214161416 q5 4567440845404408 q6 208 165 130 130 q7 1928193818861886 q8 2670267125992599 q9 7363738172057205 q10 2968310926642664 q11 574 492 493 492 q12 610 718 585 585 q13 3233362130493049 q14 276 279 259 259 q15 521 493 496 493 q16 1021106610061006 q17 1104142513231323 q18 7297715369946994 q19 774 799 838 799 q20 1883198018191819 q21 4680430743464307 q22 10881032990 990 Total cold run time: 52303 ms Total hot run time: 50477 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
csun5285 commented on code in PR #57702:
URL: https://github.com/apache/doris/pull/57702#discussion_r2533790844
##
be/src/olap/rowset/segment_v2/variant/variant_ext_meta_writer.cpp:
##
@@ -0,0 +1,186 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "olap/rowset/segment_v2/variant/variant_ext_meta_writer.h"
+
+#include
+
+#include "olap/rowset/segment_v2/indexed_column_writer.h"
+#include "olap/types.h"
+#include "vec/common/schema_util.h"
+
+namespace doris::segment_v2 {
+
+Status VariantExtMetaWriter::_ensure_inited(Writers* w) {
+if (w->inited) {
+return Status::OK();
+}
+
+// key writer: VARCHAR, value index ON, ordinal index OFF
+IndexedColumnWriterOptions dict_opts;
+dict_opts.write_value_index = true;
+dict_opts.write_ordinal_index = false;
+dict_opts.encoding = PREFIX_ENCODING;
+dict_opts.compression = _comp;
+const TypeInfo* dict_type =
get_scalar_type_info();
+w->key_writer = std::make_unique(dict_opts,
dict_type, _fw);
+RETURN_IF_ERROR(w->key_writer->init());
+
+// value writer: VARCHAR, value index OFF, ordinal index ON
+IndexedColumnWriterOptions vals_opts;
+vals_opts.write_value_index = false;
+vals_opts.write_ordinal_index = true;
+vals_opts.encoding = PLAIN_ENCODING;
+vals_opts.compression = _comp;
+const TypeInfo* vals_type =
get_scalar_type_info();
+w->val_writer = std::make_unique(vals_opts,
vals_type, _fw);
+RETURN_IF_ERROR(w->val_writer->init());
+
+w->inited = true;
+return Status::OK();
+}
+
+Status VariantExtMetaWriter::add(int32_t root_uid, const Slice& key, const
Slice& val) {
+auto& w = _writers_by_uid[root_uid];
+RETURN_IF_ERROR(_ensure_inited(&w));
+RETURN_IF_ERROR(w.key_writer->add(&key));
+RETURN_IF_ERROR(w.val_writer->add(&val));
+++w.count;
+return Status::OK();
+}
+
+Status VariantExtMetaWriter::flush_to_footer(SegmentFooterPB* footer) {
+for (auto& [uid, w] : _writers_by_uid) {
+if (!w.inited || w.count == 0) {
+continue;
+}
+doris::segment_v2::IndexedColumnMetaPB key_meta;
+doris::segment_v2::IndexedColumnMetaPB val_meta;
+RETURN_IF_ERROR(w.key_writer->finish(&key_meta));
+RETURN_IF_ERROR(w.val_writer->finish(&val_meta));
+
+// keys
+std::string k = std::string("variant_meta_keys.") +
std::to_string(uid);
+std::string v;
+key_meta.AppendToString(&v);
+auto* p1 = footer->add_file_meta_datas();
+p1->set_key(k);
+p1->set_value(v);
+
+// values
+std::string k2 = std::string("variant_meta_values.") +
std::to_string(uid);
+std::string v2;
+val_meta.AppendToString(&v2);
+auto* p2 = footer->add_file_meta_datas();
+p2->set_key(k2);
+p2->set_value(v2);
+}
+_writers_by_uid.clear();
+return Status::OK();
+}
+
+Status VariantExtMetaWriter::externalize_from_footer(SegmentFooterPB* footer) {
+// Variant meta pre-processing:
+// - Collect non-sparse subcolumns per root and externalize them as KV
(path→ColumnMetaPB bytes)
+// in sorted path order to keep stability;
+// - Collect sparse subcolumns (including buckets) per root and embed them
into the corresponding
+// root variant's ColumnMetaPB.children_columns;
+// - Keep only top-level columns (including variant roots) in
footer.columns().
+std::vector kept;
Review Comment:
Variant 的 root 列的ColumnMetaPB放在footer
里面，其他的子列的，稀疏列的ColumnMetaPB可以用一个单独的结构保存在内存里，这里就不用再动 footer 了。
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
hello-stephen commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3541331264 # Cloud UT Coverage Report Increment line coverage ` ` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/7183ab2a5f9754ff4071d3d38bb04ce574d7e839_7183ab2a5f9754ff4071d3d38bb04ce574d7e839_cloud/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/7183ab2a5f9754ff4071d3d38bb04ce574d7e839_7183ab2a5f9754ff4071d3d38bb04ce574d7e839_cloud/report/index.html) | Category | Coverage | |---|| | Function Coverage | 80.56% (1683/2089) | | Line Coverage | 66.66% (29677/44518) | | Region Coverage | 67.22% (14808/22028) | | Branch Coverage | 57.49% (7880/13706) | -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [feature][segment] Support external meta V3 with common format [doris]
eldenmoon commented on PR #57702: URL: https://github.com/apache/doris/pull/57702#issuecomment-3541190476 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
