Re: [PR] [improvement](iceberg) Reconstruct partition spec in SHOW CREATE TABLE for Iceberg tables [doris]
morningman merged PR #63240: URL: https://github.com/apache/doris/pull/63240 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](iceberg) Reconstruct partition spec in SHOW CREATE TABLE for Iceberg tables [doris]
morningman commented on PR #63240: URL: https://github.com/apache/doris/pull/63240#issuecomment-4481501383 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](iceberg) Reconstruct partition spec in SHOW CREATE TABLE for Iceberg tables [doris]
github-actions[bot] commented on PR #63240: URL: https://github.com/apache/doris/pull/63240#issuecomment-4481491397 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](iceberg) Reconstruct partition spec in SHOW CREATE TABLE for Iceberg tables [doris]
nsivarajan commented on PR #63240: URL: https://github.com/apache/doris/pull/63240#issuecomment-4480470697 @morningman Please could you assist with review and merging this PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](iceberg) Reconstruct partition spec in SHOW CREATE TABLE for Iceberg tables [doris]
hello-stephen commented on PR #63240: URL: https://github.com/apache/doris/pull/63240#issuecomment-4476516494 # FE Regression Coverage Report Increment line coverage ` 0.00% (0/154)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/63240_466370175297c648477903a35041da31c9474bd1_merge_fe/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/63240_466370175297c648477903a35041da31c9474bd1_merge_fe/report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](iceberg) Reconstruct partition spec in SHOW CREATE TABLE for Iceberg tables [doris]
nsivarajan commented on PR #63240: URL: https://github.com/apache/doris/pull/63240#issuecomment-4475615717 run cloud_p0 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](iceberg) Reconstruct partition spec in SHOW CREATE TABLE for Iceberg tables [doris]
hello-stephen commented on PR #63240: URL: https://github.com/apache/doris/pull/63240#issuecomment-4475535057 # FE Regression Coverage Report Increment line coverage ` 0.00% (0/69)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/63240_466370175297c648477903a35041da31c9474bd1_merge_fe/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/63240_466370175297c648477903a35041da31c9474bd1_merge_fe/report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](iceberg) Reconstruct partition spec in SHOW CREATE TABLE for Iceberg tables [doris]
hello-stephen commented on PR #63240: URL: https://github.com/apache/doris/pull/63240#issuecomment-4475477956 TPC-DS: Total hot run time: 170073 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 466370175297c648477903a35041da31c9474bd1, data reload: false query5 4312649 499 499 query6 329 215 199 199 query7 4284580 316 316 query8 321 226 224 224 query9 8810400539793979 query10 446 337 291 291 query11 5772240322132213 query12 187 128 125 125 query13 1282622 418 418 query14 5955535050525052 query14_14325434243074307 query15 221 198 183 183 query16 1017447 421 421 query17 1059726 584 584 query18 2467466 359 359 query19 228 219 190 190 query20 147 133 132 132 query21 213 144 125 125 query22 13630 13624 13410 13410 query23 17196 16423 16023 16023 query23_116153 16098 16177 16098 query24 7495177212881288 query24_11296129012971290 query25 582 489 439 439 query26 1322311 183 183 query27 2674595 350 350 query28 4478198519381938 query29 996 641 522 522 query30 304 233 201 201 query31 11171078950 950 query32 101 79 76 76 query33 564 380 327 327 query34 11481092638 638 query35 757 791 665 665 query36 1317134511851185 query37 154 109 94 94 query38 3227317030593059 query39 942 944 905 905 query39_1882 887 885 885 query40 237 152 136 136 query41 73 70 69 69 query42 115 112 116 112 query43 324 331 287 287 query44 query45 216 209 198 198 query46 10861160743 743 query47 2299232221332133 query48 396 437 303 303 query49 646 512 402 402 query50 971 356 253 253 query51 4265422342494223 query52 111 110 100 100 query53 263 289 212 212 query54 328 293 279 279 query55 96 95 89 89 query56 331 334 313 313 query57 1403140613331333 query58 319 285 302 285 query59 1582159913891389 query60 323 328 306 306 query61 149 154 151 151 query62 669 627 565 565 query63 255 205 206 205 query64 2427788 625 625 query65 query66 1741469 363 363 query67 30072 29927 29870 29870 query68 query69 462 346 309 309 query70 1044991 1028991 query71 314 277 271 271 query72 2965274824152415 query73 866 730 438 438 query74 5082490747414741 query75 2693260022602260 query76 23121117761 761 query77 393 417 325 325 query78 12285 12166 11694 11694 query79 14611028742 742 query80 657 576 443 443 query81 459 277 248 248 query82 1386159 128 128 query83 355 274 244 244 query84 278 137 111 111 query85 898 546 457 457 query86 397 337 291 291 query87 3422338632223222 query88 3547265726512651 query89 446 389 340 340 query90 1887174 179 174 query91 173 170 137 137 query92 76 77 72 72 query93 14841437946 946 query94 550 364 304 304 query95 660 467 344 344 query96 1077730 338 338 query97 2714269825602560 query98 241 229 232 229 query99 10971084987 987 Total cold run time: 253021 ms Total hot run time: 170073 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected]
Re: [PR] [improvement](iceberg) Reconstruct partition spec in SHOW CREATE TABLE for Iceberg tables [doris]
hello-stephen commented on PR #63240: URL: https://github.com/apache/doris/pull/63240#issuecomment-4475405277 TPC-H: Total hot run time: 31296 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 466370175297c648477903a35041da31c9474bd1, data reload: false -- Round 1 -- orders Doris NULLNULL0 0 0 NULL0 NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 17754 399438163816 q2 q3 10806 1347827 827 q4 4689475 347 347 q5 7631228221432143 q6 242 175 144 144 q7 951 799 621 621 q8 9451174716441644 q9 5183491449144914 q10 6377206118041804 q11 450 270 242 242 q12 634 434 298 298 q13 18100 339428042804 q14 259 250 235 235 q15 q16 829 766 697 697 q17 946 949 906 906 q18 6912561455685568 q19 1193118110331033 q20 519 405 295 295 q21 5874285926292629 q22 453 396 329 329 Total cold run time: 99253 ms Total hot run time: 31296 ms - Round 2, with runtime_filter_mode=off - orders Doris NULLNULL15000 42 6422171781 NULL22778155NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 4753454245674542 q2 q3 4926522946334633 q4 2129220714141414 q5 4851459546004595 q6 227 176 144 144 q7 1945171615531553 q8 2373202520422025 q9 7757733271687168 q10 4470439339563956 q11 523 381 346 346 q12 713 715 513 513 q13 3038337128232823 q14 283 271 256 256 q15 q16 677 697 619 619 q17 1249123612351235 q18 7254698668146814 q19 1095110711171107 q20 2219220419391939 q21 5334460245004500 q22 520 447 408 408 Total cold run time: 56336 ms Total hot run time: 50590 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](iceberg) Reconstruct partition spec in SHOW CREATE TABLE for Iceberg tables [doris]
hello-stephen commented on PR #63240: URL: https://github.com/apache/doris/pull/63240#issuecomment-4474540620 # FE UT Coverage Report Increment line coverage ` 77.27% (34/44)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/63240_466370175297c648477903a35041da31c9474bd1/fe_increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/63240_466370175297c648477903a35041da31c9474bd1/fe_report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](iceberg) Reconstruct partition spec in SHOW CREATE TABLE for Iceberg tables [doris]
nsivarajan commented on code in PR #63240:
URL: https://github.com/apache/doris/pull/63240#discussion_r3256377027
##
fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java:
##
@@ -4861,6 +4865,10 @@ public static void getDdlStmt(Command command, String
dbName, TableIf table, Lis
if (icebergExternalTable.hasSortOrder()) {
sb.append("\n").append(icebergExternalTable.getSortOrderSql());
}
+String partitionSpecSql =
icebergExternalTable.getPartitionSpecSql();
+if (!partitionSpecSql.isEmpty()) {
Review Comment:
This PR followed , getSortOrderSql() pattern - missed on edge case, now a
new commit fixes it. fixing getSortOrderSql() will be done in next PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [improvement](iceberg) Reconstruct partition spec in SHOW CREATE TABLE for Iceberg tables [doris]
nsivarajan commented on PR #63240: URL: https://github.com/apache/doris/pull/63240#issuecomment-4474074959 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](iceberg) Reconstruct partition spec in SHOW CREATE TABLE for Iceberg tables [doris]
nsivarajan commented on PR #63240: URL: https://github.com/apache/doris/pull/63240#issuecomment-4474047860 This PR followed , getSortOrderSql() pattern - missed on edge case, now a new commit fixes it. fixing getSortOrderSql() will be done in next PR -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](iceberg) Reconstruct partition spec in SHOW CREATE TABLE for Iceberg tables [doris]
hello-stephen commented on PR #63240: URL: https://github.com/apache/doris/pull/63240#issuecomment-4472356144 # FE UT Coverage Report Increment line coverage ` 80.95% (34/42)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/63240_9f991a88784c4e80be366439898b090b172d2ef7/fe_increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/63240_9f991a88784c4e80be366439898b090b172d2ef7/fe_report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](iceberg) Reconstruct partition spec in SHOW CREATE TABLE for Iceberg tables [doris]
hello-stephen commented on PR #63240: URL: https://github.com/apache/doris/pull/63240#issuecomment-4472244040 TPC-DS: Total hot run time: 171531 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 9f991a88784c4e80be366439898b090b172d2ef7, data reload: false query5 4322682 531 531 query6 346 222 217 217 query7 4217581 320 320 query8 325 237 223 223 query9 8829405940624059 query10 456 350 307 307 query11 5803246722342234 query12 186 132 132 132 query13 1286630 429 429 query14 6121540951015101 query14_14430440343784378 query15 220 212 189 189 query16 996 464 455 455 query17 1162762 641 641 query18 2557511 368 368 query19 260 199 160 160 query20 139 132 128 128 query21 214 144 121 121 query22 13654 13564 13480 13480 query23 17319 16250 15939 15939 query23_116136 16194 16022 16022 query24 7493176813251325 query24_11328131913011301 query25 565 499 428 428 query26 1311322 174 174 query27 2698564 346 346 query28 4481197319351935 query29 996 653 505 505 query30 308 244 197 197 query31 11231054931 931 query32 91 77 73 73 query33 533 355 331 331 query34 12001173650 650 query35 769 778 683 683 query36 1300133811351135 query37 151 107 91 91 query38 3235313930793079 query39 935 932 896 896 query39_1876 880 872 872 query40 227 150 131 131 query41 66 63 65 63 query42 110 123 111 111 query43 325 340 293 293 query44 query45 214 202 195 195 query46 11011232726 726 query47 2202226721492149 query48 386 422 300 300 query49 632 500 411 411 query50 1048365 262 262 query51 4328431942954295 query52 104 112 101 101 query53 264 304 222 222 query54 328 294 262 262 query55 95 95 87 87 query56 344 331 320 320 query57 1430140813641364 query58 314 292 282 282 query59 1637170614991499 query60 343 357 337 337 query61 216 175 182 175 query62 677 642 577 577 query63 248 211 219 211 query64 2476864 722 722 query65 query66 1743498 392 392 query67 30073 30050 29837 29837 query68 query69 480 354 318 318 query70 1033101510341015 query71 332 342 273 273 query72 3158276824422442 query73 851 809 443 443 query74 5054487747224722 query75 2730260622462246 query76 22701180796 796 query77 420 419 336 336 query78 12061 12174 11583 11583 query79 14591006748 748 query80 810 564 463 463 query81 477 279 241 241 query82 1351159 127 127 query83 356 326 254 254 query84 259 148 111 111 query85 942 552 470 470 query86 446 343 335 335 query87 3392336432173217 query88 3575266626812666 query89 456 390 342 342 query90 1790192 183 183 query91 187 170 147 147 query92 85 80 74 74 query93 14661483863 863 query94 619 348 327 327 query95 677 389 436 389 query96 1053829 328 328 query97 2681269025902590 query98 241 233 229 229 query99 11091100970 970 Total cold run time: 254082 ms Total hot run time: 171531 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected]
Re: [PR] [improvement](iceberg) Reconstruct partition spec in SHOW CREATE TABLE for Iceberg tables [doris]
github-actions[bot] commented on code in PR #63240:
URL: https://github.com/apache/doris/pull/63240#discussion_r3255348590
##
fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java:
##
@@ -4861,6 +4865,10 @@ public static void getDdlStmt(Command command, String
dbName, TableIf table, Lis
if (icebergExternalTable.hasSortOrder()) {
sb.append("\n").append(icebergExternalTable.getSortOrderSql());
}
+String partitionSpecSql =
icebergExternalTable.getPartitionSpecSql();
+if (!partitionSpecSql.isEmpty()) {
Review Comment:
This also runs when `table` is an `IcebergSysExternalTable`, because the
branch above replaces it with `((IcebergSysExternalTable)
table).getSourceTable()`. The CREATE TABLE header and column list are still
built from the system metadata table (for example `base$files`), but this new
clause is taken from the source table and can reference source columns that do
not exist in the metadata-table schema. For a partitioned source table, `SHOW
CREATE TABLE base$files` would emit a non-replayable/mismatched DDL. Please
skip source-table partition spec emission for `IcebergSysExternalTable`, or
derive it from the actual system Iceberg table if system-table DDL is intended
to include partitioning. The same issue exists in the earlier duplicated Env
path added in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [improvement](iceberg) Reconstruct partition spec in SHOW CREATE TABLE for Iceberg tables [doris]
morningman commented on PR #63240: URL: https://github.com/apache/doris/pull/63240#issuecomment-4472208178 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](iceberg) Reconstruct partition spec in SHOW CREATE TABLE for Iceberg tables [doris]
hello-stephen commented on PR #63240: URL: https://github.com/apache/doris/pull/63240#issuecomment-4472205619 TPC-H: Total hot run time: 31295 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 9f991a88784c4e80be366439898b090b172d2ef7, data reload: false -- Round 1 -- orders Doris NULLNULL0 0 0 NULL0 NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 17716 388538233823 q2 q3 10779 1382793 793 q4 4679474 339 339 q5 7601223621302130 q6 356 180 142 142 q7 919 783 621 621 q8 9405172916111611 q9 7016492649514926 q10 6415210317631763 q11 440 279 247 247 q12 688 434 294 294 q13 18190 334427632763 q14 260 258 246 246 q15 q16 820 765 701 701 q17 961 951 1077951 q18 6724562256225622 q19 1195125711501150 q20 552 428 282 282 q21 5954274525662566 q22 437 387 325 325 Total cold run time: 101107 ms Total hot run time: 31295 ms - Round 2, with runtime_filter_mode=off - orders Doris NULLNULL15000 42 6422171781 NULL22778155NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 4699461247604612 q2 q3 4821519746374637 q4 2143219913971397 q5 5070477146784678 q6 228 176 132 132 q7 1838172015461546 q8 2419211319621962 q9 7295730472487248 q10 4494438839703970 q11 531 386 358 358 q12 703 714 510 510 q13 2992339927502750 q14 265 278 251 251 q15 q16 680 697 623 623 q17 1249123612341234 q18 7321681169856811 q19 1132110710771077 q20 2221223019361936 q21 5287460944234423 q22 523 459 416 416 Total cold run time: 55911 ms Total hot run time: 50571 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement](iceberg) Reconstruct partition spec in SHOW CREATE TABLE for Iceberg tables [doris]
nsivarajan commented on code in PR #63240:
URL: https://github.com/apache/doris/pull/63240#discussion_r3255295633
##
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java:
##
@@ -459,4 +459,52 @@ public boolean hasSortOrder() {
org.apache.iceberg.SortOrder sortOrder = table.sortOrder();
return sortOrder != null && !sortOrder.isUnsorted();
}
+
+/** Reconstructs PARTITION BY LIST (...) () from the Iceberg PartitionSpec
for SHOW CREATE TABLE. */
+public String getPartitionSpecSql() {
+makeSureInitialized();
+Table table = getIcebergTable();
+PartitionSpec spec = table.spec();
+if (spec == null || spec.isUnpartitioned()) {
+return "";
+}
+List fields = new ArrayList<>();
+for (PartitionField field : spec.fields()) {
+String colName = table.schema().findColumnName(field.sourceId());
+if (colName == null) {
+continue;
+}
+org.apache.iceberg.transforms.Transform t =
field.transform();
+// isVoid/isIdentity: public interface methods; toString():
canonical spec-defined names.
+if (t.isVoid()) {
+continue;
+} else if (t.isIdentity()) {
+fields.add(colName);
+} else {
Review Comment:
Thanks! I have added quotes around for displaying column names as suggested
with latest commit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [improvement](iceberg) Reconstruct partition spec in SHOW CREATE TABLE for Iceberg tables [doris]
nsivarajan commented on PR #63240: URL: https://github.com/apache/doris/pull/63240#issuecomment-4472140958 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement] (iceberg) Reconstruct partition spec in SHOW CREATE TABLE for Iceberg tables [doris]
morningman commented on code in PR #63240:
URL: https://github.com/apache/doris/pull/63240#discussion_r3255167665
##
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java:
##
@@ -459,4 +459,52 @@ public boolean hasSortOrder() {
org.apache.iceberg.SortOrder sortOrder = table.sortOrder();
return sortOrder != null && !sortOrder.isUnsorted();
}
+
+/** Reconstructs PARTITION BY LIST (...) () from the Iceberg PartitionSpec
for SHOW CREATE TABLE. */
+public String getPartitionSpecSql() {
+makeSureInitialized();
+Table table = getIcebergTable();
+PartitionSpec spec = table.spec();
+if (spec == null || spec.isUnpartitioned()) {
+return "";
+}
+List fields = new ArrayList<>();
+for (PartitionField field : spec.fields()) {
+String colName = table.schema().findColumnName(field.sourceId());
+if (colName == null) {
+continue;
+}
+org.apache.iceberg.transforms.Transform t =
field.transform();
+// isVoid/isIdentity: public interface methods; toString():
canonical spec-defined names.
+if (t.isVoid()) {
+continue;
+} else if (t.isIdentity()) {
+fields.add(colName);
+} else {
Review Comment:
@nsivarajan PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [improvement] (iceberg) Reconstruct partition spec in SHOW CREATE TABLE for Iceberg tables [doris]
github-actions[bot] commented on code in PR #63240:
URL: https://github.com/apache/doris/pull/63240#discussion_r3255159231
##
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalTable.java:
##
@@ -459,4 +459,52 @@ public boolean hasSortOrder() {
org.apache.iceberg.SortOrder sortOrder = table.sortOrder();
return sortOrder != null && !sortOrder.isUnsorted();
}
+
+/** Reconstructs PARTITION BY LIST (...) () from the Iceberg PartitionSpec
for SHOW CREATE TABLE. */
+public String getPartitionSpecSql() {
+makeSureInitialized();
+Table table = getIcebergTable();
+PartitionSpec spec = table.spec();
+if (spec == null || spec.isUnpartitioned()) {
+return "";
+}
+List fields = new ArrayList<>();
+for (PartitionField field : spec.fields()) {
+String colName = table.schema().findColumnName(field.sourceId());
+if (colName == null) {
+continue;
+}
+org.apache.iceberg.transforms.Transform t =
field.transform();
+// isVoid/isIdentity: public interface methods; toString():
canonical spec-defined names.
+if (t.isVoid()) {
+continue;
+} else if (t.isIdentity()) {
+fields.add(colName);
+} else {
Review Comment:
`colName` is emitted raw in this and the transform cases below. Iceberg
tables can have partition columns that require quoting in Doris SQL, such as a
reserved name like `select` or a name containing special characters/backticks.
The table column list in `SHOW CREATE TABLE` is quoted, but this clause would
generate non-replayable SQL like `PARTITION BY LIST (select) ()` or `BUCKET(16,
a-b)`. Please format partition column references the same way as other DDL
output, including backtick quoting and escaping embedded backticks, before
adding them to `fields`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [improvement] (iceberg) Reconstruct partition spec in SHOW CREATE TABLE for Iceberg tables [doris]
github-actions[bot] commented on PR #63240: URL: https://github.com/apache/doris/pull/63240#issuecomment-4471764793 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement] (iceberg) Reconstruct partition spec in SHOW CREATE TABLE for Iceberg tables [doris]
github-actions[bot] commented on PR #63240: URL: https://github.com/apache/doris/pull/63240#issuecomment-4471764574 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement] (iceberg) Reconstruct partition spec in SHOW CREATE TABLE for Iceberg tables [doris]
morningman commented on PR #63240: URL: https://github.com/apache/doris/pull/63240#issuecomment-4471756420 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement] (iceberg) Reconstruct partition spec in SHOW CREATE TABLE for Iceberg tables [doris]
hello-stephen commented on PR #63240: URL: https://github.com/apache/doris/pull/63240#issuecomment-4454374462 # FE Regression Coverage Report Increment line coverage ` 0.43% (5/1162)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/63240_b44a3c2a22bbe6b2fb92d871cceec59907d7e3d6_merge_fe/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/63240_b44a3c2a22bbe6b2fb92d871cceec59907d7e3d6_merge_fe/report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement] (iceberg) Reconstruct partition spec in SHOW CREATE TABLE for Iceberg tables [doris]
nsivarajan commented on PR #63240: URL: https://github.com/apache/doris/pull/63240#issuecomment-4453675513 run external -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement] (iceberg) Reconstruct partition spec in SHOW CREATE TABLE for Iceberg tables [doris]
hello-stephen commented on PR #63240: URL: https://github.com/apache/doris/pull/63240#issuecomment-4449039409 # FE Regression Coverage Report Increment line coverage ` 0.52% (5/961)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/63240_b44a3c2a22bbe6b2fb92d871cceec59907d7e3d6_merge_fe/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/63240_b44a3c2a22bbe6b2fb92d871cceec59907d7e3d6_merge_fe/report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement] (iceberg) Reconstruct partition spec in SHOW CREATE TABLE for Iceberg tables [doris]
hello-stephen commented on PR #63240: URL: https://github.com/apache/doris/pull/63240#issuecomment-4448724925 TPC-DS: Total hot run time: 170023 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit b44a3c2a22bbe6b2fb92d871cceec59907d7e3d6, data reload: false query5 4329638 523 523 query6 314 228 197 197 query7 4225552 320 320 query8 318 228 220 220 query9 8811395639813956 query10 450 337 299 299 query11 5677241022072207 query12 204 129 131 129 query13 1268594 413 413 query14 6742531550025002 query14_14310431543054305 query15 213 198 178 178 query16 1026414 429 414 query17 1132753 627 627 query18 2716468 353 353 query19 224 209 167 167 query20 142 138 135 135 query21 220 146 133 133 query22 13620 13491 13334 13334 query23 17230 16270 15976 15976 query23_116104 16195 16147 16147 query24 7363175413771377 query24_11363135013581350 query25 581 533 479 479 query26 840 319 177 177 query27 2678589 348 348 query28 4392193419141914 query29 1013657 555 555 query30 305 239 197 197 query31 11341079940 940 query32 88 82 77 77 query33 551 358 313 313 query34 11771090650 650 query35 773 785 707 707 query36 1337138611571157 query37 151 109 94 94 query38 3236312430573057 query39 939 937 898 898 query39_1883 869 870 869 query40 242 164 141 141 query41 75 68 69 68 query42 111 109 110 109 query43 327 329 286 286 query44 query45 210 202 201 201 query46 10511182732 732 query47 2365239122422242 query48 406 426 299 299 query49 634 555 451 451 query50 715 284 230 230 query51 4323434242804280 query52 108 111 103 103 query53 262 282 216 216 query54 332 302 282 282 query55 97 94 89 89 query56 329 331 347 331 query57 1486144813781378 query58 319 293 283 283 query59 1581159714381438 query60 385 335 322 322 query61 153 155 156 155 query62 680 611 556 556 query63 244 198 209 198 query64 1733824 671 671 query65 query66 1639521 397 397 query67 30073 29892 29834 29834 query68 query69 456 353 305 305 query70 10501022985 985 query71 297 273 276 273 query72 3025270724262426 query73 861 765 414 414 query74 5068487247024702 query75 2758266723262326 query76 22741121807 807 query77 449 442 351 351 query78 12921 13002 12380 12380 query79 14911001739 739 query80 1337554 497 497 query81 525 276 241 241 query82 951 160 128 128 query83 324 272 251 251 query84 252 171 108 108 query85 908 538 449 449 query86 460 347 299 299 query87 3459337532163216 query88 3528265326262626 query89 446 387 341 341 query90 1871185 184 184 query91 179 169 144 144 query92 81 75 73 73 query93 1174948 588 588 query94 710 343 299 299 query95 665 377 341 341 query96 1051742 333 333 query97 2693267825662566 query98 236 232 226 226 query99 1115114610271027 Total cold run time: 253407 ms Total hot run time: 170023 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected]
Re: [PR] [improvement] (iceberg) Reconstruct partition spec in SHOW CREATE TABLE for Iceberg tables [doris]
hello-stephen commented on PR #63240: URL: https://github.com/apache/doris/pull/63240#issuecomment-4448646885 TPC-H: Total hot run time: 29362 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit b44a3c2a22bbe6b2fb92d871cceec59907d7e3d6, data reload: false -- Round 1 -- orders Doris NULLNULL0 0 0 NULL0 NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 17852 395538343834 q2 q3 10717 862 590 590 q4 4667457 345 345 q5 7448130511281128 q6 184 168 133 133 q7 901 960 736 736 q8 9409137412331233 q9 6017535952635263 q10 6299206818211821 q11 475 259 251 251 q12 687 409 294 294 q13 18185 330227232723 q14 285 279 256 256 q15 q16 900 860 795 795 q17 1076958 750 750 q18 6466574755645564 q19 1525125211101110 q20 509 395 265 265 q21 4664237319561956 q22 453 384 315 315 Total cold run time: 98719 ms Total hot run time: 29362 ms - Round 2, with runtime_filter_mode=off - orders Doris NULLNULL15000 42 6422171781 NULL22778155NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 4652453445424534 q2 q3 4686476842574257 q4 2088220213971397 q5 4931501752315017 q6 196 167 129 129 q7 2051177016191619 q8 3314312130873087 q9 8452863984698469 q10 4435443442334233 q11 616 423 401 401 q12 743 757 549 549 q13 3269373329192919 q14 310 328 303 303 q15 q16 756 806 683 683 q17 1326130112601260 q18 7994717871207120 q19 1139113111551131 q20 2198219719221922 q21 5990528447494749 q22 516 474 399 399 Total cold run time: 59662 ms Total hot run time: 54178 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement] (iceberg) Reconstruct partition spec in SHOW CREATE TABLE for Iceberg tables [doris]
hello-stephen commented on PR #63240: URL: https://github.com/apache/doris/pull/63240#issuecomment-4448065275 TPC-H: Total hot run time: 29380 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit b44a3c2a22bbe6b2fb92d871cceec59907d7e3d6, data reload: false -- Round 1 -- orders Doris NULLNULL0 0 0 NULL0 NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 17660 387838303830 q2 q3 10704 875 604 604 q4 4661466 354 354 q5 7470134011471147 q6 216 171 142 142 q7 966 949 753 753 q8 10004 140313321332 q9 6177537253615361 q10 6307208318071807 q11 466 265 256 256 q12 697 414 292 292 q13 18265 325527192719 q14 293 284 264 264 q15 q16 903 873 792 792 q17 10411023743 743 q18 6510574955455545 q19 17601202995 995 q20 501 396 257 257 q21 4690226518811881 q22 424 356 306 306 Total cold run time: 99715 ms Total hot run time: 29380 ms - Round 2, with runtime_filter_mode=off - orders Doris NULLNULL15000 42 6422171781 NULL22778155NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 4241412540984098 q2 q3 4618478341674167 q4 2109214413851385 q5 4920498552504985 q6 189 165 132 132 q7 2041206816671667 q8 3441320231743174 q9 8471849984098409 q10 4491449042254225 q11 607 413 391 391 q12 693 757 511 511 q13 3248365930033003 q14 305 314 274 274 q15 q16 764 776 716 716 q17 1335132913831329 q18 8031712271287122 q19 1148117211401140 q20 2241227519491949 q21 6077550349654965 q22 587 505 469 469 Total cold run time: 59557 ms Total hot run time: 54111 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement] (iceberg) Reconstruct partition spec in SHOW CREATE TABLE for Iceberg tables [doris]
nsivarajan commented on PR #63240: URL: https://github.com/apache/doris/pull/63240#issuecomment-4447677650 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [improvement] (iceberg) Reconstruct partition spec in SHOW CREATE TABLE for Iceberg tables [doris]
hello-stephen commented on PR #63240: URL: https://github.com/apache/doris/pull/63240#issuecomment-4447658682 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
