Re: [PR] [fix](catalog) Mask sensitive properties in show create catalog [doris]
Gabriel39 merged PR #63234: URL: https://github.com/apache/doris/pull/63234 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](catalog) Mask sensitive properties in show create catalog [doris]
github-actions[bot] commented on PR #63234: URL: https://github.com/apache/doris/pull/63234#issuecomment-4531179486 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](catalog) Mask sensitive properties in show create catalog [doris]
Gabriel39 commented on PR #63234: URL: https://github.com/apache/doris/pull/63234#issuecomment-4531155919 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](catalog) Mask sensitive properties in show create catalog [doris]
hello-stephen commented on PR #63234: URL: https://github.com/apache/doris/pull/63234#issuecomment-4519022891 # FE Regression Coverage Report Increment line coverage ` 100.00% (1/1)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/63234_30421552a0932f4713550fa082b45e5eaa3faf02_merge_fe/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/63234_30421552a0932f4713550fa082b45e5eaa3faf02_merge_fe/report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](catalog) Mask sensitive properties in show create catalog [doris]
hello-stephen commented on PR #63234: URL: https://github.com/apache/doris/pull/63234#issuecomment-4518395670 # FE UT Coverage Report Increment line coverage ` 100.00% (1/1)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/63234_30421552a0932f4713550fa082b45e5eaa3faf02/fe_increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/63234_30421552a0932f4713550fa082b45e5eaa3faf02/fe_report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](catalog) Mask sensitive properties in show create catalog [doris]
hello-stephen commented on PR #63234: URL: https://github.com/apache/doris/pull/63234#issuecomment-4518119514 TPC-DS: Total hot run time: 169942 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 30421552a0932f4713550fa082b45e5eaa3faf02, data reload: false query5 4306657 533 533 query6 331 219 200 200 query7 4222590 303 303 query8 331 234 219 219 query9 8821396039873960 query10 459 347 300 300 query11 5831240921812181 query12 170 129 126 126 query13 1300608 454 454 query14 5969537950465046 query14_14342436042914291 query15 208 201 180 180 query16 996 461 394 394 query17 1116715 579 579 query18 2617487 348 348 query19 212 194 175 175 query20 133 134 129 129 query21 214 140 113 113 query22 13677 13539 13377 13377 query23 17072 16301 15924 15924 query23_116219 16189 16242 16189 query24 7434176313081308 query24_11281129812671267 query25 543 465 416 416 query26 1326334 174 174 query27 2670549 342 342 query28 4450195519501950 query29 992 658 514 514 query30 310 240 196 196 query31 11131083952 952 query32 94 82 74 74 query33 561 363 315 315 query34 11801118654 654 query35 766 777 684 684 query36 1330132612161216 query37 158 107 93 93 query38 3200318531093109 query39 922 921 898 898 query39_1881 870 866 866 query40 223 153 125 125 query41 71 70 69 69 query42 111 111 111 111 query43 326 330 287 287 query44 query45 218 205 194 194 query46 10871199711 711 query47 2279231621552155 query48 399 429 293 293 query49 654 504 408 408 query50 981 349 247 247 query51 4286432942414241 query52 106 107 101 101 query53 262 275 207 207 query54 326 285 266 266 query55 98 93 87 87 query56 312 325 311 311 query57 1413139113021302 query58 318 278 267 267 query59 1558158914291429 query60 333 339 336 336 query61 184 175 176 175 query62 671 615 566 566 query63 286 200 201 200 query64 2390791 631 631 query65 query66 1664487 357 357 query67 30018 29893 29853 29853 query68 query69 475 339 303 303 query70 987 929 949 929 query71 296 271 276 271 query72 2941265324162416 query73 857 786 425 425 query74 5059489947194719 query75 2680260722512251 query76 22931141794 794 query77 390 402 327 327 query78 12209 12088 11643 11643 query79 14391052724 724 query80 644 532 451 451 query81 452 279 241 241 query82 1365156 119 119 query83 352 277 252 252 query84 263 136 111 111 query85 869 542 442 442 query86 412 341 310 310 query87 3416338632013201 query88 3485262426322624 query89 444 383 335 335 query90 1883180 182 180 query91 181 169 137 137 query92 76 78 75 75 query93 14131495940 940 query94 559 364 307 307 query95 686 394 445 394 query96 995 806 350 350 query97 2699270625552555 query98 248 229 223 223 query99 11541110999 999 Total cold run time: 252539 ms Total hot run time: 169942 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected]
Re: [PR] [fix](catalog) Mask sensitive properties in show create catalog [doris]
hello-stephen commented on PR #63234: URL: https://github.com/apache/doris/pull/63234#issuecomment-4518051930 TPC-H: Total hot run time: 31479 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 30421552a0932f4713550fa082b45e5eaa3faf02, data reload: false -- Round 1 -- orders Doris NULLNULL0 0 0 NULL0 NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 17698 406839493949 q2 q3 10777 1372808 808 q4 4683474 344 344 q5 7659231822182218 q6 243 183 143 143 q7 959 783 639 639 q8 9370189515441544 q9 5127493849104910 q10 6395209417481748 q11 440 266 244 244 q12 626 410 297 297 q13 18107 348028002800 q14 262 267 238 238 q15 q16 822 778 713 713 q17 921 939 988 939 q18 7036579156425642 q19 1302134010211021 q20 622 475 292 292 q21 6117283926752675 q22 461 448 315 315 Total cold run time: 99627 ms Total hot run time: 31479 ms - Round 2, with runtime_filter_mode=off - orders Doris NULLNULL15000 42 6422171781 NULL22778155NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 4811474646944694 q2 q3 5031537846274627 q4 2178222814131413 q5 4846482246824682 q6 234 184 134 134 q7 1856178515361536 q8 2476214422662144 q9 7790747072917291 q10 4475439239943994 q11 572 388 355 355 q12 708 729 518 518 q13 2997345928052805 q14 277 275 244 244 q15 q16 683 698 611 611 q17 1284126312561256 q18 7358687166966696 q19 1079109310871087 q20 2214221319301930 q21 5452512745954595 q22 540 467 393 393 Total cold run time: 56861 ms Total hot run time: 51005 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](catalog) Mask sensitive properties in show create catalog [doris]
xyla commented on PR #63234: URL: https://github.com/apache/doris/pull/63234#issuecomment-4517858627 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](catalog) Mask sensitive properties in show create catalog [doris]
github-actions[bot] commented on code in PR #63234:
URL: https://github.com/apache/doris/pull/63234#discussion_r3286482343
##
fe/fe-core/src/main/java/org/apache/doris/common/util/DatasourcePrintableMap.java:
##
@@ -58,6 +59,7 @@ public class DatasourcePrintableMap extends
BasicPrintableMap {
SENSITIVE_KEY.addAll(ConnectorPropertiesUtils.getSensitiveKeys(S3Properties.class));
SENSITIVE_KEY.addAll(ConnectorPropertiesUtils.getSensitiveKeys(AliyunDLFBaseProperties.class));
SENSITIVE_KEY.addAll(ConnectorPropertiesUtils.getSensitiveKeys(AWSGlueMetaStoreBaseProperties.class));
+
SENSITIVE_KEY.addAll(ConnectorPropertiesUtils.getSensitiveKeys(IcebergRestProperties.class));
SENSITIVE_KEY.addAll(ConnectorPropertiesUtils.getSensitiveKeys(GCSProperties.class));
Review Comment:
This registers Iceberg REST keys only if they are marked `sensitive` in
`IcebergRestProperties`. The OAuth2 token path is a parallel supported
authentication mode (`iceberg.rest.oauth2.token`), but that property is not
marked sensitive, so a catalog created with a bearer token instead of
`iceberg.rest.oauth2.credential` will still have the raw token printed by `SHOW
CREATE CATALOG`. Please mark the token property sensitive as well and add
coverage for the token case, otherwise this fix only closes one of the two
OAuth2 secret paths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](catalog) Mask sensitive properties in show create catalog [doris]
Gabriel39 commented on PR #63234: URL: https://github.com/apache/doris/pull/63234#issuecomment-4515987056 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](catalog) Mask sensitive properties in show create catalog [doris]
github-actions[bot] commented on PR #63234: URL: https://github.com/apache/doris/pull/63234#issuecomment-4515960674 PR approved by at least one committer and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](catalog) Mask sensitive properties in show create catalog [doris]
github-actions[bot] commented on PR #63234: URL: https://github.com/apache/doris/pull/63234#issuecomment-4515961069 PR approved by anyone and no changes requested. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](catalog) Mask sensitive properties in show create catalog [doris]
hello-stephen commented on PR #63234: URL: https://github.com/apache/doris/pull/63234#issuecomment-4448239038 # FE Regression Coverage Report Increment line coverage ` 6.25% (1/16)` :tada: [Increment coverage report](http://coverage.selectdb-in.cc/coverage/63234_c532bff04b03eb41e32784df0191177072d18868_merge_fe/increment_report/index.html) [Complete coverage report](http://coverage.selectdb-in.cc/coverage/63234_c532bff04b03eb41e32784df0191177072d18868_merge_fe/report/index.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](catalog) Mask sensitive properties in show create catalog [doris]
hello-stephen commented on PR #63234: URL: https://github.com/apache/doris/pull/63234#issuecomment-4448087084 TPC-DS: Total hot run time: 171383 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit c532bff04b03eb41e32784df0191177072d18868, data reload: false query5 4316665 534 534 query6 337 215 195 195 query7 4274537 317 317 query8 318 250 220 220 query9 8837400139723972 query10 461 342 319 319 query11 5816239521712171 query12 178 127 127 127 query13 1277604 445 445 query14 6731532850065006 query14_14297431042714271 query15 215 204 182 182 query16 1032457 428 428 query17 1184759 608 608 query18 2732503 345 345 query19 222 206 171 171 query20 139 132 128 128 query21 228 144 117 117 query22 13632 13963 14356 13963 query23 17340 16562 16155 16155 query23_116395 16221 16280 16221 query24 8102186514251425 query24_11411137313611361 query25 584 511 462 462 query26 1297308 175 175 query27 2688601 338 338 query28 4368195019541950 query29 1000632 505 505 query30 299 237 196 196 query31 11171044919 919 query32 83 71 73 71 query33 525 336 295 295 query34 11561177652 652 query35 788 770 657 657 query36 1371140611711171 query37 146 100 85 85 query38 3201315830473047 query39 930 907 894 894 query39_1892 894 890 890 query40 239 162 133 133 query41 66 59 59 59 query42 104 104 104 104 query43 324 326 283 283 query44 query45 217 201 192 192 query46 11131215726 726 query47 2297232822002200 query48 397 404 293 293 query49 639 524 436 436 query50 712 287 225 225 query51 4284425942544254 query52 103 102 94 94 query53 245 271 208 208 query54 304 269 246 246 query55 135 89 83 83 query56 294 300 299 299 query57 1404138713291329 query58 299 259 268 259 query59 1545162614211421 query60 342 343 326 326 query61 154 156 156 156 query62 666 617 571 571 query63 252 197 197 197 query64 2338816 684 684 query65 query66 1677510 384 384 query67 30020 29962 29795 29795 query68 query69 465 334 306 306 query70 10401040977 977 query71 310 276 273 273 query72 3035280225532553 query73 820 782 447 447 query74 5064490747194719 query75 2806265023162316 query76 22751154738 738 query77 423 452 352 352 query78 12950 12946 12484 12484 query79 1465987 715 715 query80 710 600 520 520 query81 460 282 249 249 query82 1331158 128 128 query83 357 288 261 261 query84 277 145 122 122 query85 920 561 432 432 query86 384 338 312 312 query87 3421338432553255 query88 3571267926542654 query89 443 379 336 336 query90 1876181 186 181 query91 176 168 140 140 query92 81 78 72 72 query93 963 949 570 570 query94 537 338 292 292 query95 650 455 340 340 query96 1047759 363 363 query97 2698267925672567 query98 237 231 222 222 query99 11241101988 988 Total cold run time: 254971 ms Total hot run time: 171383 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected]
Re: [PR] [fix](catalog) Mask sensitive properties in show create catalog [doris]
hello-stephen commented on PR #63234: URL: https://github.com/apache/doris/pull/63234#issuecomment-4448033198 TPC-H: Total hot run time: 29628 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit c532bff04b03eb41e32784df0191177072d18868, data reload: false -- Round 1 -- orders Doris NULLNULL0 0 0 NULL0 NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 17708 389638263826 q2 q3 10701 894 629 629 q4 4669458 341 341 q5 7451132111521152 q6 209 167 137 137 q7 946 939 747 747 q8 9911139313351335 q9 6777554654365436 q10 6361207918211821 q11 477 262 252 252 q12 684 423 295 295 q13 18136 331727172717 q14 288 279 258 258 q15 q16 906 882 792 792 q17 10041032733 733 q18 6503570256125612 q19 1169119811211121 q20 496 384 259 259 q21 4605227218621862 q22 422 352 303 303 Total cold run time: 99423 ms Total hot run time: 29628 ms - Round 2, with runtime_filter_mode=off - orders Doris NULLNULL15000 42 6422171781 NULL22778155NULLNULL2023-12-26 18:27:23 2023-12-26 18:42:55 NULLutf-8 NULLNULL q1 4172407941884079 q2 q3 4638480141564156 q4 2100214013891389 q5 4962497952734979 q6 193 164 127 127 q7 2044221116651665 q8 3486317131433143 q9 8676851084028402 q10 4483447642684268 q11 622 434 389 389 q12 691 746 526 526 q13 3239362629172917 q14 312 304 288 288 q15 q16 804 780 690 690 q17 1329132414381324 q18 8023724971217121 q19 1180118711551155 q20 2299220919321932 q21 6072538749034903 q22 567 550 425 425 Total cold run time: 59892 ms Total hot run time: 53878 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](catalog) Mask sensitive properties in show create catalog [doris]
xyla commented on PR #63234: URL: https://github.com/apache/doris/pull/63234#issuecomment-4447157835 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](catalog) Mask sensitive properties in show create catalog [doris]
Copilot commented on code in PR #63234:
URL: https://github.com/apache/doris/pull/63234#discussion_r3238749284
##
fe/fe-core/src/test/java/org/apache/doris/datasource/ExternalCatalogTest.java:
##
@@ -144,6 +145,33 @@ public void testExternalCatalogAutoAnalyze() throws
Exception {
Assertions.assertTrue(catalog.enableAutoAnalyze());
}
+@Test
+public void testShowCreateCatalogMasksSensitiveProperties() throws
Exception {
+String createStmt = "create catalog mask_iceberg_rest properties(\n"
++ "\"type\" = \"iceberg\",\n"
++ "\"iceberg.catalog.type\" = \"rest\",\n"
++ "\"iceberg.rest.uri\" = \"http://localhost:8181\",\n";
++ "\"warehouse\" = \"test_db\",\n"
++ "\"iceberg.rest.security.type\" = \"oauth2\",\n"
++ "\"iceberg.rest.oauth2.credential\" =
\"super-secret-pat\",\n"
++ "\"iceberg.rest.oauth2.server-uri\" =
\"http://localhost:8181/v1/oauth/tokens\",\n";
++ "\"iceberg.rest.oauth2.scope\" =
\"session:role:TEST_ROLE\"\n"
++ ");";
+
+NereidsParser nereidsParser = new NereidsParser();
+LogicalPlan logicalPlan = nereidsParser.parseSingle(createStmt);
+if (logicalPlan instanceof CreateCatalogCommand) {
+((CreateCatalogCommand) logicalPlan).run(rootCtx, null);
+}
Review Comment:
The catalog creation is wrapped in an `instanceof` check with no `else`
branch. If parsing returns a different plan type (e.g., due to a future parser
change), the test will silently skip the creation step and then fail on
`showCreateCatalog` with a misleading error. Add an
`Assertions.assertTrue(logicalPlan instanceof CreateCatalogCommand)` (or `fail`
in the else branch) to make the failure mode explicit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](catalog) Mask sensitive properties in show create catalog [doris]
hello-stephen commented on PR #63234: URL: https://github.com/apache/doris/pull/63234#issuecomment-4446986659 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
