dwsmith1983 commented on code in PR #6059:
URL: https://github.com/apache/datafusion-comet/pull/6059#discussion_r4104565515
##########
native/core/src/parquet/objectstore/azure.rs:
##########
@@ -79,16 +124,167 @@ const HADOOP_MSI_ENDPOINT: &str =
"fs.azure.account.oauth2.msi.endpoint";
const HADOOP_MSI_AUTHORITY: &str = "fs.azure.account.oauth2.msi.authority";
const HADOOP_WI_TOKEN_FILE: &str = "fs.azure.account.oauth2.token.file";
const HADOOP_SAS_PREFIX: &str = "fs.azure.sas.";
+const HADOOP_SAS_FIXED_TOKEN: &str = "fs.azure.sas.fixed.token";
+const HADOOP_OAUTH_PROVIDER_TYPE: &str =
"fs.azure.account.oauth.provider.type";
+/// Simple class names of the `org.apache.hadoop.fs.azurebfs.oauth2` token
providers the
+/// native scan can satisfy.
+const HADOOP_MSI_PROVIDER_CLASS: &str = "MsiTokenProvider";
+const HADOOP_WI_PROVIDER_CLASS: &str = "WorkloadIdentityTokenProvider";
+const HADOOP_CLIENT_CREDS_PROVIDER_CLASS: &str = "ClientCredsTokenProvider";
+/// Keys Hadoop reads for `MsiTokenProvider` in a way that accepts an empty
string, which
+/// is how a system-assigned identity is configured. A blank value is absent,
not an error.
+const HADOOP_MSI_OPTIONAL_KEYS: &[&str] = &[HADOOP_OAUTH_CLIENT_ID,
HADOOP_MSI_TENANT];
+const HADOOP_AUTH_TYPE: &str = "fs.azure.account.auth.type";
+/// Hadoop credential keys, the `AzureConfigKey` each translates to and the
mechanism each
+/// belongs to.
+const HADOOP_CREDENTIAL_MAPPINGS: &[(&str, AzureConfigKey, AuthMechanism)] = &[
+ (
+ HADOOP_KEY,
+ AzureConfigKey::AccessKey,
+ AuthMechanism::SharedKey,
+ ),
+ (
+ HADOOP_OAUTH_CLIENT_ID,
+ AzureConfigKey::ClientId,
+ AuthMechanism::OAuth,
+ ),
+ (
+ HADOOP_OAUTH_CLIENT_SECRET,
+ AzureConfigKey::ClientSecret,
+ AuthMechanism::OAuth,
+ ),
+ (
+ HADOOP_MSI_TENANT,
+ AzureConfigKey::AuthorityId,
+ AuthMechanism::OAuth,
+ ),
+ (
+ HADOOP_MSI_ENDPOINT,
+ AzureConfigKey::MsiEndpoint,
+ AuthMechanism::OAuth,
+ ),
+ (
+ HADOOP_MSI_AUTHORITY,
+ AzureConfigKey::AuthorityHost,
+ AuthMechanism::OAuth,
+ ),
+ (
+ HADOOP_WI_TOKEN_FILE,
+ AzureConfigKey::FederatedTokenFile,
+ AuthMechanism::OAuth,
+ ),
+];
+/// Hadoop keys that each select an auth mechanism with no native counterpart,
and the
+/// mechanism each belongs to.
+const HADOOP_UNSUPPORTED_MECHANISM_KEYS: &[(&str, AuthMechanism)] = &[
+ ("fs.azure.sas.token.provider.type", AuthMechanism::Sas),
+ ("fs.azure.account.keyprovider", AuthMechanism::SharedKey),
Review Comment:
Done.
`fs.azure.account.keyprovider=org.apache.hadoop.fs.azurebfs.services.SimpleKeyProvider`
is now treated the same as leaving the setting out, since
`getStorageAccountKey` falls back to that class and it reads
`fs.azure.account.key`. With the account key present it builds as `SharedKey`,
with or without an explicit auth type and with the provider set globally or per
account. Without the key it is an error that names `fs.azure.account.key`,
because Hadoop's `SimpleKeyProvider` fails there too and the scan must not fall
through to another credential.
The class name must match exactly. Hadoop passes the value untrimmed to
`Configuration.getClassByName`, so a padded name is a class it cannot load and
is rejected here as a custom provider. Any other class, including
`ShellDecryptionKeyProvider`, a bare `SimpleKeyProvider` or a lookalike in
another package, is still rejected. The user guide row for
`fs.azure.account.keyprovider` now says this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
