dwsmith1983 opened a new issue, #6605:
URL: https://github.com/apache/datafusion-comet/issues/6605
### Describe the bug
The native Parquet scan resolves Azure credentials from the Hadoop
`fs.azure.*` settings, and the goal is to pick the identity Hadoop's ABFS
driver would. After #6059 a few gaps remain where the two can disagree. None of
them comes from #6059; they were found while checking it against Hadoop 3.4.2's
`AbfsConfiguration`.
1. **Container-scoped OAuth keys.** Since Hadoop 3.4.2, `getPasswordString`
probes `<key>.<container>.<host>` before the account and global forms, and
`getTokenProvider` reads the client id, secret and endpoint, the MSI tenant,
endpoint and authority, and the token file through it. The native scan reads
that form only for the SAS fixed token. With
`fs.azure.account.oauth2.client.secret.data.myacct.dfs.core.windows.net` set
next to an account-level secret, Hadoop authenticates as the container's
principal and the native scan as the account's. With only the container-scoped
secret, the native scan fails.
2. **Fabric hosts.** `ENDPOINT_SUFFIXES` lists only the `core.windows.net`
suffixes, but `object_store` also accepts `onelake.dfs.fabric.microsoft.com`.
Account-scoped keys for a Fabric host are never found, so a configured
principal is missed and the scan falls through to ambient `AZURE_*` variables
or managed identity.
3. **Account-scoped key forms.** Hadoop's `accountConf` builds
`<key>.<host>` from the raw URL authority, port included. The native scan also
probes the other endpoint suffix and the short `<key>.<account>` form, ahead of
the global key, and drops an explicit port. A stale key Hadoop never reads can
therefore win over the global one.
4. **Credential provider stores.** `Configuration.getPassword` consults
`hadoop.security.credential.provider.path` (for example JCEKS) before plain
configuration. The native scan only sees plain configuration, so a token kept
in a store is invisible and a lower-priority plaintext value is used instead.
5. **OAuth provider class lookup.** `fs.azure.account.oauth.provider.type`
does not apply the `getTokenProviderClass` auth-type guard that the SAS
provider class now does. Hadoop fails ("Failed to initialize null") where the
native scan builds a store, so this only matters if the driver path changes.
6. **Error text.** The "Scheme of URL is not Azure" error formats the whole
URL, which would echo a password in the userinfo.
### Steps to reproduce
Unit-level: build the store with `create_store_with_env` and an empty
environment for each configuration above, and compare with what Hadoop 3.4.2
`AbfsConfiguration` resolves.
### Expected behavior
The native scan either resolves the same credential Hadoop does or fails
with an error naming the key, and never falls through to the environment when
Hadoop configured a mechanism.
### Additional context
Hadoop 3.4.1 and older have no container-scoped lookup at all, so items 1
and 3 depend on the Hadoop version on the driver.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
