dwsmith1983 commented on code in PR #7221:
URL: https://github.com/apache/datafusion-comet/pull/7221#discussion_r4371404075
##########
spark/src/main/scala/org/apache/comet/objectstore/AbfsAuthResolver.scala:
##########
@@ -467,6 +469,38 @@ private[comet] object AbfsAuthResolver extends Logging {
Markers.ERROR_MESSAGE -> message)
}
+ /**
+ * Why the native store cannot read `uri` under `outcome`, or None when it
can: a mechanism
+ * Hadoop named that runs Java code per request, a built-in OAuth provider
with no
+ * `object_store` counterpart, or a resolution that failed. All are known on
the driver, so the
+ * scan falls back to Spark's own reader, which authenticates through
hadoop-azure, instead of
+ * planning natively and failing every task.
+ */
+ def nativeFallbackReason(uri: URI, outcome: Outcome): Option[String] = {
+ // The container and host only: a password in the user info stays out of
the reason.
+ val authority = Option(uri.getRawAuthority).getOrElse("")
+ val at = authority.lastIndexOf('@')
+ val location = s"${uri.getScheme}://" +
+ (if (at < 0) authority
+ else authority.substring(0, at).takeWhile(_ != ':') +
authority.substring(at))
+ def unsupported(mechanism: String): String =
+ s"Native scan does not support ABFS authentication $mechanism for
$location"
+ outcome match {
+ case Declined(authType, providerClass, assertionProviderClass) =>
+ val classes = providerClass.map(c => s" with provider
$c").getOrElse("") +
+ assertionProviderClass.map(c => s" and client assertion provider
$c").getOrElse("")
+ Some(unsupported(s"auth type $authType$classes"))
+ case Resolved(
+ AuthTypes.OAUTH,
+ Some(c @ (Providers.REFRESH_TOKEN | Providers.USER_PASSWORD)),
+ _) =>
+ Some(unsupported(s"auth type ${AuthTypes.OAUTH} with provider $c"))
+ case Failed(_, message) =>
Review Comment:
`location` leaves out the password, but `message` goes in as is. Three
`Failed` messages in this file are built from the full URI: `$uri has invalid
authority.` and `Invalid URI '$uri' has a malformed authority...` in
`authorityParts`, and `Invalid URI $uri - account name is not fully qualified.`
in `sharedKey`. For `abfss://data:pw@accta/x.parquet` with a key set for
`accta`, `sharedKey` returns that last message, and the reason reads `... for
abfss://data@accta: Invalid URI abfss://data:pw@accta/x.parquet - account name
is not fully qualified.` Hadoop reads `data:pw` as the container, so that URL
doesn't read for Spark either, but the password still lands in text people
share. Before this change the same text was in the task error. Now it is also
in the extended explain output and in the fallback warning when
`spark.comet.explain.fallback.log.enabled` is on.
The password test at line 785 uses `Custom`, so it takes the `Declined` arm
and doesn't reach these messages. Could the `Failed` arm strip the user info as
`location` does, or the three messages use the same form? A test with a
SharedKey key, an account name without a dot and a password in the URL would
cover it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
