zhangfengcdt commented on code in PR #7363:
URL: https://github.com/apache/datafusion-comet/pull/7363#discussion_r4634510632


##########
spark/src/main/scala/org/apache/comet/serde/operator/CometIcebergNativeScan.scala:
##########
@@ -607,6 +608,94 @@ object CometIcebergNativeScan extends 
CometOperatorSerde[CometBatchScanExec] wit
     }
   }
 
+  /** Activation key for the `CometS3CredentialProvider` SPI on the native 
Iceberg path. */
+  private[comet] val IcebergCredentialProviderClassKey = 
"s3.comet.credential.provider.class"
+
+  /** The provider that refreshes REST-vended credentials (Spark 4.x builds 
only). */
+  private[comet] val VendedCredentialProviderClass =
+    "org.apache.comet.cloud.s3.IcebergVendedS3CredentialProvider"
+
+  // Iceberg's AwsClientProperties.REFRESH_CREDENTIALS_ENDPOINT and 
REFRESH_CREDENTIALS_ENABLED.
+  private val RefreshCredentialsEndpointKey = 
"client.refresh-credentials-endpoint"
+  private val RefreshCredentialsEnabledKey = 
"client.refresh-credentials-enabled"
+
+  // Iceberg's CatalogProperties.URI, which a relative refresh endpoint 
resolves against.
+  private val CatalogUriKey = "uri"
+
+  /**
+   * Whether the provider can initialize in this JVM. It ships in Comet's jar, 
so it and the AWS
+   * SDK interface it implements come from Comet's loader. It builds its 
delegate through
+   * `AwsClientProperties` from the context loader and casts the result to 
that interface, so both
+   * loaders must see one copy of the SDK. A provider that fails to initialize 
falls back to the
+   * default chain on every task, so the check runs once here instead.
+   */
+  private lazy val vendedCredentialProviderAvailable: Boolean = {
+    val cometLoader = getClass.getClassLoader
+    val usable =
+      vendedCredentialProviderUsable(cometLoader, 
ClassLoaders.contextOrDefault(cometLoader))
+    if (!usable) {
+      logInfo(
+        s"$VendedCredentialProviderClass is not usable in this JVM, so native 
Iceberg scans " +
+          "keep the credentials vended at table load and are not refreshed")
+    }
+    usable
+  }
+
+  /**
+   * The check behind [[vendedCredentialProviderAvailable]], for the loaders 
Comet and Iceberg
+   * come from. It fails when Comet's loader can't see the AWS SDK, such as 
when the only copy is
+   * on Spark's user-jar loader, and when `AwsClientProperties` links against 
another copy of it.
+   */
+  private[comet] def vendedCredentialProviderUsable(
+      cometLoader: ClassLoader,
+      contextLoader: ClassLoader): Boolean =
+    // scalastyle:off classforname
+    try {
+      Class.forName(VendedCredentialProviderClass, false, cometLoader)
+      val sdkProvider = Class.forName(AwsCredentialsProviderClass, false, 
cometLoader)
+      val clientProperties = Class.forName(AwsClientPropertiesClass, false, 
contextLoader)
+      Class.forName(AwsCredentialsProviderClass, false, 
clientProperties.getClassLoader) eq
+        sdkProvider
+    } catch {
+      // A LinkageError, such as part of the SDK missing, means the same as a 
missing class here.
+      case _: ClassNotFoundException | _: LinkageError => false
+    }
+  // scalastyle:on classforname
+
+  private val AwsCredentialsProviderClass =
+    "software.amazon.awssdk.auth.credentials.AwsCredentialsProvider"
+  private val AwsClientPropertiesClass = 
"org.apache.iceberg.aws.AwsClientProperties"
+
+  /**
+   * Names [[VendedCredentialProviderClass]] as the credential provider when 
the catalog returned
+   * a refresh endpoint and the user named no provider.
+   *
+   * Without a provider, the native FileIO signs with the `s3.*` credentials 
vended when the
+   * driver loaded the table, and a scan that outlives them fails with 
`ExpiredToken`. Iceberg's
+   * `S3FileIO` refreshes them through the endpoint instead. An explicit 
provider key, including
+   * an empty one, is left alone so it stays an opt-out. Builds without the 
provider class (Spark
+   * 3.x) keep the vended credentials as they are, and so does a catalog with 
no `uri`: Iceberg's
+   * `VendedCredentialsProvider` needs it to reach the catalog whether the 
endpoint is relative or
+   * absolute.
+   */
+  def withVendedCredentialProvider(properties: Map[String, String]): 
Map[String, String] =
+    withVendedCredentialProvider(properties, vendedCredentialProviderAvailable)
+
+  private[comet] def withVendedCredentialProvider(
+      properties: Map[String, String],
+      providerAvailable: Boolean): Map[String, String] = {
+    val hasEndpoint = 
properties.get(RefreshCredentialsEndpointKey).exists(_.trim.nonEmpty)

Review Comment:
   I think Comet treats client.refresh-credentials-enabled=" true " as enabled 
because it only checks for false. Iceberg 1.10 and 1.11 parse the raw value 
with Boolean.parseBoolean, so they treat it as disabled. 
   
   I think the delegate uses static credentials will never refreshes with the 
new endpoint. This does not seem to be expected.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to