parthchandra commented on issue #2341: URL: https://github.com/apache/datafusion-comet/issues/2341#issuecomment-3628188735
The AWS SDKs (both V1 and V2) provide a basic set of authentication providers which are used by the (Java) S3Client to authenticate a request for any AWS resource. (https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials-explicit.html). `Hadoop-aws` extends these to provide a Hadoop specific method of accessing S3 (https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#Authenticating_with_S3) which allows Hadoop applications to access S3 via the HDFS client API (as opposed to the S3 client API). Almost all Java and Hadoop based engines use the HDFS client APIs (via Hadoop-aws) to access S3. Spark, for instance, uses the HDFS APIs. Some implementations like Iceberg (java) can use both methods to access S3 (and allow users to choose which API should be used). The Hadoop client APIs also allow end users to specify their own credentials providers. An example use case is a credentials provider that will first authenticate a user against a proprietary authentication system, apply any organization specific access control, and then interface with AWS to get AWS credentials. The application then uses these credentials with the HDFS client which in turn uses them with the AWS SDK. Some credentials providers will also implement credentials caching and credentials refreshing which is useful for long running jobs that are typical of batch workloads in Spark. Almost all such custom credentials providers are written in Java, and because they are organization specific, cannot be implemented by us (Comet/OpenDAL) in native code. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
