AwasthiSomesh commented on issue #10078:
URL: https://github.com/apache/iceberg/issues/10078#issuecomment-2352331487
@andythsu @clamar14
**Please try with aws custom credential provider follow the below approach
if needed, i resolved this issue using below implementation**
**********************************************
===============
val spark = SparkSession.builder().master("local[*]")
// .config("spark.sql.defaultCatalog", "glue_catalog")
.config("spark.sql.extensions",
"org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
.config("spark.sql.catalog.glue_catalog",
"org.apache.iceberg.spark.SparkCatalog")
.config("spark.sql.catalog.glue_catalog.warehouse",
"s3://bucket/test/icebergsorted/otfdb/")
.config("spark.sql.catalog.glue_catalog.catalog-impl",
"org.apache.iceberg.aws.glue.GlueCatalog")
.config("spark.sql.catalog.glue_catalog.io-impl",
"org.apache.iceberg.aws.s3.S3FileIO")
.config("spark.sql.catalog.glue_catalog.client.credentials-provider",
"example.CustomCredentialProvider")
.config("spark.sql.catalog.glue_catalog.client.region", "us-east-1")
.config("spark.sql.catalog.glue_catalog.client.credentials-provider.accessKeyId",
"<accesskey>")
.config("spark.sql.catalog.glue_catalog.client.credentials-provider.secretAccessKey",
"<secretkey>")
.getOrCreate()
===============
**Please note the CustomCredentialProvider package mentioned above, it was
extended as below,**
=============
class CustomCredentialProvider extends AwsCredentialsProvider {
private var credentials: AwsCredentials = null
def this(keys: util.Map[String, String]) {
this()
credentials = AwsBasicCredentials.create(keys.get("accessKeyId"),
keys.get("secretAccessKey"))
}
override def resolveCredentials: AwsCredentials = this.credentials
}
object CustomCredentialProvider{
/**
Create a credentials provider that always returns the provided set of
credentials.
*/
def create(keys: util.Map[String, String]): CustomCredentialProvider = {
new CustomCredentialProvider(keys)
}
}
================
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
