krutoileshii opened a new issue, #10392: URL: https://github.com/apache/seatunnel/issues/10392
### Search before asking - [x] I had searched in the [feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement. ### Description My company currently utilizes starburst-catalog (glue compatible). When using the catalog, one must supply it an access-key-id and secret-acces-key. However, the AWS SDK 2.** doesn't have a factory to read them. so there is no existing client.credentials-provider to supply those. I've ran into the same issue with Flink (1.20.3) and had to create a custom wrapper below to be able to pass those creds. Would it be possible to add a similar wrapper for iceberg sink? Once can currently set the following: ``` sink { Iceberg { catalog_name = "seatunnel_test" iceberg.catalog.config = { warehouse = "s3://your-bucket/warehouse/" catalog-impl = "org.apache.iceberg.aws.glue.GlueCatalog" io-impl = "org.apache.iceberg.aws.s3.S3FileIO" client.region = "your-region", // THESE ARE CURRENTLY AVAILABLE glue.endpoint = 'YOUR_GLUE_ENDPOINT', glue.id = "GLUE_CATALOG_ID" (starburst-glue) in my case. , client.credentials-provider =YOUR_PROVIDER, Client Credential Providers that could currently be used: 1. software.amazon.awssdk.auth.credentials.SystemPropertyCredentialsProvider 2. software.amazon.awssdk.auth.credentials.internal.SystemSettingsCredentialsProvider 3. software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider 4. custom_wrapper_around_cred_provider // THESE ARE NOT AVAILABLE client.credentials-provider.access-key-id = 'YOUR_ACCESS_KEY', client.credentials-provider.secret-access-key = 'YOUR_SECRET_ACCESS_KEY' } namespace = "seatunnel_namespace" table = "iceberg_sink_table" iceberg.table.write-props = { write.format.default = "parquet" write.target-file-size-bytes = 536870912 } iceberg.table.primary-keys = "id" iceberg.table.partition-keys = "f_datetime" iceberg.table.upsert-mode-enabled = true iceberg.table.schema-evolution-enabled = true case_sensitive = true } } ``` CLASS: ```java package org.allosource.auth; import software.amazon.awssdk.auth.credentials.AwsBasicCredentials; import software.amazon.awssdk.auth.credentials.AwsCredentials; import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider; import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider; import java.util.Map; public class GlueStaticCredentialsWrapper implements AwsCredentialsProvider { private final StaticCredentialsProvider delegate; // Iceberg's factory looks specifically for this method signature public static GlueStaticCredentialsWrapper create(Map<String, String> properties) { return new GlueStaticCredentialsWrapper(properties); } private GlueStaticCredentialsWrapper(Map<String, String> properties) { // Properties passed here have the prefix "client.credentials-provider." removed String accessKey = properties.get("access-key-id"); String secretKey = properties.get("secret-access-key"); if (accessKey == null || secretKey == null) { throw new IllegalArgumentException("Missing credentials. Use 'client.credentials-provider.access-key-id' " + "and 'client.credentials-provider.secret-access-key' in your catalog configuration."); } this.delegate = StaticCredentialsProvider.create( AwsBasicCredentials.create(accessKey.trim(), secretKey.trim()) ); } @Override public AwsCredentials resolveCredentials() { return delegate.resolveCredentials(); } } ``` POM ``` <?xml version="1.0" encoding="UTF-8"?> <project xmlns="http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> <modelVersion>4.0.0</modelVersion> <groupId>org.allosource.auth</groupId> <artifactId>glue-auth-wrapper</artifactId> <version>1.0-SNAPSHOT</version> <properties> <maven.compiler.source>11</maven.compiler.source> <maven.compiler.target>11</maven.compiler.target> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <aws.sdk.version>2.33.0</aws.sdk.version> </properties> <dependencies> <!-- The primary interface for your credentials wrapper --> <dependency> <groupId>software.amazon.awssdk</groupId> <artifactId>auth</artifactId> <version>${aws.sdk.version}</version> <scope>provided</scope> </dependency> <dependency> <groupId>software.amazon.awssdk</groupId> <artifactId>identity-spi</artifactId> <version>${aws.sdk.version}</version> <scope>provided</scope> </dependency> </dependencies> <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <version>3.11.0</version> <configuration> <release>11</release> </configuration> </plugin> </plugins> </build> </project> ``` ### Usage Scenario Use this to configure access and secret for Glue catalog when using custom implementations. ### Related issues _No response_ ### Are you willing to submit a PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
