Hi team, I'm trying to read and write from and to S3 using a custom AWS
Credential Provider using Flink v1.9.2 on YARN.
I followed the instructions to create a plugins directory in our Flink
distribution location and copy the FS implementation (I'm using s3-fs-hadoop)
package into it. I have also placed the package that contains our custom
CredentialsProvider implementation in that same directory as well.
$ ls /flink-1.9.2/plugins/s3-fs-hadoop/
total 20664
14469 Jun 17 10:57 aws-hadoop-utils-0.0.9.jar <-- contains our custom
CredentialsProvider class
21141329 Jul 28 15:43 flink-s3-fs-hadoop-1.9.2.jar
I've placed this directory in the java classpath when running the Flink
application. I have added the 'fs.s3a.assumed.role.credentials.provider' and
'fs.s3a.assumed.role.arn' to our flink-conf.yaml as well. When trying to run a
basic app that reads a file, I get the following exception:
Caused by: java.io.IOException: Class class
com.gs.ep.da.lake.aws.CustomAwsCredentialProvider does not implement
AWSCredentialsProvider
at
org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProvider(S3AUtils.java:400)
at
org.apache.hadoop.fs.s3a.S3AUtils.createAWSCredentialProviderSet(S3AUtils.java:367)
at
org.apache.hadoop.fs.s3a.S3ClientFactory$DefaultS3ClientFactory.createS3Client(S3ClientFactory.java:73)
Have I missed a step here? Do I need to make the packages also available in my
YARN classpath as well? I saw some discussion that suggest that there were some
related problems around this that were resolved in v1.10 [1][2][3].
[1] https://issues.apache.org/jira/browse/FLINK-14574
[2] https://issues.apache.org/jira/browse/FLINK-13044
[3] https://issues.apache.org/jira/browse/FLINK-11956
Best,
Andreas
________________________________
Your Personal Data: We may collect and process information about you that may
be subject to data protection laws. For more information about how we use and
disclose your personal data, how we protect your information, our legal basis
to use your information, your rights and who you can contact, please refer to:
www.gs.com/privacy-notices<http://www.gs.com/privacy-notices>
