[ https://issues.apache.org/jira/browse/FLINK-8439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16554260#comment-16554260 ]
ASF GitHub Bot commented on FLINK-8439: --------------------------------------- GitHub user azagrebin opened a pull request: https://github.com/apache/flink/pull/6405 [FLINK-8439] Add Flink shading to AWS credential provider s3 hadoop c… ## What is the purpose of the change This PR refactors S3 Hadoop and Presto file systems and adds Flink shading to AWS credential provider config. ## Brief change log - extract `AbstractS3FileSystemFactory` base class from `s3hadoop.` and `s3presto.S3FileSystemFactory`s - extract hadoop configuration logic into `HadoopConfigLoader` with Flink shading of certain Hadoop configs - add Flink shading to AWS credential provider config of `S3FileSystemFactory`s ## Verifying this change run unit tests ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (yes) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) You can merge this pull request into a Git repository by running: $ git pull https://github.com/azagrebin/flink FLINK-8439 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/6405.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #6405 ---- commit 44c6eafb6b0757deb89f4e4a7e9bb237f7336428 Author: Andrey Zagrebin <azagrebin@...> Date: 2018-07-23T16:10:55Z [FLINK-8439] Add Flink shading to AWS credential provider s3 hadoop config ---- > Document using a custom AWS Credentials Provider with flink-3s-fs-hadoop > ------------------------------------------------------------------------ > > Key: FLINK-8439 > URL: https://issues.apache.org/jira/browse/FLINK-8439 > Project: Flink > Issue Type: Improvement > Components: Documentation > Reporter: Dyana Rose > Priority: Critical > Labels: pull-request-available > Fix For: 1.4.3, 1.5.3 > > > This came up when using the s3 for the file system backend and running under > ECS. > With no credentials in the container, hadoop-aws will default to EC2 instance > level credentials when accessing S3. However when running under ECS, you will > generally want to default to the task definition's IAM role. > In this case you need to set the hadoop property > {code:java} > fs.s3a.aws.credentials.provider{code} > to a fully qualified class name(s). see [hadoop-aws > docs|https://github.com/apache/hadoop/blob/1ba491ff907fc5d2618add980734a3534e2be098/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/index.md] > This works as expected when you add this setting to flink-conf.yaml but there > is a further 'gotcha.' Because the AWS sdk is shaded, the actual full class > name for, in this case, the ContainerCredentialsProvider is > {code:java} > org.apache.flink.fs.s3hadoop.shaded.com.amazonaws.auth.ContainerCredentialsProvider{code} > > meaning the full setting is: > {code:java} > fs.s3a.aws.credentials.provider: > org.apache.flink.fs.s3hadoop.shaded.com.amazonaws.auth.ContainerCredentialsProvider{code} > If you instead set it to the unshaded class name you will see a very > confusing error stating that the ContainerCredentialsProvider doesn't > implement AWSCredentialsProvider (which it most certainly does.) > Adding this information (how to specify alternate Credential Providers, and > the name space gotcha) to the [AWS deployment > docs|https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/deployment/aws.html] > would be useful to anyone else using S3. -- This message was sent by Atlassian JIRA (v7.6.3#76005)