Pavel Penkov created FLINK-24595:
------------------------------------
Summary: Programmatic configuration of S3 doesn't pass parameters
to Hadoop FS
Key: FLINK-24595
URL: https://issues.apache.org/jira/browse/FLINK-24595
Project: Flink
Issue Type: Bug
Components: Connectors / Hadoop Compatibility
Affects Versions: 1.14.0
Environment: Flink 1.14.0
JDK 8
{{openjdk version "1.8.0_302"}}
{{OpenJDK Runtime Environment (Zulu 8.56.0.23-CA-macos-aarch64) (build
1.8.0_302-b08)}}
{{OpenJDK 64-Bit Server VM (Zulu 8.56.0.23-CA-macos-aarch64) (build 25.302-b08,
mixed mode)}}
Reporter: Pavel Penkov
Attachments: FlinkApp.java, TickingSource.java, flink_exception.txt
When running in mini-cluster mode Flink apparently doesn't pass S3
configuration to underlying Hadoop FS. With a code like this
{code:java}
Configuration conf = new Configuration();
conf.setString("s3.endpoint", "http://localhost:4566";);
conf.setString("s3.aws.credentials.provider","org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider");
conf.setString("s3.access.key", "harvester");
conf.setString("s3.secret.key", "harvester");
StreamExecutionEnvironment env =
StreamExecutionEnvironment.createLocalEnvironment(conf);
{code}
Application fails with an exception with most relevant error being {{Caused by:
org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials
provided by SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider
InstanceProfileCredentialsProvider : com.amazonaws.SdkClientException: Failed
to connect to service endpoint: }}
So Hadoop lists all the providers but it should use only the one set in
configuration. Full project that reproduces this behaviour is available at
[https://github.com/PavelPenkov/flink-s3-conf] and relevant files are attached
to this issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
