Apparently Flink 1.14.0 doesn't correctly translate S3 options when they
are set programmatically. I'm creating a local environment like this to
connect to local MinIO instance:
val flinkConf = new Configuration()
flinkConf.setString("s3.endpoint", "http://127.0.0.1:9000";)
flinkConf.setString("s3.aws.credentials.provider",
"org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider")
val env =
StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(flinkConf)
Then StreamingFileSink fails with a huge stack trace with most relevant
messages being Caused by:
org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials
provided by SimpleAWSCredentialsProvider
EnvironmentVariableCredentialsProvider InstanceProfileCredentialsProvider :
com.amazonaws.SdkClientException: Failed to connect to service endpoint: which
means that Hadoop tried to enumerate all of the credential providers
instead of using the one set in configuration. What am I doing wrong?
