[
https://issues.apache.org/jira/browse/FLINK-31095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17691810#comment-17691810
]
Sylvia Lin edited comment on FLINK-31095 at 2/21/23 10:09 PM:
--------------------------------------------------------------
Thanks for reply! [~martijnvisser] [~tzulitai]
Based on the hadoop doc you shared above, I checked here, seems we do have to
configure something:
[https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/s3/#hadooppresto-s3-file-systems-plugins]
But i didn't find the related s3 hadoop Flink config list here:
[https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/]
Any other place I can take a look? Or it follows certain convention?
was (Author: JIRAUSER292782):
Thanks for reply! [~martijnvisser] [~tzulitai]
I checked here, seems we do have to configure something:
[https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/s3/#hadooppresto-s3-file-systems-plugins]
But i didn't find the related s3 hadoop Flink config list here:
[https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/]
Any other place I can take a look? Or it follows certain convention?
> FileSink doesn't work with s3a on EKS
> -------------------------------------
>
> Key: FLINK-31095
> URL: https://issues.apache.org/jira/browse/FLINK-31095
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
> Affects Versions: 1.16.1
> Reporter: Sylvia Lin
> Priority: Major
>
> FileSink gives below exception on AWS EKS cluster:
> {code:java}
> Caused by: java.lang.UnsupportedOperationException: This s3 file system
> implementation does not support recoverable writers.
> at
> org.apache.flink.fs.s3.common.FlinkS3FileSystem.createRecoverableWriter(FlinkS3FileSystem.java:136)
> ~[?:?]
> at
> org.apache.flink.core.fs.PluginFileSystemFactory$ClassLoaderFixingFileSystem.createRecoverableWriter(PluginFileSystemFactory.java:134)
> ~[flink-dist-1.16.1.jar:1.16.1]
> at
> org.apache.flink.connector.file.sink.FileSink$RowFormatBuilder.createBucketWriter(FileSink.java:475)
> ~[flink-connector-files-1.16.1.jar:1.16.1]
> at
> org.apache.flink.connector.file.sink.FileSink$RowFormatBuilder.getCommittableSerializer(FileSink.java:466)
> ~[flink-connector-files-1.16.1.jar:1.16.1]
> at
> org.apache.flink.connector.file.sink.FileSink.getCommittableSerializer(FileSink.java:175)
> ~[flink-connector-files-1.16.1.jar:1.16.1]{code}
> [https://github.com/apache/flink/blob/278dc7b793303d228f7816585054629708983af6/flink-filesystems/flink-s3-fs-base/src/main/java/org/apache/flink/fs/s3/common/FlinkS3FileSystem.java#LL136C16-L136C16]
> And this may be related to
> https://issues.apache.org/jira/browse/FLINK-23487?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
