Fred Teunissen created FLINK-37749:
--------------------------------------
Summary: FileSystem Connector fails writing to S3 bucket
Key: FLINK-37749
URL: https://issues.apache.org/jira/browse/FLINK-37749
Project: Flink
Issue Type: Bug
Components: Connectors / FileSystem
Affects Versions: 1.20.1, 1.19.2, 1.20.0
Environment: Docker running:
- Flink 1.20.1
- minio
Reporter: Fred Teunissen
Attachments: filesystem-connector-s3-issue.tgz
Writing to an output file on S3 using the filesystem connector results in a
crash.
The output file is defined as follows.
{code:java}
CREATE TABLE <tableName>> (
bool BOOLEAN,
`someInt` INT
)
PARTITIONED BY (someInt)
WITH (
'connector' = 'filesystem',
'path' = 's3a://tdata',
'format' = 'csv',
-- 'format' = 'parquet',
'partition.default-name' = 'null'
);{code}
Note that the 'path' refers to an existing bucket that is hosted using Minio.
The job starts OK but as soon as the first checkpoint is being triggered the
job crashes with the following exception:
{code:java}
"Caused by: org.apache.flink.util.SerializedThrowable: java.io.IOException:
Stream closed." (full details in the attached zip file).{code}
The issue can be reproduced by unzipping the attached zip file that contains a
docker-compose file to spin up the Flink cluster and Minio container
and using the instructions in the README.MD to generate test data resulting in
the job crash.
Cause is unknown but I suspect more (or other) .jars need to be added to the
plugins/s3 directory as documented here:
https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/plugins/
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
