Martijn Visser created FLINK-41575:
--------------------------------------
Summary: Checkpoint file merging fails on flink-s3-fs-native
Key: FLINK-41575
URL: https://issues.apache.org/jira/browse/FLINK-41575
Project: Flink
Issue Type: Bug
Components: Connectors / FileSystem, Runtime / Checkpointing
Affects Versions: 2.3.0, 2.4.0
Reporter: Martijn Visser
With execution.checkpointing.file-merging.enabled: true and checkpoints on
flink-s3-fs-native, the first checkpoint fails with java.io.IOException: Stream
is closed at NativeS3OutputStream.write(NativeS3OutputStream.java:124) from
FileMergingCheckpointStateOutputStream.flushToFile, and the job fails. File
merging calls sync() after every logical file and keeps writing to the same
physical file (returnPhysicalFileForNextReuse;
shouldSyncAfterClosingLogicalFile always returns true).
NativeS3OutputStream#sync() uploads the object and seals the stream. Reproduced
on SeaweedFS with RocksDB incremental checkpoints: native S3 without file
merging and file:// with file merging both complete their checkpoints. ForSt
with a remote primary dir also syncs and keeps writing
(ByteBufferWritableFSDataOutputStream#sync), so it likely fails the same way.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)