Martijn Visser created FLINK-41575:
--------------------------------------

             Summary: Checkpoint file merging fails on flink-s3-fs-native
                 Key: FLINK-41575
                 URL: https://issues.apache.org/jira/browse/FLINK-41575
             Project: Flink
          Issue Type: Bug
          Components: Connectors / FileSystem, Runtime / Checkpointing
    Affects Versions: 2.3.0, 2.4.0
            Reporter: Martijn Visser


With execution.checkpointing.file-merging.enabled: true and checkpoints on 
flink-s3-fs-native, the first checkpoint fails with java.io.IOException: Stream 
is closed at NativeS3OutputStream.write(NativeS3OutputStream.java:124) from 
FileMergingCheckpointStateOutputStream.flushToFile, and the job fails. File 
merging calls sync() after every logical file and keeps writing to the same 
physical file (returnPhysicalFileForNextReuse; 
shouldSyncAfterClosingLogicalFile always returns true). 
NativeS3OutputStream#sync() uploads the object and seals the stream. Reproduced 
on SeaweedFS with RocksDB incremental checkpoints: native S3 without file 
merging and file:// with file merging both complete their checkpoints. ForSt 
with a remote primary dir also syncs and keeps writing 
(ByteBufferWritableFSDataOutputStream#sync), so it likely fails the same way.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to