Yun Tang created FLINK-25429: -------------------------------- Summary: Avoid to close output streams twice during uploading changelogs Key: FLINK-25429 URL: https://issues.apache.org/jira/browse/FLINK-25429 Project: Flink Issue Type: Improvement Components: Runtime / State Backends Reporter: Yun Tang Assignee: Yun Tang
Current uploader implementation would close {{stream}} and {{fsStream}} one by one, which lead to {{fsStream}} closed twice. {code:java} try (FSDataOutputStream fsStream = fileSystem.create(path, NO_OVERWRITE)) { fsStream.write(compression ? 1 : 0); try (OutputStreamWithPos stream = wrap(fsStream); ) { final Map<UploadTask, Map<StateChangeSet, Long>> tasksOffsets = new HashMap<>(); for (UploadTask task : tasks) { tasksOffsets.put(task, format.write(stream, task.changeSets)); } FileStateHandle handle = new FileStateHandle(path, stream.getPos()); // WARN: streams have to be closed before returning the results // otherwise JM may receive invalid handles return new LocalResult(tasksOffsets, handle); } } {code} Not all file system supports to close same stream twice. -- This message was sent by Atlassian Jira (v8.20.1#820001)