[
https://issues.apache.org/jira/browse/FLINK-17820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17112651#comment-17112651
]
Roman Khachatryan commented on FLINK-17820:
-------------------------------------------
The data to FsCheckpointStateOutputStream is written through DataOutputStream.
Without flushing DataOutputStream some data can be left in its buffer. Flushing
DataOutputStream also flushes the underlying FsCheckpointStateOutputStream.
Although in current JDK DataOutputStream doesn't buffer data, I think it's
risky to rely on it.
> Memory threshold is ignored for channel state
> ---------------------------------------------
>
> Key: FLINK-17820
> URL: https://issues.apache.org/jira/browse/FLINK-17820
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Runtime / Task
> Affects Versions: 1.11.0
> Reporter: Roman Khachatryan
> Assignee: Roman Khachatryan
> Priority: Critical
> Labels: pull-request-available
> Fix For: 1.11.0
>
>
> Config parameter state.backend.fs.memory-threshold is ignored for channel
> state. Causing each subtask to have a file per checkpoint. Regardless of the
> size of channel state (of this subtask).
> This also causes slow cleanup and delays the next checkpoint.
>
> The problem is that {{ChannelStateCheckpointWriter.finishWriteAndResult}}
> calls flush(); which actually flushes the data on disk.
>
> From FSDataOutputStream.flush Javadoc:
> A completed flush does not mean that the data is necessarily persistent. Data
> persistence can is only assumed after calls to close() or sync().
>
> Possible solutions:
> 1. not to flush in {{ChannelStateCheckpointWriter.finishWriteAndResult (which
> can lead to data loss in a wrapping stream).}}
> {{2. change }}{{FsCheckpointStateOutputStream.flush behavior}}
> {{3. wrap }}{{FsCheckpointStateOutputStream to prevent flush}}{{}}{{}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
