[ https://issues.apache.org/jira/browse/FLINK-14396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
zhijiang updated FLINK-14396: ----------------------------- Parent: FLINK-14551 Issue Type: Sub-task (was: Task) > Implement rudimentary non-blocking network output > ------------------------------------------------- > > Key: FLINK-14396 > URL: https://issues.apache.org/jira/browse/FLINK-14396 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Network > Reporter: zhijiang > Assignee: zhijiang > Priority: Minor > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Considering the mailbox model and unaligned checkpoints requirements in > future, task network output should be non-blocking. In other words, as long > as output is available, it should never block for a subsequent/future single > record write. > In the first version, we only implement the non-blocking output for the most > regular case, and do not solve the following cases which still keep the > previous behavior. > * Big record which might span multiple buffers > * Flatmap-like operators which might emit multiple records in every process > * Broadcast watermark which might request multiple buffers at a time > The solution is providing the RecordWriter#isAvailable method and respective > LocalBufferPool#isAvailable for judging the output beforehand. As long as > there is at-least one available buffer in LocalBufferPool, the RecordWriter > is available for network output in most cases. This doesn’t include runtime > handling of this non-blocking and availability behavior in > StreamInputProcessor > Note: It requires the minimum number of buffers in output LocalBufferPool > adjusting to (numberOfSubpartitions + 1) and also adjusting the monitor of > backpressure future. -- This message was sent by Atlassian Jira (v8.3.4#803005)