[ https://issues.apache.org/jira/browse/FLINK-10995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Piotr Nowojski closed FLINK-10995. ---------------------------------- > Copy intermediate serialization results only once for broadcast mode > -------------------------------------------------------------------- > > Key: FLINK-10995 > URL: https://issues.apache.org/jira/browse/FLINK-10995 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Task > Affects Versions: 1.8.0, 1.9.0 > Reporter: zhijiang > Assignee: zhijiang > Priority: Minor > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 40m > Remaining Estimate: 0h > > The emitted records from operator would be firstly serialized into > intermediate bytes array in {{RecordSerializer}}, then copy the intermediate > results into target buffers for different sub partitions. For broadcast > mode, the same intermediate results would be copied as many times as the > number of sub partitions, and this would affect the performance seriously in > large scale jobs. > We can copy to only one target buffer which would be shared by all the sub > partitions to reduce the overheads. For emitting latency marker in broadcast > mode, we should flush the previous shared target buffers first, and then > request a new buffer for the target sub partition to send latency marker. -- This message was sent by Atlassian Jira (v8.3.4#803005)