[ https://issues.apache.org/jira/browse/FLINK-10158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16582241#comment-16582241 ]
zhijiang commented on FLINK-10158: ---------------------------------- I already improved it in [FLINK-9913|https://issues.apache.org/jira/browse/FLINK-9913], and there will be only one serializer for all the subpartitions, so the memories of intermediate serialization buffers are decreased a lot. > The DataOutputSerializer may consume excessive memory > ----------------------------------------------------- > > Key: FLINK-10158 > URL: https://issues.apache.org/jira/browse/FLINK-10158 > Project: Flink > Issue Type: Improvement > Components: Core > Affects Versions: 1.4.2, 1.5.2, 1.6.0 > Reporter: aitozi > Assignee: aitozi > Priority: Major > > I found that the dataOutputSerializer clear the intermediate buffer when the > buffer exceed the 5M (as a fixed configuration), But when we encountered the > rebalance or keyByPartition and the downstream has a large parallel it will > also consume a lot memory, we can do two things : > 1. make this config configurable > 2. like the https://issues.apache.org/jira/projects/FLINK/issues/FLINK-1326 > mentioned, we can make the serializer one for the output > What's your idea ? [~StephanEwen] -- This message was sent by Atlassian JIRA (v7.6.3#76005)