[ https://issues.apache.org/jira/browse/FLINK-15140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
zhijiang reassigned FLINK-15140: -------------------------------- Assignee: Yingjie Cao > Shuffle data compression does not work with BroadcastRecordWriter. > ------------------------------------------------------------------ > > Key: FLINK-15140 > URL: https://issues.apache.org/jira/browse/FLINK-15140 > Project: Flink > Issue Type: Bug > Components: Runtime / Network > Affects Versions: 1.10.0 > Reporter: Yingjie Cao > Assignee: Yingjie Cao > Priority: Blocker > Labels: pull-request-available > Fix For: 1.10.0 > > Time Spent: 10m > Remaining Estimate: 0h > > I tested the newest code of master branch last weekend with more test cases. > Unfortunately, several problems were encountered, including a bug of > compression. > When BroadcastRecordWriter is used, for pipelined mode, because the > compressor copies the data back to the input buffer, however, the underlying > buffer is shared when BroadcastRecordWriter is used. So we can not copy the > compressed buffer back to the input buffer if the underlying buffer is > shared. For blocking mode, we wrongly recycle the buffer when buffer is not > compressed, and the problem is also triggered when BroadcastRecordWriter is > used. > To fix the problem, for blocking shuffle, the reference counter should be > maintained correctly, for pipelined shuffle, the simplest way maybe disable > compression when the underlying buffer is shared. I will open a PR to fix the > problem. -- This message was sent by Atlassian Jira (v8.3.4#803005)