[
https://issues.apache.org/jira/browse/FLINK-15140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yingjie Cao updated FLINK-15140:
--------------------------------
Description:
I tested the newest code of master branch last weekend with more test cases.
Unfortunately, several problems were encountered, including a bug of
compression.
When BroadcastRecordWriter is used, for pipelined mode, because the compressor
copies the data back to the input buffer, however, the underlying buffer is
shared when BroadcastRecordWriter is used. So we can not copy the compressed
buffer back to the input buffer if the underlying buffer is shared. For
blocking mode, we wrongly recycle the buffer when buffer is not compressed, and
the problem is also triggered when BroadcastRecordWriter is used.
To fix the problem, for blocking shuffle, the reference counter should be
maintained correctly, for pipelined shuffle, the simplest way maybe disable
compression when the underlying buffer is shared. I will open a PR to fix the
problem.
was:
I tested the newest code of master branch last weekend with more test cases.
Unfortunately, several problems were encountered, including a bug of
compression.
When BroadcastRecordWriter is used, for pipelined mode, because the compressor
copies the data back to the input buffer, however, the underlying buffer is
shared when BroadcastRecordWriter is used. So we can not copy the compressed
buffer back to the input buffer if the underlying buffer is shared. For
blocking mode, we wrongly recycle the buffer when buffer is not compressed, and
the problem is also triggered when BroadcastRecordWriter is used.
To fix the problem, for blocking shuffle, the reference counter should be
maintained correctly, for pipelined shuffle, the simplest way maybe disable
compression is the underlying buffer is shared. I will open a PR to fix the
problem.
> Shuffle data compression does not work with BroadcastRecordWriter.
> ------------------------------------------------------------------
>
> Key: FLINK-15140
> URL: https://issues.apache.org/jira/browse/FLINK-15140
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
> Affects Versions: 1.10.0
> Reporter: Yingjie Cao
> Priority: Blocker
> Fix For: 1.10.0
>
>
> I tested the newest code of master branch last weekend with more test cases.
> Unfortunately, several problems were encountered, including a bug of
> compression.
> When BroadcastRecordWriter is used, for pipelined mode, because the
> compressor copies the data back to the input buffer, however, the underlying
> buffer is shared when BroadcastRecordWriter is used. So we can not copy the
> compressed buffer back to the input buffer if the underlying buffer is
> shared. For blocking mode, we wrongly recycle the buffer when buffer is not
> compressed, and the problem is also triggered when BroadcastRecordWriter is
> used.
> To fix the problem, for blocking shuffle, the reference counter should be
> maintained correctly, for pipelined shuffle, the simplest way maybe disable
> compression when the underlying buffer is shared. I will open a PR to fix the
> problem.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
