Weijie Guo created FLINK-29053:
----------------------------------
Summary: Hybrid shuffle has concurrent modification of buffer when
compression is enabled
Key: FLINK-29053
URL: https://issues.apache.org/jira/browse/FLINK-29053
Project: Flink
Issue Type: Bug
Components: Runtime / Network
Affects Versions: 1.16.0
Reporter: Weijie Guo
Fix For: 1.16.0
When the downstream thread obtains the buffer and consuming it, if the data is
compressed in the spilling thread and copied to the original buffer in the same
time, since the two threads share the same memory data, the consuming thread
will consume incorrect data, causing problems such as deserialize the data
disorder.
Considering that the downstream consumption is prohibited during compression,
or block spilling thread when the downstream consumption is not completed will
have a great impact on performance. I think we should move the compression
operation to the write thread and store the compressed buffer directly in
memory.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
