Mate Czagany created FLINK-40335:
------------------------------------
Summary: Hybrid shuffle index files can be corrupted by a
ByteBuffer shared across result partitions
Key: FLINK-40335
URL: https://issues.apache.org/jira/browse/FLINK-40335
Project: Flink
Issue Type: Bug
Components: Runtime / Network
Reporter: Mate Czagany
h1. Summary
When using the hybrid shuffle mode, it's easy to run into this issue which is
caused by using a shared ByteBuffer.
TaskManagers first log exceptions from the index cache's removal listener
* BufferUnderflowException
* BufferOverflowException
* IllegalArgumentException: Negative position
Guava swallows these, and eventually terminate when a blocking shuffle I/O
thread hits
```
IOException: The spill file is corrupt: premature end of file as an uncaught
exception.
```
This issue has been present since the introduction of the tiered storage (1.18)
h1. Proposed fix
Create one `ProducerMergedPartitionFileDataIndexRegionHelper` per
`ProducerMergedPartitionFileIndex` instead of sharing a singleton. The cost is
one 24-byte direct buffer per active index instead of one per JVM
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
