Wang Shuo created SPARK-26418:
---------------------------------
Summary: Only OpenBlocks without any ChunkFetch for one stream
will cause memory leak in ExternalShuffleService
Key: SPARK-26418
URL: https://issues.apache.org/jira/browse/SPARK-26418
Project: Spark
Issue Type: Bug
Components: Shuffle
Affects Versions: 2.4.0
Reporter: Wang Shuo
In current code path, OneForOneStreamManager holds StreamState in a Map named
streams.
A StreamState is initialized and put into streams when OpenBlocks request
received.
One specific StreamState is removed from streams in two scenarios below:
# The last chunk of a stream is fetched
# The connection of ChunkFetch is closed
StreamState will never be clean up, if OpenBlocks request is received without
and following ChunkFetch request. This will cause memory leak in server side,
which is harmful for long running service such as ExternalShuffleService.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
