[
https://issues.apache.org/jira/browse/SPARK-26418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcelo Masiero Vanzin resolved SPARK-26418.
--------------------------------------------
Resolution: Duplicate
> Only OpenBlocks without any ChunkFetch for one stream will cause memory leak
> in ExternalShuffleService
> ------------------------------------------------------------------------------------------------------
>
> Key: SPARK-26418
> URL: https://issues.apache.org/jira/browse/SPARK-26418
> Project: Spark
> Issue Type: Bug
> Components: Shuffle
> Affects Versions: 2.4.0
> Reporter: Wang Shuo
> Priority: Major
>
> In current code path, OneForOneStreamManager holds StreamState in a Map
> named streams.
> A StreamState is initialized and put into streams when OpenBlocks request
> received.
> One specific StreamState is removed from streams in two scenarios below:
> # The last chunk of a stream is fetched
> # The connection of ChunkFetch is closed
> StreamState will never be clean up, if OpenBlocks request is received without
> and following ChunkFetch request. This will cause memory leak in server
> side, which is harmful for long running service such as
> ExternalShuffleService.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
