[
https://issues.apache.org/jira/browse/SPARK-26418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16725679#comment-16725679
]
Apache Spark commented on SPARK-26418:
--------------------------------------
User 'wangshuo128' has created a pull request for this issue:
https://github.com/apache/spark/pull/23355
> Only OpenBlocks without any ChunkFetch for one stream will cause memory leak
> in ExternalShuffleService
> ------------------------------------------------------------------------------------------------------
>
> Key: SPARK-26418
> URL: https://issues.apache.org/jira/browse/SPARK-26418
> Project: Spark
> Issue Type: Bug
> Components: Shuffle
> Affects Versions: 2.4.0
> Reporter: Wang Shuo
> Priority: Major
>
> In current code path, OneForOneStreamManager holds StreamState in a Map
> named streams.
> A StreamState is initialized and put into streams when OpenBlocks request
> received.
> One specific StreamState is removed from streams in two scenarios below:
> # The last chunk of a stream is fetched
> # The connection of ChunkFetch is closed
> StreamState will never be clean up, if OpenBlocks request is received without
> and following ChunkFetch request. This will cause memory leak in server
> side, which is harmful for long running service such as
> ExternalShuffleService.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
