wangshuo128 opened a new pull request #23355: Only OpenBlocks without any ChunkFetch for one stream will cause memory leak in ExternalShuffleService URL: https://github.com/apache/spark/pull/23355 ## What changes were proposed in this pull request? In current code path, OneForOneStreamManager holds StreamState in a Map named streams. A StreamState is initialized and put into streams when OpenBlocks request received. One specific StreamState is removed from streams in two scenarios below: 1. The last chunk of a stream is fetched 2. The connection of ChunkFetch is closed StreamState will never be clean up, if OpenBlocks request is received without and following ChunkFetch request. This will cause memory leak in server side, which is harmful for long running service such as ExternalShuffleService. Could we call StreamManager#registerChannel to associate the channel with StreamState when server handles OpenBlocks request? ## How was this patch tested? New test added.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
