[jira] [Commented] (SPARK-7183) Memory leak in netty shuffle with spark standalone cluster

2016-03-23 Thread Andy Sloane (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15208654#comment-15208654
 ] 

Andy Sloane commented on SPARK-7183:


FWIW, this also affects Spark 1.2.


> Memory leak in netty shuffle with spark standalone cluster
> --
>
> Key: SPARK-7183
> URL: https://issues.apache.org/jira/browse/SPARK-7183
> Project: Spark
>  Issue Type: Bug
>  Components: Shuffle
>Affects Versions: 1.3.0
>Reporter: Jack Hu
>Assignee: Liang-Chi Hsieh
>  Labels: memory-leak, netty, shuffle
> Fix For: 1.4.0
>
>
> There is slow leak in netty shuffle with spark cluster in 
> {{TransportRequestHandler.streamIds}}
> In spark cluster, there are some reusable netty connections between two block 
> managers to get/send blocks between worker/drivers. These connections are 
> handled by the {{org.apache.spark.network.server.TransportRequestHandler}} in 
> server side. This handler keep tracking all the streamids negotiate by RPC 
> when shuffle data need transform in these two block managers and the streamid 
> is keeping increasing, and never get a chance to be deleted exception this 
> connection is dropped (seems never happen in normal running).
> Here are some detail logs of this  {{TransportRequestHandler}} (Note: we add 
> a log a print the total size of {{TransportRequestHandler.streamIds}}, the 
> log is "Current set size is N of 
> org.apache.spark.network.server.TransportRequestHandler@ADDRESS", this set 
> size is keeping increasing in our test)
> {quote}
> 15/04/22 21:00:16 DEBUG TransportServer: Shuffle server started on port :46288
> 15/04/22 21:00:16 INFO NettyBlockTransferService: Server created on 46288
> 15/04/22 21:00:31 INFO TransportRequestHandler: Created 
> TransportRequestHandler 
> org.apache.spark.network.server.TransportRequestHandler@29a4f3e7
> 15/04/22 21:00:32 TRACE MessageDecoder: Received message RpcRequest: 
> RpcRequest\{requestId=6655045571437304938, message=[B@59778678\}
> 15/04/22 21:00:32 TRACE NettyBlockRpcServer: Received request: 
> OpenBlocks\{appId=app-20150422210016-, execId=, 
> blockIds=[broadcast_1_piece0]}
> 15/04/22 21:00:32 TRACE NettyBlockRpcServer: Registered streamId 
> 1387459488000 with 1 buffers
> 15/04/22 21:00:33 TRACE TransportRequestHandler: Sent result 
> RpcResponse\{requestId=6655045571437304938, response=[B@d2840b\} to client 
> /10.111.7.150:33802
> 15/04/22 21:00:33 TRACE MessageDecoder: Received message ChunkFetchRequest: 
> ChunkFetchRequest\{streamChunkId=StreamChunkId\{streamId=1387459488000, 
> chunkIndex=0}}
> 15/04/22 21:00:33 TRACE TransportRequestHandler: Received req from 
> /10.111.7.150:33802 to fetch block StreamChunkId\{streamId=1387459488000, 
> chunkIndex=0\}
> 15/04/22 21:00:33 INFO TransportRequestHandler: Current set size is 1 of 
> org.apache.spark.network.server.TransportRequestHandler@29a4f3e7
> 15/04/22 21:00:33 TRACE OneForOneStreamManager: Removing stream id 
> 1387459488000
> 15/04/22 21:00:33 TRACE TransportRequestHandler: Sent result 
> ChunkFetchSuccess\{streamChunkId=StreamChunkId\{streamId=1387459488000, 
> chunkIndex=0}, buffer=NioManagedBuffer\{buf=java.nio.HeapByteBuffer[pos=0 
> lim=3839 cap=3839]}} to client /10.111.7.150:33802
> 15/04/22 21:00:34 TRACE MessageDecoder: Received message RpcRequest: 
> RpcRequest\{requestId=6660601528868866371, message=[B@42bed1b8\}
> 15/04/22 21:00:34 TRACE NettyBlockRpcServer: Received request: 
> OpenBlocks\{appId=app-20150422210016-, execId=, 
> blockIds=[broadcast_3_piece0]}
> 15/04/22 21:00:34 TRACE NettyBlockRpcServer: Registered streamId 
> 1387459488001 with 1 buffers
> 15/04/22 21:00:34 TRACE TransportRequestHandler: Sent result 
> RpcResponse\{requestId=6660601528868866371, response=[B@7fa3fb60\} to client 
> /10.111.7.150:33802
> 15/04/22 21:00:34 TRACE MessageDecoder: Received message ChunkFetchRequest: 
> ChunkFetchRequest\{streamChunkId=StreamChunkId\{streamId=1387459488001, 
> chunkIndex=0}}
> 15/04/22 21:00:34 TRACE TransportRequestHandler: Received req from 
> /10.111.7.150:33802 to fetch block StreamChunkId\{streamId=1387459488001, 
> chunkIndex=0\}
> 15/04/22 21:00:34 INFO TransportRequestHandler: Current set size is 2 of 
> org.apache.spark.network.server.TransportRequestHandler@29a4f3e7
> 15/04/22 21:00:34 TRACE OneForOneStreamManager: Removing stream id 
> 1387459488001
> 15/04/22 21:00:34 TRACE TransportRequestHandler: Sent result 
> ChunkFetchSuccess\{streamChunkId=StreamChunkId\{streamId=1387459488001, 
> chunkIndex=0}, buffer=NioManagedBuffer\{buf=java.nio.HeapByteBuffer[pos=0 
> lim=4277 cap=4277]}} to client /10.111.7.150:33802
> 15/04/22 21:00:34 TRACE MessageDecoder: Received message RpcRequest: 
> RpcRequest\{requestId=8454597410163901330, message=[B@19c673d1\}
> 15/04/22 21:00:3

[jira] [Commented] (SPARK-7183) Memory leak in netty shuffle with spark standalone cluster

2015-05-13 Thread Aaron Davidson (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14543226#comment-14543226
 ] 

Aaron Davidson commented on SPARK-7183:
---

Sorry for delay, this should be fine to backport, it's a relatively 
straightforward fix.

> Memory leak in netty shuffle with spark standalone cluster
> --
>
> Key: SPARK-7183
> URL: https://issues.apache.org/jira/browse/SPARK-7183
> Project: Spark
>  Issue Type: Bug
>  Components: Shuffle
>Affects Versions: 1.3.0
>Reporter: Jack Hu
>Assignee: Liang-Chi Hsieh
>  Labels: memory-leak, netty, shuffle
> Fix For: 1.4.0
>
>
> There is slow leak in netty shuffle with spark cluster in 
> {{TransportRequestHandler.streamIds}}
> In spark cluster, there are some reusable netty connections between two block 
> managers to get/send blocks between worker/drivers. These connections are 
> handled by the {{org.apache.spark.network.server.TransportRequestHandler}} in 
> server side. This handler keep tracking all the streamids negotiate by RPC 
> when shuffle data need transform in these two block managers and the streamid 
> is keeping increasing, and never get a chance to be deleted exception this 
> connection is dropped (seems never happen in normal running).
> Here are some detail logs of this  {{TransportRequestHandler}} (Note: we add 
> a log a print the total size of {{TransportRequestHandler.streamIds}}, the 
> log is "Current set size is N of 
> org.apache.spark.network.server.TransportRequestHandler@ADDRESS", this set 
> size is keeping increasing in our test)
> {quote}
> 15/04/22 21:00:16 DEBUG TransportServer: Shuffle server started on port :46288
> 15/04/22 21:00:16 INFO NettyBlockTransferService: Server created on 46288
> 15/04/22 21:00:31 INFO TransportRequestHandler: Created 
> TransportRequestHandler 
> org.apache.spark.network.server.TransportRequestHandler@29a4f3e7
> 15/04/22 21:00:32 TRACE MessageDecoder: Received message RpcRequest: 
> RpcRequest\{requestId=6655045571437304938, message=[B@59778678\}
> 15/04/22 21:00:32 TRACE NettyBlockRpcServer: Received request: 
> OpenBlocks\{appId=app-20150422210016-, execId=, 
> blockIds=[broadcast_1_piece0]}
> 15/04/22 21:00:32 TRACE NettyBlockRpcServer: Registered streamId 
> 1387459488000 with 1 buffers
> 15/04/22 21:00:33 TRACE TransportRequestHandler: Sent result 
> RpcResponse\{requestId=6655045571437304938, response=[B@d2840b\} to client 
> /10.111.7.150:33802
> 15/04/22 21:00:33 TRACE MessageDecoder: Received message ChunkFetchRequest: 
> ChunkFetchRequest\{streamChunkId=StreamChunkId\{streamId=1387459488000, 
> chunkIndex=0}}
> 15/04/22 21:00:33 TRACE TransportRequestHandler: Received req from 
> /10.111.7.150:33802 to fetch block StreamChunkId\{streamId=1387459488000, 
> chunkIndex=0\}
> 15/04/22 21:00:33 INFO TransportRequestHandler: Current set size is 1 of 
> org.apache.spark.network.server.TransportRequestHandler@29a4f3e7
> 15/04/22 21:00:33 TRACE OneForOneStreamManager: Removing stream id 
> 1387459488000
> 15/04/22 21:00:33 TRACE TransportRequestHandler: Sent result 
> ChunkFetchSuccess\{streamChunkId=StreamChunkId\{streamId=1387459488000, 
> chunkIndex=0}, buffer=NioManagedBuffer\{buf=java.nio.HeapByteBuffer[pos=0 
> lim=3839 cap=3839]}} to client /10.111.7.150:33802
> 15/04/22 21:00:34 TRACE MessageDecoder: Received message RpcRequest: 
> RpcRequest\{requestId=6660601528868866371, message=[B@42bed1b8\}
> 15/04/22 21:00:34 TRACE NettyBlockRpcServer: Received request: 
> OpenBlocks\{appId=app-20150422210016-, execId=, 
> blockIds=[broadcast_3_piece0]}
> 15/04/22 21:00:34 TRACE NettyBlockRpcServer: Registered streamId 
> 1387459488001 with 1 buffers
> 15/04/22 21:00:34 TRACE TransportRequestHandler: Sent result 
> RpcResponse\{requestId=6660601528868866371, response=[B@7fa3fb60\} to client 
> /10.111.7.150:33802
> 15/04/22 21:00:34 TRACE MessageDecoder: Received message ChunkFetchRequest: 
> ChunkFetchRequest\{streamChunkId=StreamChunkId\{streamId=1387459488001, 
> chunkIndex=0}}
> 15/04/22 21:00:34 TRACE TransportRequestHandler: Received req from 
> /10.111.7.150:33802 to fetch block StreamChunkId\{streamId=1387459488001, 
> chunkIndex=0\}
> 15/04/22 21:00:34 INFO TransportRequestHandler: Current set size is 2 of 
> org.apache.spark.network.server.TransportRequestHandler@29a4f3e7
> 15/04/22 21:00:34 TRACE OneForOneStreamManager: Removing stream id 
> 1387459488001
> 15/04/22 21:00:34 TRACE TransportRequestHandler: Sent result 
> ChunkFetchSuccess\{streamChunkId=StreamChunkId\{streamId=1387459488001, 
> chunkIndex=0}, buffer=NioManagedBuffer\{buf=java.nio.HeapByteBuffer[pos=0 
> lim=4277 cap=4277]}} to client /10.111.7.150:33802
> 15/04/22 21:00:34 TRACE MessageDecoder: Received message RpcRequest: 
> RpcRequest\{requestId=8

[jira] [Commented] (SPARK-7183) Memory leak in netty shuffle with spark standalone cluster

2015-05-07 Thread Sean Owen (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14532502#comment-14532502
 ] 

Sean Owen commented on SPARK-7183:
--

[~ilikerps] what do you think? ok to back port? 

> Memory leak in netty shuffle with spark standalone cluster
> --
>
> Key: SPARK-7183
> URL: https://issues.apache.org/jira/browse/SPARK-7183
> Project: Spark
>  Issue Type: Bug
>  Components: Shuffle
>Affects Versions: 1.3.0
>Reporter: Jack Hu
>Assignee: Liang-Chi Hsieh
>  Labels: memory-leak, netty, shuffle
> Fix For: 1.4.0
>
>
> There is slow leak in netty shuffle with spark cluster in 
> {{TransportRequestHandler.streamIds}}
> In spark cluster, there are some reusable netty connections between two block 
> managers to get/send blocks between worker/drivers. These connections are 
> handled by the {{org.apache.spark.network.server.TransportRequestHandler}} in 
> server side. This handler keep tracking all the streamids negotiate by RPC 
> when shuffle data need transform in these two block managers and the streamid 
> is keeping increasing, and never get a chance to be deleted exception this 
> connection is dropped (seems never happen in normal running).
> Here are some detail logs of this  {{TransportRequestHandler}} (Note: we add 
> a log a print the total size of {{TransportRequestHandler.streamIds}}, the 
> log is "Current set size is N of 
> org.apache.spark.network.server.TransportRequestHandler@ADDRESS", this set 
> size is keeping increasing in our test)
> {quote}
> 15/04/22 21:00:16 DEBUG TransportServer: Shuffle server started on port :46288
> 15/04/22 21:00:16 INFO NettyBlockTransferService: Server created on 46288
> 15/04/22 21:00:31 INFO TransportRequestHandler: Created 
> TransportRequestHandler 
> org.apache.spark.network.server.TransportRequestHandler@29a4f3e7
> 15/04/22 21:00:32 TRACE MessageDecoder: Received message RpcRequest: 
> RpcRequest\{requestId=6655045571437304938, message=[B@59778678\}
> 15/04/22 21:00:32 TRACE NettyBlockRpcServer: Received request: 
> OpenBlocks\{appId=app-20150422210016-, execId=, 
> blockIds=[broadcast_1_piece0]}
> 15/04/22 21:00:32 TRACE NettyBlockRpcServer: Registered streamId 
> 1387459488000 with 1 buffers
> 15/04/22 21:00:33 TRACE TransportRequestHandler: Sent result 
> RpcResponse\{requestId=6655045571437304938, response=[B@d2840b\} to client 
> /10.111.7.150:33802
> 15/04/22 21:00:33 TRACE MessageDecoder: Received message ChunkFetchRequest: 
> ChunkFetchRequest\{streamChunkId=StreamChunkId\{streamId=1387459488000, 
> chunkIndex=0}}
> 15/04/22 21:00:33 TRACE TransportRequestHandler: Received req from 
> /10.111.7.150:33802 to fetch block StreamChunkId\{streamId=1387459488000, 
> chunkIndex=0\}
> 15/04/22 21:00:33 INFO TransportRequestHandler: Current set size is 1 of 
> org.apache.spark.network.server.TransportRequestHandler@29a4f3e7
> 15/04/22 21:00:33 TRACE OneForOneStreamManager: Removing stream id 
> 1387459488000
> 15/04/22 21:00:33 TRACE TransportRequestHandler: Sent result 
> ChunkFetchSuccess\{streamChunkId=StreamChunkId\{streamId=1387459488000, 
> chunkIndex=0}, buffer=NioManagedBuffer\{buf=java.nio.HeapByteBuffer[pos=0 
> lim=3839 cap=3839]}} to client /10.111.7.150:33802
> 15/04/22 21:00:34 TRACE MessageDecoder: Received message RpcRequest: 
> RpcRequest\{requestId=6660601528868866371, message=[B@42bed1b8\}
> 15/04/22 21:00:34 TRACE NettyBlockRpcServer: Received request: 
> OpenBlocks\{appId=app-20150422210016-, execId=, 
> blockIds=[broadcast_3_piece0]}
> 15/04/22 21:00:34 TRACE NettyBlockRpcServer: Registered streamId 
> 1387459488001 with 1 buffers
> 15/04/22 21:00:34 TRACE TransportRequestHandler: Sent result 
> RpcResponse\{requestId=6660601528868866371, response=[B@7fa3fb60\} to client 
> /10.111.7.150:33802
> 15/04/22 21:00:34 TRACE MessageDecoder: Received message ChunkFetchRequest: 
> ChunkFetchRequest\{streamChunkId=StreamChunkId\{streamId=1387459488001, 
> chunkIndex=0}}
> 15/04/22 21:00:34 TRACE TransportRequestHandler: Received req from 
> /10.111.7.150:33802 to fetch block StreamChunkId\{streamId=1387459488001, 
> chunkIndex=0\}
> 15/04/22 21:00:34 INFO TransportRequestHandler: Current set size is 2 of 
> org.apache.spark.network.server.TransportRequestHandler@29a4f3e7
> 15/04/22 21:00:34 TRACE OneForOneStreamManager: Removing stream id 
> 1387459488001
> 15/04/22 21:00:34 TRACE TransportRequestHandler: Sent result 
> ChunkFetchSuccess\{streamChunkId=StreamChunkId\{streamId=1387459488001, 
> chunkIndex=0}, buffer=NioManagedBuffer\{buf=java.nio.HeapByteBuffer[pos=0 
> lim=4277 cap=4277]}} to client /10.111.7.150:33802
> 15/04/22 21:00:34 TRACE MessageDecoder: Received message RpcRequest: 
> RpcRequest\{requestId=8454597410163901330, message=[B@19c673d1\}
> 15/04/2

[jira] [Commented] (SPARK-7183) Memory leak in netty shuffle with spark standalone cluster

2015-05-06 Thread Jack Hu (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14532108#comment-14532108
 ] 

Jack Hu commented on SPARK-7183:


Hi, [~sowen]

Do we plan to add this to 1.3+? If there is any plan to release more minor 
release for 1.3+ like 1.3.2. 


> Memory leak in netty shuffle with spark standalone cluster
> --
>
> Key: SPARK-7183
> URL: https://issues.apache.org/jira/browse/SPARK-7183
> Project: Spark
>  Issue Type: Bug
>  Components: Shuffle
>Affects Versions: 1.3.0
>Reporter: Jack Hu
>Assignee: Liang-Chi Hsieh
>  Labels: memory-leak, netty, shuffle
> Fix For: 1.4.0
>
>
> There is slow leak in netty shuffle with spark cluster in 
> {{TransportRequestHandler.streamIds}}
> In spark cluster, there are some reusable netty connections between two block 
> managers to get/send blocks between worker/drivers. These connections are 
> handled by the {{org.apache.spark.network.server.TransportRequestHandler}} in 
> server side. This handler keep tracking all the streamids negotiate by RPC 
> when shuffle data need transform in these two block managers and the streamid 
> is keeping increasing, and never get a chance to be deleted exception this 
> connection is dropped (seems never happen in normal running).
> Here are some detail logs of this  {{TransportRequestHandler}} (Note: we add 
> a log a print the total size of {{TransportRequestHandler.streamIds}}, the 
> log is "Current set size is N of 
> org.apache.spark.network.server.TransportRequestHandler@ADDRESS", this set 
> size is keeping increasing in our test)
> {quote}
> 15/04/22 21:00:16 DEBUG TransportServer: Shuffle server started on port :46288
> 15/04/22 21:00:16 INFO NettyBlockTransferService: Server created on 46288
> 15/04/22 21:00:31 INFO TransportRequestHandler: Created 
> TransportRequestHandler 
> org.apache.spark.network.server.TransportRequestHandler@29a4f3e7
> 15/04/22 21:00:32 TRACE MessageDecoder: Received message RpcRequest: 
> RpcRequest\{requestId=6655045571437304938, message=[B@59778678\}
> 15/04/22 21:00:32 TRACE NettyBlockRpcServer: Received request: 
> OpenBlocks\{appId=app-20150422210016-, execId=, 
> blockIds=[broadcast_1_piece0]}
> 15/04/22 21:00:32 TRACE NettyBlockRpcServer: Registered streamId 
> 1387459488000 with 1 buffers
> 15/04/22 21:00:33 TRACE TransportRequestHandler: Sent result 
> RpcResponse\{requestId=6655045571437304938, response=[B@d2840b\} to client 
> /10.111.7.150:33802
> 15/04/22 21:00:33 TRACE MessageDecoder: Received message ChunkFetchRequest: 
> ChunkFetchRequest\{streamChunkId=StreamChunkId\{streamId=1387459488000, 
> chunkIndex=0}}
> 15/04/22 21:00:33 TRACE TransportRequestHandler: Received req from 
> /10.111.7.150:33802 to fetch block StreamChunkId\{streamId=1387459488000, 
> chunkIndex=0\}
> 15/04/22 21:00:33 INFO TransportRequestHandler: Current set size is 1 of 
> org.apache.spark.network.server.TransportRequestHandler@29a4f3e7
> 15/04/22 21:00:33 TRACE OneForOneStreamManager: Removing stream id 
> 1387459488000
> 15/04/22 21:00:33 TRACE TransportRequestHandler: Sent result 
> ChunkFetchSuccess\{streamChunkId=StreamChunkId\{streamId=1387459488000, 
> chunkIndex=0}, buffer=NioManagedBuffer\{buf=java.nio.HeapByteBuffer[pos=0 
> lim=3839 cap=3839]}} to client /10.111.7.150:33802
> 15/04/22 21:00:34 TRACE MessageDecoder: Received message RpcRequest: 
> RpcRequest\{requestId=6660601528868866371, message=[B@42bed1b8\}
> 15/04/22 21:00:34 TRACE NettyBlockRpcServer: Received request: 
> OpenBlocks\{appId=app-20150422210016-, execId=, 
> blockIds=[broadcast_3_piece0]}
> 15/04/22 21:00:34 TRACE NettyBlockRpcServer: Registered streamId 
> 1387459488001 with 1 buffers
> 15/04/22 21:00:34 TRACE TransportRequestHandler: Sent result 
> RpcResponse\{requestId=6660601528868866371, response=[B@7fa3fb60\} to client 
> /10.111.7.150:33802
> 15/04/22 21:00:34 TRACE MessageDecoder: Received message ChunkFetchRequest: 
> ChunkFetchRequest\{streamChunkId=StreamChunkId\{streamId=1387459488001, 
> chunkIndex=0}}
> 15/04/22 21:00:34 TRACE TransportRequestHandler: Received req from 
> /10.111.7.150:33802 to fetch block StreamChunkId\{streamId=1387459488001, 
> chunkIndex=0\}
> 15/04/22 21:00:34 INFO TransportRequestHandler: Current set size is 2 of 
> org.apache.spark.network.server.TransportRequestHandler@29a4f3e7
> 15/04/22 21:00:34 TRACE OneForOneStreamManager: Removing stream id 
> 1387459488001
> 15/04/22 21:00:34 TRACE TransportRequestHandler: Sent result 
> ChunkFetchSuccess\{streamChunkId=StreamChunkId\{streamId=1387459488001, 
> chunkIndex=0}, buffer=NioManagedBuffer\{buf=java.nio.HeapByteBuffer[pos=0 
> lim=4277 cap=4277]}} to client /10.111.7.150:33802
> 15/04/22 21:00:34 TRACE MessageDecoder: Received message RpcRequest: 
> RpcReq

[jira] [Commented] (SPARK-7183) Memory leak in netty shuffle with spark standalone cluster

2015-04-28 Thread Apache Spark (JIRA) 

[ 
https://issues.apache.org/jira/browse/SPARK-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516765#comment-14516765
 ] 

Apache Spark commented on SPARK-7183:
-

User 'viirya' has created a pull request for this issue:
https://github.com/apache/spark/pull/5743

> Memory leak in netty shuffle with spark standalone cluster
> --
>
> Key: SPARK-7183
> URL: https://issues.apache.org/jira/browse/SPARK-7183
> Project: Spark
>  Issue Type: Bug
>  Components: Shuffle
>Affects Versions: 1.3.0
>Reporter: Jack Hu
>  Labels: memory-leak, netty, shuffle
>
> There is slow leak in netty shuffle with spark cluster in 
> {{TransportRequestHandler.streamIds}}
> In spark cluster, there are some reusable netty connections between two block 
> managers to get/send blocks between worker/drivers. These connections are 
> handled by the {{org.apache.spark.network.server.TransportRequestHandler}} in 
> server side. This handler keep tracking all the streamids negotiate by RPC 
> when shuffle data need transform in these two block managers and the streamid 
> is keeping increasing, and never get a chance to be deleted exception this 
> connection is dropped (seems never happen in normal running).
> Here are some detail logs of this  {{TransportRequestHandler}} (Note: we add 
> a log a print the total size of {{TransportRequestHandler.streamIds}}, the 
> log is "Current set size is N of 
> org.apache.spark.network.server.TransportRequestHandler@ADDRESS", this set 
> size is keeping increasing in our test)
> {quote}
> 15/04/22 21:00:16 DEBUG TransportServer: Shuffle server started on port :46288
> 15/04/22 21:00:16 INFO NettyBlockTransferService: Server created on 46288
> 15/04/22 21:00:31 INFO TransportRequestHandler: Created 
> TransportRequestHandler 
> org.apache.spark.network.server.TransportRequestHandler@29a4f3e7
> 15/04/22 21:00:32 TRACE MessageDecoder: Received message RpcRequest: 
> RpcRequest\{requestId=6655045571437304938, message=[B@59778678\}
> 15/04/22 21:00:32 TRACE NettyBlockRpcServer: Received request: 
> OpenBlocks\{appId=app-20150422210016-, execId=, 
> blockIds=[broadcast_1_piece0]}
> 15/04/22 21:00:32 TRACE NettyBlockRpcServer: Registered streamId 
> 1387459488000 with 1 buffers
> 15/04/22 21:00:33 TRACE TransportRequestHandler: Sent result 
> RpcResponse\{requestId=6655045571437304938, response=[B@d2840b\} to client 
> /10.111.7.150:33802
> 15/04/22 21:00:33 TRACE MessageDecoder: Received message ChunkFetchRequest: 
> ChunkFetchRequest\{streamChunkId=StreamChunkId\{streamId=1387459488000, 
> chunkIndex=0}}
> 15/04/22 21:00:33 TRACE TransportRequestHandler: Received req from 
> /10.111.7.150:33802 to fetch block StreamChunkId\{streamId=1387459488000, 
> chunkIndex=0\}
> 15/04/22 21:00:33 INFO TransportRequestHandler: Current set size is 1 of 
> org.apache.spark.network.server.TransportRequestHandler@29a4f3e7
> 15/04/22 21:00:33 TRACE OneForOneStreamManager: Removing stream id 
> 1387459488000
> 15/04/22 21:00:33 TRACE TransportRequestHandler: Sent result 
> ChunkFetchSuccess\{streamChunkId=StreamChunkId\{streamId=1387459488000, 
> chunkIndex=0}, buffer=NioManagedBuffer\{buf=java.nio.HeapByteBuffer[pos=0 
> lim=3839 cap=3839]}} to client /10.111.7.150:33802
> 15/04/22 21:00:34 TRACE MessageDecoder: Received message RpcRequest: 
> RpcRequest\{requestId=6660601528868866371, message=[B@42bed1b8\}
> 15/04/22 21:00:34 TRACE NettyBlockRpcServer: Received request: 
> OpenBlocks\{appId=app-20150422210016-, execId=, 
> blockIds=[broadcast_3_piece0]}
> 15/04/22 21:00:34 TRACE NettyBlockRpcServer: Registered streamId 
> 1387459488001 with 1 buffers
> 15/04/22 21:00:34 TRACE TransportRequestHandler: Sent result 
> RpcResponse\{requestId=6660601528868866371, response=[B@7fa3fb60\} to client 
> /10.111.7.150:33802
> 15/04/22 21:00:34 TRACE MessageDecoder: Received message ChunkFetchRequest: 
> ChunkFetchRequest\{streamChunkId=StreamChunkId\{streamId=1387459488001, 
> chunkIndex=0}}
> 15/04/22 21:00:34 TRACE TransportRequestHandler: Received req from 
> /10.111.7.150:33802 to fetch block StreamChunkId\{streamId=1387459488001, 
> chunkIndex=0\}
> 15/04/22 21:00:34 INFO TransportRequestHandler: Current set size is 2 of 
> org.apache.spark.network.server.TransportRequestHandler@29a4f3e7
> 15/04/22 21:00:34 TRACE OneForOneStreamManager: Removing stream id 
> 1387459488001
> 15/04/22 21:00:34 TRACE TransportRequestHandler: Sent result 
> ChunkFetchSuccess\{streamChunkId=StreamChunkId\{streamId=1387459488001, 
> chunkIndex=0}, buffer=NioManagedBuffer\{buf=java.nio.HeapByteBuffer[pos=0 
> lim=4277 cap=4277]}} to client /10.111.7.150:33802
> 15/04/22 21:00:34 TRACE MessageDecoder: Received message RpcRequest: 
> RpcRequest\{requestId=8454597410163901330, message=[B@19c673d1\}
> 15/04/22 21:00:34 TRA