[jira] [Commented] (SPARK-7183) Memory leak in netty shuffle with spark standalone cluster
[ https://issues.apache.org/jira/browse/SPARK-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15208654#comment-15208654 ] Andy Sloane commented on SPARK-7183: FWIW, this also affects Spark 1.2. > Memory leak in netty shuffle with spark standalone cluster > -- > > Key: SPARK-7183 > URL: https://issues.apache.org/jira/browse/SPARK-7183 > Project: Spark > Issue Type: Bug > Components: Shuffle >Affects Versions: 1.3.0 >Reporter: Jack Hu >Assignee: Liang-Chi Hsieh > Labels: memory-leak, netty, shuffle > Fix For: 1.4.0 > > > There is slow leak in netty shuffle with spark cluster in > {{TransportRequestHandler.streamIds}} > In spark cluster, there are some reusable netty connections between two block > managers to get/send blocks between worker/drivers. These connections are > handled by the {{org.apache.spark.network.server.TransportRequestHandler}} in > server side. This handler keep tracking all the streamids negotiate by RPC > when shuffle data need transform in these two block managers and the streamid > is keeping increasing, and never get a chance to be deleted exception this > connection is dropped (seems never happen in normal running). > Here are some detail logs of this {{TransportRequestHandler}} (Note: we add > a log a print the total size of {{TransportRequestHandler.streamIds}}, the > log is "Current set size is N of > org.apache.spark.network.server.TransportRequestHandler@ADDRESS", this set > size is keeping increasing in our test) > {quote} > 15/04/22 21:00:16 DEBUG TransportServer: Shuffle server started on port :46288 > 15/04/22 21:00:16 INFO NettyBlockTransferService: Server created on 46288 > 15/04/22 21:00:31 INFO TransportRequestHandler: Created > TransportRequestHandler > org.apache.spark.network.server.TransportRequestHandler@29a4f3e7 > 15/04/22 21:00:32 TRACE MessageDecoder: Received message RpcRequest: > RpcRequest\{requestId=6655045571437304938, message=[B@59778678\} > 15/04/22 21:00:32 TRACE NettyBlockRpcServer: Received request: > OpenBlocks\{appId=app-20150422210016-, execId=, > blockIds=[broadcast_1_piece0]} > 15/04/22 21:00:32 TRACE NettyBlockRpcServer: Registered streamId > 1387459488000 with 1 buffers > 15/04/22 21:00:33 TRACE TransportRequestHandler: Sent result > RpcResponse\{requestId=6655045571437304938, response=[B@d2840b\} to client > /10.111.7.150:33802 > 15/04/22 21:00:33 TRACE MessageDecoder: Received message ChunkFetchRequest: > ChunkFetchRequest\{streamChunkId=StreamChunkId\{streamId=1387459488000, > chunkIndex=0}} > 15/04/22 21:00:33 TRACE TransportRequestHandler: Received req from > /10.111.7.150:33802 to fetch block StreamChunkId\{streamId=1387459488000, > chunkIndex=0\} > 15/04/22 21:00:33 INFO TransportRequestHandler: Current set size is 1 of > org.apache.spark.network.server.TransportRequestHandler@29a4f3e7 > 15/04/22 21:00:33 TRACE OneForOneStreamManager: Removing stream id > 1387459488000 > 15/04/22 21:00:33 TRACE TransportRequestHandler: Sent result > ChunkFetchSuccess\{streamChunkId=StreamChunkId\{streamId=1387459488000, > chunkIndex=0}, buffer=NioManagedBuffer\{buf=java.nio.HeapByteBuffer[pos=0 > lim=3839 cap=3839]}} to client /10.111.7.150:33802 > 15/04/22 21:00:34 TRACE MessageDecoder: Received message RpcRequest: > RpcRequest\{requestId=6660601528868866371, message=[B@42bed1b8\} > 15/04/22 21:00:34 TRACE NettyBlockRpcServer: Received request: > OpenBlocks\{appId=app-20150422210016-, execId=, > blockIds=[broadcast_3_piece0]} > 15/04/22 21:00:34 TRACE NettyBlockRpcServer: Registered streamId > 1387459488001 with 1 buffers > 15/04/22 21:00:34 TRACE TransportRequestHandler: Sent result > RpcResponse\{requestId=6660601528868866371, response=[B@7fa3fb60\} to client > /10.111.7.150:33802 > 15/04/22 21:00:34 TRACE MessageDecoder: Received message ChunkFetchRequest: > ChunkFetchRequest\{streamChunkId=StreamChunkId\{streamId=1387459488001, > chunkIndex=0}} > 15/04/22 21:00:34 TRACE TransportRequestHandler: Received req from > /10.111.7.150:33802 to fetch block StreamChunkId\{streamId=1387459488001, > chunkIndex=0\} > 15/04/22 21:00:34 INFO TransportRequestHandler: Current set size is 2 of > org.apache.spark.network.server.TransportRequestHandler@29a4f3e7 > 15/04/22 21:00:34 TRACE OneForOneStreamManager: Removing stream id > 1387459488001 > 15/04/22 21:00:34 TRACE TransportRequestHandler: Sent result > ChunkFetchSuccess\{streamChunkId=StreamChunkId\{streamId=1387459488001, > chunkIndex=0}, buffer=NioManagedBuffer\{buf=java.nio.HeapByteBuffer[pos=0 > lim=4277 cap=4277]}} to client /10.111.7.150:33802 > 15/04/22 21:00:34 TRACE MessageDecoder: Received message RpcRequest: > RpcRequest\{requestId=8454597410163901330, message=[B@19c673d1\} > 15/04/22 21:00:3
[jira] [Commented] (SPARK-7183) Memory leak in netty shuffle with spark standalone cluster
[ https://issues.apache.org/jira/browse/SPARK-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14543226#comment-14543226 ] Aaron Davidson commented on SPARK-7183: --- Sorry for delay, this should be fine to backport, it's a relatively straightforward fix. > Memory leak in netty shuffle with spark standalone cluster > -- > > Key: SPARK-7183 > URL: https://issues.apache.org/jira/browse/SPARK-7183 > Project: Spark > Issue Type: Bug > Components: Shuffle >Affects Versions: 1.3.0 >Reporter: Jack Hu >Assignee: Liang-Chi Hsieh > Labels: memory-leak, netty, shuffle > Fix For: 1.4.0 > > > There is slow leak in netty shuffle with spark cluster in > {{TransportRequestHandler.streamIds}} > In spark cluster, there are some reusable netty connections between two block > managers to get/send blocks between worker/drivers. These connections are > handled by the {{org.apache.spark.network.server.TransportRequestHandler}} in > server side. This handler keep tracking all the streamids negotiate by RPC > when shuffle data need transform in these two block managers and the streamid > is keeping increasing, and never get a chance to be deleted exception this > connection is dropped (seems never happen in normal running). > Here are some detail logs of this {{TransportRequestHandler}} (Note: we add > a log a print the total size of {{TransportRequestHandler.streamIds}}, the > log is "Current set size is N of > org.apache.spark.network.server.TransportRequestHandler@ADDRESS", this set > size is keeping increasing in our test) > {quote} > 15/04/22 21:00:16 DEBUG TransportServer: Shuffle server started on port :46288 > 15/04/22 21:00:16 INFO NettyBlockTransferService: Server created on 46288 > 15/04/22 21:00:31 INFO TransportRequestHandler: Created > TransportRequestHandler > org.apache.spark.network.server.TransportRequestHandler@29a4f3e7 > 15/04/22 21:00:32 TRACE MessageDecoder: Received message RpcRequest: > RpcRequest\{requestId=6655045571437304938, message=[B@59778678\} > 15/04/22 21:00:32 TRACE NettyBlockRpcServer: Received request: > OpenBlocks\{appId=app-20150422210016-, execId=, > blockIds=[broadcast_1_piece0]} > 15/04/22 21:00:32 TRACE NettyBlockRpcServer: Registered streamId > 1387459488000 with 1 buffers > 15/04/22 21:00:33 TRACE TransportRequestHandler: Sent result > RpcResponse\{requestId=6655045571437304938, response=[B@d2840b\} to client > /10.111.7.150:33802 > 15/04/22 21:00:33 TRACE MessageDecoder: Received message ChunkFetchRequest: > ChunkFetchRequest\{streamChunkId=StreamChunkId\{streamId=1387459488000, > chunkIndex=0}} > 15/04/22 21:00:33 TRACE TransportRequestHandler: Received req from > /10.111.7.150:33802 to fetch block StreamChunkId\{streamId=1387459488000, > chunkIndex=0\} > 15/04/22 21:00:33 INFO TransportRequestHandler: Current set size is 1 of > org.apache.spark.network.server.TransportRequestHandler@29a4f3e7 > 15/04/22 21:00:33 TRACE OneForOneStreamManager: Removing stream id > 1387459488000 > 15/04/22 21:00:33 TRACE TransportRequestHandler: Sent result > ChunkFetchSuccess\{streamChunkId=StreamChunkId\{streamId=1387459488000, > chunkIndex=0}, buffer=NioManagedBuffer\{buf=java.nio.HeapByteBuffer[pos=0 > lim=3839 cap=3839]}} to client /10.111.7.150:33802 > 15/04/22 21:00:34 TRACE MessageDecoder: Received message RpcRequest: > RpcRequest\{requestId=6660601528868866371, message=[B@42bed1b8\} > 15/04/22 21:00:34 TRACE NettyBlockRpcServer: Received request: > OpenBlocks\{appId=app-20150422210016-, execId=, > blockIds=[broadcast_3_piece0]} > 15/04/22 21:00:34 TRACE NettyBlockRpcServer: Registered streamId > 1387459488001 with 1 buffers > 15/04/22 21:00:34 TRACE TransportRequestHandler: Sent result > RpcResponse\{requestId=6660601528868866371, response=[B@7fa3fb60\} to client > /10.111.7.150:33802 > 15/04/22 21:00:34 TRACE MessageDecoder: Received message ChunkFetchRequest: > ChunkFetchRequest\{streamChunkId=StreamChunkId\{streamId=1387459488001, > chunkIndex=0}} > 15/04/22 21:00:34 TRACE TransportRequestHandler: Received req from > /10.111.7.150:33802 to fetch block StreamChunkId\{streamId=1387459488001, > chunkIndex=0\} > 15/04/22 21:00:34 INFO TransportRequestHandler: Current set size is 2 of > org.apache.spark.network.server.TransportRequestHandler@29a4f3e7 > 15/04/22 21:00:34 TRACE OneForOneStreamManager: Removing stream id > 1387459488001 > 15/04/22 21:00:34 TRACE TransportRequestHandler: Sent result > ChunkFetchSuccess\{streamChunkId=StreamChunkId\{streamId=1387459488001, > chunkIndex=0}, buffer=NioManagedBuffer\{buf=java.nio.HeapByteBuffer[pos=0 > lim=4277 cap=4277]}} to client /10.111.7.150:33802 > 15/04/22 21:00:34 TRACE MessageDecoder: Received message RpcRequest: > RpcRequest\{requestId=8
[jira] [Commented] (SPARK-7183) Memory leak in netty shuffle with spark standalone cluster
[ https://issues.apache.org/jira/browse/SPARK-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14532502#comment-14532502 ] Sean Owen commented on SPARK-7183: -- [~ilikerps] what do you think? ok to back port? > Memory leak in netty shuffle with spark standalone cluster > -- > > Key: SPARK-7183 > URL: https://issues.apache.org/jira/browse/SPARK-7183 > Project: Spark > Issue Type: Bug > Components: Shuffle >Affects Versions: 1.3.0 >Reporter: Jack Hu >Assignee: Liang-Chi Hsieh > Labels: memory-leak, netty, shuffle > Fix For: 1.4.0 > > > There is slow leak in netty shuffle with spark cluster in > {{TransportRequestHandler.streamIds}} > In spark cluster, there are some reusable netty connections between two block > managers to get/send blocks between worker/drivers. These connections are > handled by the {{org.apache.spark.network.server.TransportRequestHandler}} in > server side. This handler keep tracking all the streamids negotiate by RPC > when shuffle data need transform in these two block managers and the streamid > is keeping increasing, and never get a chance to be deleted exception this > connection is dropped (seems never happen in normal running). > Here are some detail logs of this {{TransportRequestHandler}} (Note: we add > a log a print the total size of {{TransportRequestHandler.streamIds}}, the > log is "Current set size is N of > org.apache.spark.network.server.TransportRequestHandler@ADDRESS", this set > size is keeping increasing in our test) > {quote} > 15/04/22 21:00:16 DEBUG TransportServer: Shuffle server started on port :46288 > 15/04/22 21:00:16 INFO NettyBlockTransferService: Server created on 46288 > 15/04/22 21:00:31 INFO TransportRequestHandler: Created > TransportRequestHandler > org.apache.spark.network.server.TransportRequestHandler@29a4f3e7 > 15/04/22 21:00:32 TRACE MessageDecoder: Received message RpcRequest: > RpcRequest\{requestId=6655045571437304938, message=[B@59778678\} > 15/04/22 21:00:32 TRACE NettyBlockRpcServer: Received request: > OpenBlocks\{appId=app-20150422210016-, execId=, > blockIds=[broadcast_1_piece0]} > 15/04/22 21:00:32 TRACE NettyBlockRpcServer: Registered streamId > 1387459488000 with 1 buffers > 15/04/22 21:00:33 TRACE TransportRequestHandler: Sent result > RpcResponse\{requestId=6655045571437304938, response=[B@d2840b\} to client > /10.111.7.150:33802 > 15/04/22 21:00:33 TRACE MessageDecoder: Received message ChunkFetchRequest: > ChunkFetchRequest\{streamChunkId=StreamChunkId\{streamId=1387459488000, > chunkIndex=0}} > 15/04/22 21:00:33 TRACE TransportRequestHandler: Received req from > /10.111.7.150:33802 to fetch block StreamChunkId\{streamId=1387459488000, > chunkIndex=0\} > 15/04/22 21:00:33 INFO TransportRequestHandler: Current set size is 1 of > org.apache.spark.network.server.TransportRequestHandler@29a4f3e7 > 15/04/22 21:00:33 TRACE OneForOneStreamManager: Removing stream id > 1387459488000 > 15/04/22 21:00:33 TRACE TransportRequestHandler: Sent result > ChunkFetchSuccess\{streamChunkId=StreamChunkId\{streamId=1387459488000, > chunkIndex=0}, buffer=NioManagedBuffer\{buf=java.nio.HeapByteBuffer[pos=0 > lim=3839 cap=3839]}} to client /10.111.7.150:33802 > 15/04/22 21:00:34 TRACE MessageDecoder: Received message RpcRequest: > RpcRequest\{requestId=6660601528868866371, message=[B@42bed1b8\} > 15/04/22 21:00:34 TRACE NettyBlockRpcServer: Received request: > OpenBlocks\{appId=app-20150422210016-, execId=, > blockIds=[broadcast_3_piece0]} > 15/04/22 21:00:34 TRACE NettyBlockRpcServer: Registered streamId > 1387459488001 with 1 buffers > 15/04/22 21:00:34 TRACE TransportRequestHandler: Sent result > RpcResponse\{requestId=6660601528868866371, response=[B@7fa3fb60\} to client > /10.111.7.150:33802 > 15/04/22 21:00:34 TRACE MessageDecoder: Received message ChunkFetchRequest: > ChunkFetchRequest\{streamChunkId=StreamChunkId\{streamId=1387459488001, > chunkIndex=0}} > 15/04/22 21:00:34 TRACE TransportRequestHandler: Received req from > /10.111.7.150:33802 to fetch block StreamChunkId\{streamId=1387459488001, > chunkIndex=0\} > 15/04/22 21:00:34 INFO TransportRequestHandler: Current set size is 2 of > org.apache.spark.network.server.TransportRequestHandler@29a4f3e7 > 15/04/22 21:00:34 TRACE OneForOneStreamManager: Removing stream id > 1387459488001 > 15/04/22 21:00:34 TRACE TransportRequestHandler: Sent result > ChunkFetchSuccess\{streamChunkId=StreamChunkId\{streamId=1387459488001, > chunkIndex=0}, buffer=NioManagedBuffer\{buf=java.nio.HeapByteBuffer[pos=0 > lim=4277 cap=4277]}} to client /10.111.7.150:33802 > 15/04/22 21:00:34 TRACE MessageDecoder: Received message RpcRequest: > RpcRequest\{requestId=8454597410163901330, message=[B@19c673d1\} > 15/04/2
[jira] [Commented] (SPARK-7183) Memory leak in netty shuffle with spark standalone cluster
[ https://issues.apache.org/jira/browse/SPARK-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14532108#comment-14532108 ] Jack Hu commented on SPARK-7183: Hi, [~sowen] Do we plan to add this to 1.3+? If there is any plan to release more minor release for 1.3+ like 1.3.2. > Memory leak in netty shuffle with spark standalone cluster > -- > > Key: SPARK-7183 > URL: https://issues.apache.org/jira/browse/SPARK-7183 > Project: Spark > Issue Type: Bug > Components: Shuffle >Affects Versions: 1.3.0 >Reporter: Jack Hu >Assignee: Liang-Chi Hsieh > Labels: memory-leak, netty, shuffle > Fix For: 1.4.0 > > > There is slow leak in netty shuffle with spark cluster in > {{TransportRequestHandler.streamIds}} > In spark cluster, there are some reusable netty connections between two block > managers to get/send blocks between worker/drivers. These connections are > handled by the {{org.apache.spark.network.server.TransportRequestHandler}} in > server side. This handler keep tracking all the streamids negotiate by RPC > when shuffle data need transform in these two block managers and the streamid > is keeping increasing, and never get a chance to be deleted exception this > connection is dropped (seems never happen in normal running). > Here are some detail logs of this {{TransportRequestHandler}} (Note: we add > a log a print the total size of {{TransportRequestHandler.streamIds}}, the > log is "Current set size is N of > org.apache.spark.network.server.TransportRequestHandler@ADDRESS", this set > size is keeping increasing in our test) > {quote} > 15/04/22 21:00:16 DEBUG TransportServer: Shuffle server started on port :46288 > 15/04/22 21:00:16 INFO NettyBlockTransferService: Server created on 46288 > 15/04/22 21:00:31 INFO TransportRequestHandler: Created > TransportRequestHandler > org.apache.spark.network.server.TransportRequestHandler@29a4f3e7 > 15/04/22 21:00:32 TRACE MessageDecoder: Received message RpcRequest: > RpcRequest\{requestId=6655045571437304938, message=[B@59778678\} > 15/04/22 21:00:32 TRACE NettyBlockRpcServer: Received request: > OpenBlocks\{appId=app-20150422210016-, execId=, > blockIds=[broadcast_1_piece0]} > 15/04/22 21:00:32 TRACE NettyBlockRpcServer: Registered streamId > 1387459488000 with 1 buffers > 15/04/22 21:00:33 TRACE TransportRequestHandler: Sent result > RpcResponse\{requestId=6655045571437304938, response=[B@d2840b\} to client > /10.111.7.150:33802 > 15/04/22 21:00:33 TRACE MessageDecoder: Received message ChunkFetchRequest: > ChunkFetchRequest\{streamChunkId=StreamChunkId\{streamId=1387459488000, > chunkIndex=0}} > 15/04/22 21:00:33 TRACE TransportRequestHandler: Received req from > /10.111.7.150:33802 to fetch block StreamChunkId\{streamId=1387459488000, > chunkIndex=0\} > 15/04/22 21:00:33 INFO TransportRequestHandler: Current set size is 1 of > org.apache.spark.network.server.TransportRequestHandler@29a4f3e7 > 15/04/22 21:00:33 TRACE OneForOneStreamManager: Removing stream id > 1387459488000 > 15/04/22 21:00:33 TRACE TransportRequestHandler: Sent result > ChunkFetchSuccess\{streamChunkId=StreamChunkId\{streamId=1387459488000, > chunkIndex=0}, buffer=NioManagedBuffer\{buf=java.nio.HeapByteBuffer[pos=0 > lim=3839 cap=3839]}} to client /10.111.7.150:33802 > 15/04/22 21:00:34 TRACE MessageDecoder: Received message RpcRequest: > RpcRequest\{requestId=6660601528868866371, message=[B@42bed1b8\} > 15/04/22 21:00:34 TRACE NettyBlockRpcServer: Received request: > OpenBlocks\{appId=app-20150422210016-, execId=, > blockIds=[broadcast_3_piece0]} > 15/04/22 21:00:34 TRACE NettyBlockRpcServer: Registered streamId > 1387459488001 with 1 buffers > 15/04/22 21:00:34 TRACE TransportRequestHandler: Sent result > RpcResponse\{requestId=6660601528868866371, response=[B@7fa3fb60\} to client > /10.111.7.150:33802 > 15/04/22 21:00:34 TRACE MessageDecoder: Received message ChunkFetchRequest: > ChunkFetchRequest\{streamChunkId=StreamChunkId\{streamId=1387459488001, > chunkIndex=0}} > 15/04/22 21:00:34 TRACE TransportRequestHandler: Received req from > /10.111.7.150:33802 to fetch block StreamChunkId\{streamId=1387459488001, > chunkIndex=0\} > 15/04/22 21:00:34 INFO TransportRequestHandler: Current set size is 2 of > org.apache.spark.network.server.TransportRequestHandler@29a4f3e7 > 15/04/22 21:00:34 TRACE OneForOneStreamManager: Removing stream id > 1387459488001 > 15/04/22 21:00:34 TRACE TransportRequestHandler: Sent result > ChunkFetchSuccess\{streamChunkId=StreamChunkId\{streamId=1387459488001, > chunkIndex=0}, buffer=NioManagedBuffer\{buf=java.nio.HeapByteBuffer[pos=0 > lim=4277 cap=4277]}} to client /10.111.7.150:33802 > 15/04/22 21:00:34 TRACE MessageDecoder: Received message RpcRequest: > RpcReq
[jira] [Commented] (SPARK-7183) Memory leak in netty shuffle with spark standalone cluster
[ https://issues.apache.org/jira/browse/SPARK-7183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516765#comment-14516765 ] Apache Spark commented on SPARK-7183: - User 'viirya' has created a pull request for this issue: https://github.com/apache/spark/pull/5743 > Memory leak in netty shuffle with spark standalone cluster > -- > > Key: SPARK-7183 > URL: https://issues.apache.org/jira/browse/SPARK-7183 > Project: Spark > Issue Type: Bug > Components: Shuffle >Affects Versions: 1.3.0 >Reporter: Jack Hu > Labels: memory-leak, netty, shuffle > > There is slow leak in netty shuffle with spark cluster in > {{TransportRequestHandler.streamIds}} > In spark cluster, there are some reusable netty connections between two block > managers to get/send blocks between worker/drivers. These connections are > handled by the {{org.apache.spark.network.server.TransportRequestHandler}} in > server side. This handler keep tracking all the streamids negotiate by RPC > when shuffle data need transform in these two block managers and the streamid > is keeping increasing, and never get a chance to be deleted exception this > connection is dropped (seems never happen in normal running). > Here are some detail logs of this {{TransportRequestHandler}} (Note: we add > a log a print the total size of {{TransportRequestHandler.streamIds}}, the > log is "Current set size is N of > org.apache.spark.network.server.TransportRequestHandler@ADDRESS", this set > size is keeping increasing in our test) > {quote} > 15/04/22 21:00:16 DEBUG TransportServer: Shuffle server started on port :46288 > 15/04/22 21:00:16 INFO NettyBlockTransferService: Server created on 46288 > 15/04/22 21:00:31 INFO TransportRequestHandler: Created > TransportRequestHandler > org.apache.spark.network.server.TransportRequestHandler@29a4f3e7 > 15/04/22 21:00:32 TRACE MessageDecoder: Received message RpcRequest: > RpcRequest\{requestId=6655045571437304938, message=[B@59778678\} > 15/04/22 21:00:32 TRACE NettyBlockRpcServer: Received request: > OpenBlocks\{appId=app-20150422210016-, execId=, > blockIds=[broadcast_1_piece0]} > 15/04/22 21:00:32 TRACE NettyBlockRpcServer: Registered streamId > 1387459488000 with 1 buffers > 15/04/22 21:00:33 TRACE TransportRequestHandler: Sent result > RpcResponse\{requestId=6655045571437304938, response=[B@d2840b\} to client > /10.111.7.150:33802 > 15/04/22 21:00:33 TRACE MessageDecoder: Received message ChunkFetchRequest: > ChunkFetchRequest\{streamChunkId=StreamChunkId\{streamId=1387459488000, > chunkIndex=0}} > 15/04/22 21:00:33 TRACE TransportRequestHandler: Received req from > /10.111.7.150:33802 to fetch block StreamChunkId\{streamId=1387459488000, > chunkIndex=0\} > 15/04/22 21:00:33 INFO TransportRequestHandler: Current set size is 1 of > org.apache.spark.network.server.TransportRequestHandler@29a4f3e7 > 15/04/22 21:00:33 TRACE OneForOneStreamManager: Removing stream id > 1387459488000 > 15/04/22 21:00:33 TRACE TransportRequestHandler: Sent result > ChunkFetchSuccess\{streamChunkId=StreamChunkId\{streamId=1387459488000, > chunkIndex=0}, buffer=NioManagedBuffer\{buf=java.nio.HeapByteBuffer[pos=0 > lim=3839 cap=3839]}} to client /10.111.7.150:33802 > 15/04/22 21:00:34 TRACE MessageDecoder: Received message RpcRequest: > RpcRequest\{requestId=6660601528868866371, message=[B@42bed1b8\} > 15/04/22 21:00:34 TRACE NettyBlockRpcServer: Received request: > OpenBlocks\{appId=app-20150422210016-, execId=, > blockIds=[broadcast_3_piece0]} > 15/04/22 21:00:34 TRACE NettyBlockRpcServer: Registered streamId > 1387459488001 with 1 buffers > 15/04/22 21:00:34 TRACE TransportRequestHandler: Sent result > RpcResponse\{requestId=6660601528868866371, response=[B@7fa3fb60\} to client > /10.111.7.150:33802 > 15/04/22 21:00:34 TRACE MessageDecoder: Received message ChunkFetchRequest: > ChunkFetchRequest\{streamChunkId=StreamChunkId\{streamId=1387459488001, > chunkIndex=0}} > 15/04/22 21:00:34 TRACE TransportRequestHandler: Received req from > /10.111.7.150:33802 to fetch block StreamChunkId\{streamId=1387459488001, > chunkIndex=0\} > 15/04/22 21:00:34 INFO TransportRequestHandler: Current set size is 2 of > org.apache.spark.network.server.TransportRequestHandler@29a4f3e7 > 15/04/22 21:00:34 TRACE OneForOneStreamManager: Removing stream id > 1387459488001 > 15/04/22 21:00:34 TRACE TransportRequestHandler: Sent result > ChunkFetchSuccess\{streamChunkId=StreamChunkId\{streamId=1387459488001, > chunkIndex=0}, buffer=NioManagedBuffer\{buf=java.nio.HeapByteBuffer[pos=0 > lim=4277 cap=4277]}} to client /10.111.7.150:33802 > 15/04/22 21:00:34 TRACE MessageDecoder: Received message RpcRequest: > RpcRequest\{requestId=8454597410163901330, message=[B@19c673d1\} > 15/04/22 21:00:34 TRA