[jira] [Commented] (SPARK-14437) Spark using Netty RPC gets wrong address in some setups
[
https://issues.apache.org/jira/browse/SPARK-14437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15707201#comment-15707201
]
Alex Jiang commented on SPARK-14437:
[~hogeland] Did you get your issue resolved in 2.0.0? We are seeing a similar
issue if we run our app in intellij. If we run our app via command line, like
"jar -jar app.jar -c app.conf", everything was fine. However, it got issue when
running in intellij.
used by: java.lang.RuntimeException: Stream '/jars/classes' was not found.
at
org.apache.spark.network.client.TransportResponseHandler.handle(TransportResponseHandler.java:222)
at
org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:121)
at
org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:51)
at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
at
io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
at
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
at
org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:85)
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)
at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at
io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
at java.lang.Thread.run(Thread.java:745)
> Spark using Netty RPC gets wrong address in some setups
> ---
>
> Key: SPARK-14437
> URL: https://issues.apache.org/jira/browse/SPARK-14437
> Project: Spark
> Issue Type: Bug
> Components: Block Manager, Spark Core
>Affects Versions: 1.6.0, 1.6.1
> Environment: AWS, Docker, Flannel
>Reporter: Kevin Hogeland
>Assignee: Shixiong Zhu
> Fix For: 2.0.0
>
>
> Netty can't get the correct origin address in certain network setups. Spark
> should handle this, as relying on Netty correctly reporting all addresses
> leads to incompatible and unpredictable network states. We're currently using
> Docker with Flannel on AWS. Container communication looks something like:
> {{Container 1 (1.2.3.1) -> Docker host A (1.2.3.0) -> Docker host B (4.5.6.0)
> -> Container 2 (4.5.6.1)}}
> If the client in that setup is Container 1 (1.2.3.4), Netty channels from
> there to Container 2 will have a client address of 1.2.3.0.
> The {{RequestMessage}} object that is sent over the wire already contains a
> {{senderAddress}} field that the sender can use to specify their address. In
> {{NettyRpcEnv#internalReceive}}, this is replaced with the Netty client
> socket address when null. {{senderAddress}} in the messages sent from the
> executors is currently always null, meaning all messages will have these
> incorrect addresses (we've switched back to Akka as a temporary workaround
> for this). The executor should send its address explicitly so that the driver
> doesn't attempt to infer addresses based on possibly incorrect information
> from Netty.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-16246) Too many block-manager-slave-async-thread opened (TIMED_WAITING) for spark Kafka streaming
Alex Jiang created SPARK-16246: -- Summary: Too many block-manager-slave-async-thread opened (TIMED_WAITING) for spark Kafka streaming Key: SPARK-16246 URL: https://issues.apache.org/jira/browse/SPARK-16246 Project: Spark Issue Type: Bug Components: Spark Core, Streaming Affects Versions: 1.6.1 Reporter: Alex Jiang I don't know if our spark streaming issue is related to this (https://issues.apache.org/jira/browse/SPARK-15558). Basically we have one Kafka receiver on each executor, and it ran fine for a while. Then, the executor had a lot of waiting thread accumulated (Thread 1224: block-manager-slave-async-thread-pool-1083 (TIMED_WAITING)). And the executor kept open such new thread. Eventually, it reached the maximum number of the thread on that executor and Kafka receiver on that executor failed. Could someone please shed some light on this? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15558) Deadlock when retreiving shuffled cached data
[
https://issues.apache.org/jira/browse/SPARK-15558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352361#comment-15352361
]
Alex Jiang commented on SPARK-15558:
I don't know if our spark streaming issue is related to this.
Basically we have one Kafka receiver on each executor, and it ran fine for a
while. Then, the executor had a lot of waiting thread accumulated (Thread
1224: block-manager-slave-async-thread-pool-1083 (TIMED_WAITING)). And the
executor kept open such new thread. Eventually, it reached the maximum number
of the thread on that executor and Kafka receiver on that executor failed.
Could someone please shed some light on this?
> Deadlock when retreiving shuffled cached data
> -
>
> Key: SPARK-15558
> URL: https://issues.apache.org/jira/browse/SPARK-15558
> Project: Spark
> Issue Type: Bug
>Affects Versions: 1.6.1
>Reporter: Fabiano Francesconi
> Attachments: screenshot-1.png
>
>
> Spark-1.6.1-bin-hadoop2.6 hangs when trying to retrieving shuffled cached
> data from another host. The job I am currently executing is fetching data
> using async actions and persisting these RDDs into main memory (they all
> fit). Later on, at the point in which it is currently hanging, the
> application is retrieving this cached data but it hangs. The application,
> once the timeout set in the Await.results call is met, crashes.
> This problem is reproducible at every executing, although the point in which
> it hangs it is not.
> I have also tried activating:
> {code}
> spark.memory.useLegacyMode=true
> {code}
> as mentioned in SPARK-13566 guessing a similar deadlock as the one given
> between MemoryStore and BlockManager. Unfortunately, this didn't help.
> The only plausible (albeit debatable) solution would be to use speculation
> mode.
> Configuration:
> {code}
> /usr/local/tl/spark-latest/bin/spark-submit \
> --executor-memory 80G \
> --total-executor-cores 90 \
> --driver-memory 8G \
> {code}
> Stack trace:
> {code}
> "sparkExecutorActorSystem-akka.remote.default-remote-dispatcher-55" #293
> daemon prio=5 os_prio=0 tid=0x7f99d4004000 nid=0x4e80 waiting on
> condition [0x7f9946bfb000]
>java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x7f9b541a6570> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at
> scala.concurrent.forkjoin.ForkJoinPool.idleAwaitWork(ForkJoinPool.java:2135)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2067)
> at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> "sparkExecutorActorSystem-akka.remote.default-remote-dispatcher-54" #292
> daemon prio=5 os_prio=0 tid=0x7f99d4002000 nid=0x4e6d waiting on
> condition [0x7f98c86b6000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x7f9b541a6570> (a
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:2075)
> at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> "Executor task launch worker-43" #236 daemon prio=5 os_prio=0
> tid=0x7f9950001800 nid=0x4acc waiting on condition [0x7f9a2c4be000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x7fab3f081300> (a
> scala.concurrent.impl.Promise$CompletionLatch)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
> at
> scala.concurrent.impl.Promise$DefaultPromise.tryAwait(Promise.scala:202)
> at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:218)
> at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
> at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
> at
> scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> at scala.concurrent.Await$.result(package.scala:107)
> at
> org.apache.spark.network.BlockTransferService.fetchBlockSync(BlockTransferService.scala:102)
> at
> org.apache.spark.storage.BlockManager$$
[jira] [Commented] (SPARK-13288) [1.6.0] Memory leak in Spark streaming
[ https://issues.apache.org/jira/browse/SPARK-13288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15349714#comment-15349714 ] Alex Jiang commented on SPARK-13288: Agree. For DirectStream, we don't need to create multiple streams. [~jfc...@us.ibm.com] > [1.6.0] Memory leak in Spark streaming > -- > > Key: SPARK-13288 > URL: https://issues.apache.org/jira/browse/SPARK-13288 > Project: Spark > Issue Type: Bug > Components: Streaming >Affects Versions: 1.6.0 > Environment: Bare metal cluster > RHEL 6.6 >Reporter: JESSE CHEN > Labels: streaming > > Streaming in 1.6 seems to have a memory leak. > Running the same streaming app in Spark 1.5.1 and 1.6, all things equal, 1.6 > showed a gradual increasing processing time. > The app is simple: 1 Kafka receiver of tweet stream and 20 executors > processing the tweets in 5-second batches. > Spark 1.5.0 handles this smoothly and did not show increasing processing time > in the 40-minute test; but 1.6 showed increasing time about 8 minutes into > the test. Please see chart here: > https://ibm.box.com/s/7q4ulik70iwtvyfhoj1dcl4nc469b116 > I captured heap dumps in two version and did a comparison. I noticed the Byte > is using 50X more space in 1.5.1. > Here are some top classes in heap histogram and references. > Heap Histogram > > All Classes (excluding platform) > 1.6.0 Streaming 1.5.1 Streaming > Class Instance Count Total Size Class Instance Count Total > Size > class [B 84533,227,649,599 class [B5095 > 62,938,466 > class [C 44682 4,255,502 class [C130482 > 12,844,182 > class java.lang.reflect.Method90591,177,670 class > java.lang.String 130171 1,562,052 > > > References by TypeReferences by Type > > class [B [0x640039e38]class [B [0x6c020bb08] > > > Referrers by Type Referrers by Type > > Class Count Class Count > java.nio.HeapByteBuffer 3239 > sun.security.util.DerInputBuffer1233 > sun.security.util.DerInputBuffer 1233 > sun.security.util.ObjectIdentifier 620 > sun.security.util.ObjectIdentifier620 [[B 397 > [Ljava.lang.Object; 408 java.lang.reflect.Method > 326 > > The total size by class B is 3GB in 1.5.1 and only 60MB in 1.6.0. > The Java.nio.HeapByteBuffer referencing class did not show up in top in > 1.5.1. > I have also placed jstack output for 1.5.1 and 1.6.0 online..you can get them > here > https://ibm.box.com/sparkstreaming-jstack160 > https://ibm.box.com/sparkstreaming-jstack151 > Jesse -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13288) [1.6.0] Memory leak in Spark streaming
[ https://issues.apache.org/jira/browse/SPARK-13288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15346956#comment-15346956 ] Alex Jiang commented on SPARK-13288: [~jfc...@us.ibm.com] Did you use Kafka createStream or createDirectStream? > [1.6.0] Memory leak in Spark streaming > -- > > Key: SPARK-13288 > URL: https://issues.apache.org/jira/browse/SPARK-13288 > Project: Spark > Issue Type: Bug > Components: Streaming >Affects Versions: 1.6.0 > Environment: Bare metal cluster > RHEL 6.6 >Reporter: JESSE CHEN > Labels: streaming > > Streaming in 1.6 seems to have a memory leak. > Running the same streaming app in Spark 1.5.1 and 1.6, all things equal, 1.6 > showed a gradual increasing processing time. > The app is simple: 1 Kafka receiver of tweet stream and 20 executors > processing the tweets in 5-second batches. > Spark 1.5.0 handles this smoothly and did not show increasing processing time > in the 40-minute test; but 1.6 showed increasing time about 8 minutes into > the test. Please see chart here: > https://ibm.box.com/s/7q4ulik70iwtvyfhoj1dcl4nc469b116 > I captured heap dumps in two version and did a comparison. I noticed the Byte > is using 50X more space in 1.5.1. > Here are some top classes in heap histogram and references. > Heap Histogram > > All Classes (excluding platform) > 1.6.0 Streaming 1.5.1 Streaming > Class Instance Count Total Size Class Instance Count Total > Size > class [B 84533,227,649,599 class [B5095 > 62,938,466 > class [C 44682 4,255,502 class [C130482 > 12,844,182 > class java.lang.reflect.Method90591,177,670 class > java.lang.String 130171 1,562,052 > > > References by TypeReferences by Type > > class [B [0x640039e38]class [B [0x6c020bb08] > > > Referrers by Type Referrers by Type > > Class Count Class Count > java.nio.HeapByteBuffer 3239 > sun.security.util.DerInputBuffer1233 > sun.security.util.DerInputBuffer 1233 > sun.security.util.ObjectIdentifier 620 > sun.security.util.ObjectIdentifier620 [[B 397 > [Ljava.lang.Object; 408 java.lang.reflect.Method > 326 > > The total size by class B is 3GB in 1.5.1 and only 60MB in 1.6.0. > The Java.nio.HeapByteBuffer referencing class did not show up in top in > 1.5.1. > I have also placed jstack output for 1.5.1 and 1.6.0 online..you can get them > here > https://ibm.box.com/sparkstreaming-jstack160 > https://ibm.box.com/sparkstreaming-jstack151 > Jesse -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13288) [1.6.0] Memory leak in Spark streaming
[ https://issues.apache.org/jira/browse/SPARK-13288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15346951#comment-15346951 ] Alex Jiang commented on SPARK-13288: [~roberto_hashi...@hotmail.com] Thanks for your confirmation! > [1.6.0] Memory leak in Spark streaming > -- > > Key: SPARK-13288 > URL: https://issues.apache.org/jira/browse/SPARK-13288 > Project: Spark > Issue Type: Bug > Components: Streaming >Affects Versions: 1.6.0 > Environment: Bare metal cluster > RHEL 6.6 >Reporter: JESSE CHEN > Labels: streaming > > Streaming in 1.6 seems to have a memory leak. > Running the same streaming app in Spark 1.5.1 and 1.6, all things equal, 1.6 > showed a gradual increasing processing time. > The app is simple: 1 Kafka receiver of tweet stream and 20 executors > processing the tweets in 5-second batches. > Spark 1.5.0 handles this smoothly and did not show increasing processing time > in the 40-minute test; but 1.6 showed increasing time about 8 minutes into > the test. Please see chart here: > https://ibm.box.com/s/7q4ulik70iwtvyfhoj1dcl4nc469b116 > I captured heap dumps in two version and did a comparison. I noticed the Byte > is using 50X more space in 1.5.1. > Here are some top classes in heap histogram and references. > Heap Histogram > > All Classes (excluding platform) > 1.6.0 Streaming 1.5.1 Streaming > Class Instance Count Total Size Class Instance Count Total > Size > class [B 84533,227,649,599 class [B5095 > 62,938,466 > class [C 44682 4,255,502 class [C130482 > 12,844,182 > class java.lang.reflect.Method90591,177,670 class > java.lang.String 130171 1,562,052 > > > References by TypeReferences by Type > > class [B [0x640039e38]class [B [0x6c020bb08] > > > Referrers by Type Referrers by Type > > Class Count Class Count > java.nio.HeapByteBuffer 3239 > sun.security.util.DerInputBuffer1233 > sun.security.util.DerInputBuffer 1233 > sun.security.util.ObjectIdentifier 620 > sun.security.util.ObjectIdentifier620 [[B 397 > [Ljava.lang.Object; 408 java.lang.reflect.Method > 326 > > The total size by class B is 3GB in 1.5.1 and only 60MB in 1.6.0. > The Java.nio.HeapByteBuffer referencing class did not show up in top in > 1.5.1. > I have also placed jstack output for 1.5.1 and 1.6.0 online..you can get them > here > https://ibm.box.com/sparkstreaming-jstack160 > https://ibm.box.com/sparkstreaming-jstack151 > Jesse -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13288) [1.6.0] Memory leak in Spark streaming
[ https://issues.apache.org/jira/browse/SPARK-13288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15346736#comment-15346736 ] Alex Jiang commented on SPARK-13288: [~roberto_hashi...@hotmail.com] Did you get chance to try the 1.5.1? > [1.6.0] Memory leak in Spark streaming > -- > > Key: SPARK-13288 > URL: https://issues.apache.org/jira/browse/SPARK-13288 > Project: Spark > Issue Type: Bug > Components: Streaming >Affects Versions: 1.6.0 > Environment: Bare metal cluster > RHEL 6.6 >Reporter: JESSE CHEN > Labels: streaming > > Streaming in 1.6 seems to have a memory leak. > Running the same streaming app in Spark 1.5.1 and 1.6, all things equal, 1.6 > showed a gradual increasing processing time. > The app is simple: 1 Kafka receiver of tweet stream and 20 executors > processing the tweets in 5-second batches. > Spark 1.5.0 handles this smoothly and did not show increasing processing time > in the 40-minute test; but 1.6 showed increasing time about 8 minutes into > the test. Please see chart here: > https://ibm.box.com/s/7q4ulik70iwtvyfhoj1dcl4nc469b116 > I captured heap dumps in two version and did a comparison. I noticed the Byte > is using 50X more space in 1.5.1. > Here are some top classes in heap histogram and references. > Heap Histogram > > All Classes (excluding platform) > 1.6.0 Streaming 1.5.1 Streaming > Class Instance Count Total Size Class Instance Count Total > Size > class [B 84533,227,649,599 class [B5095 > 62,938,466 > class [C 44682 4,255,502 class [C130482 > 12,844,182 > class java.lang.reflect.Method90591,177,670 class > java.lang.String 130171 1,562,052 > > > References by TypeReferences by Type > > class [B [0x640039e38]class [B [0x6c020bb08] > > > Referrers by Type Referrers by Type > > Class Count Class Count > java.nio.HeapByteBuffer 3239 > sun.security.util.DerInputBuffer1233 > sun.security.util.DerInputBuffer 1233 > sun.security.util.ObjectIdentifier 620 > sun.security.util.ObjectIdentifier620 [[B 397 > [Ljava.lang.Object; 408 java.lang.reflect.Method > 326 > > The total size by class B is 3GB in 1.5.1 and only 60MB in 1.6.0. > The Java.nio.HeapByteBuffer referencing class did not show up in top in > 1.5.1. > I have also placed jstack output for 1.5.1 and 1.6.0 online..you can get them > here > https://ibm.box.com/sparkstreaming-jstack160 > https://ibm.box.com/sparkstreaming-jstack151 > Jesse -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-13288) [1.6.0] Memory leak in Spark streaming
[ https://issues.apache.org/jira/browse/SPARK-13288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15345987#comment-15345987 ] Alex Jiang commented on SPARK-13288: Similar issue seen on our development. > [1.6.0] Memory leak in Spark streaming > -- > > Key: SPARK-13288 > URL: https://issues.apache.org/jira/browse/SPARK-13288 > Project: Spark > Issue Type: Bug > Components: Streaming >Affects Versions: 1.6.0 > Environment: Bare metal cluster > RHEL 6.6 >Reporter: JESSE CHEN > Labels: streaming > > Streaming in 1.6 seems to have a memory leak. > Running the same streaming app in Spark 1.5.1 and 1.6, all things equal, 1.6 > showed a gradual increasing processing time. > The app is simple: 1 Kafka receiver of tweet stream and 20 executors > processing the tweets in 5-second batches. > Spark 1.5.0 handles this smoothly and did not show increasing processing time > in the 40-minute test; but 1.6 showed increasing time about 8 minutes into > the test. Please see chart here: > https://ibm.box.com/s/7q4ulik70iwtvyfhoj1dcl4nc469b116 > I captured heap dumps in two version and did a comparison. I noticed the Byte > is using 50X more space in 1.5.1. > Here are some top classes in heap histogram and references. > Heap Histogram > > All Classes (excluding platform) > 1.6.0 Streaming 1.5.1 Streaming > Class Instance Count Total Size Class Instance Count Total > Size > class [B 84533,227,649,599 class [B5095 > 62,938,466 > class [C 44682 4,255,502 class [C130482 > 12,844,182 > class java.lang.reflect.Method90591,177,670 class > java.lang.String 130171 1,562,052 > > > References by TypeReferences by Type > > class [B [0x640039e38]class [B [0x6c020bb08] > > > Referrers by Type Referrers by Type > > Class Count Class Count > java.nio.HeapByteBuffer 3239 > sun.security.util.DerInputBuffer1233 > sun.security.util.DerInputBuffer 1233 > sun.security.util.ObjectIdentifier 620 > sun.security.util.ObjectIdentifier620 [[B 397 > [Ljava.lang.Object; 408 java.lang.reflect.Method > 326 > > The total size by class B is 3GB in 1.5.1 and only 60MB in 1.6.0. > The Java.nio.HeapByteBuffer referencing class did not show up in top in > 1.5.1. > I have also placed jstack output for 1.5.1 and 1.6.0 online..you can get them > here > https://ibm.box.com/sparkstreaming-jstack160 > https://ibm.box.com/sparkstreaming-jstack151 > Jesse -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
