You can add -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation for nimbus and supervisor, so it can produce .gc log. Then you check your supervisor's gc log and nimbus's gc log. At the same time, you can check jstack files of nimbus and supervisor.
-----邮件原件----- 发件人: ght230 [mailto:[email protected]] 发送时间: 2015年12月18日 9:24 收件人: dev 抄送: user 主题: Re: Re: java.lang.OutOfMemoryError: Direct buffer memory Hi I have checked supervisor log and nimbus log, it seems nothing abnormal. ght230 From: Manu Zhang Date: 2015-12-18 08:27 To: dev CC: user Subject: Re: java.lang.OutOfMemoryError: Direct buffer memory It could be other issues that trigger these netty connection problems and finally OOM. Could you check your supervisor log and nimbus log as well ? On Thu, Dec 17, 2015 at 3:46 PM, ght230 <[email protected]> wrote: > Hi : > > I ran a simple wordcount topology on storm0.9.6, found 2 abnormal > behaviors 1. Netty reconnect so many time. > 2. direct memory continuously growing. and finally throw > java.lang.OutOfMemoryError. > > following is the unexpected information: > > 2015-12-15T21:30:18.546+0800 b.s.d.executor [INFO] Activating spout > emitSpout:(1) > 2015-12-15T21:30:18.578+0800 b.s.d.executor [INFO] Preparing bolt > __system:(-1) > 2015-12-15T21:30:18.614+0800 b.s.d.executor [INFO] Prepared bolt > __system:(-1) > 2015-12-15T21:38:37.453+0800 b.s.m.n.Client [ERROR] failed to send 943 > messages to Netty-Client-eda.local/192.168.137.2:6708: > java.lang.OutOfMemoryError: Direct buffer memory > 2015-12-15T21:38:37.457+0800 b.s.m.n.Client [ERROR] failed to send 944 > messages to Netty-Client-eda.local/192.168.137.2:6708: > java.nio.channels.ClosedChannelException > 2015-12-15T21:38:37.464+0800 b.s.m.n.Client [ERROR] failed to send 945 > messages to Netty-Client-eda.local/192.168.137.2:6708: > java.nio.channels.ClosedChannelException > 2015-12-15T21:38:37.472+0800 b.s.m.n.Client [ERROR] failed to send 944 > messages to Netty-Client-eda.local/192.168.137.2:6708: > java.nio.channels.ClosedChannelException > 2015-12-15T21:38:37.477+0800 b.s.m.n.Client [ERROR] failed to send 943 > messages to Netty-Client-eda.local/192.168.137.2:6708: > java.nio.channels.ClosedChannelException > 2015-12-15T21:38:37.479+0800 b.s.m.n.Client [ERROR] failed to send 938 > messages to Netty-Client-eda.local/192.168.137.2:6708: > java.nio.channels.ClosedChannelException > 2015-12-15T21:38:37.481+0800 b.s.m.n.Client [ERROR] failed to send 946 > messages to Netty-Client-eda.local/192.168.137.2:6708: > java.nio.channels.ClosedChannelException > 2015-12-15T21:38:37.482+0800 b.s.m.n.Client [ERROR] failed to send 943 > messages to Netty-Client-eda.local/192.168.137.2:6708: > java.nio.channels.ClosedChannelException > 2015-12-15T21:38:37.484+0800 b.s.m.n.Client [ERROR] failed to send 943 > messages to Netty-Client-eda.local/192.168.137.2:6708: > java.nio.channels.ClosedChannelException > 2015-12-15T21:38:37.485+0800 b.s.m.n.Client [ERROR] failed to send 941 > messages to Netty-Client-eda.local/192.168.137.2:6708: > java.nio.channels.ClosedChannelException > 2015-12-15T21:38:37.490+0800 b.s.m.n.Client [ERROR] failed to send 942 > messages to Netty-Client-eda.local/192.168.137.2:6708: > java.nio.channels.ClosedChannelException > 2015-12-15T21:38:37.504+0800 b.s.m.n.Client [ERROR] failed to send 941 > messages to Netty-Client-eda.local/192.168.137.2:6708: > java.nio.channels.ClosedChannelException > 2015-12-15T21:38:37.505+0800 b.s.m.n.Client [ERROR] failed to send 943 > messages to Netty-Client-eda.local/192.168.137.2:6708: > java.nio.channels.ClosedChannelException > 2015-12-15T21:38:37.507+0800 b.s.m.n.Client [ERROR] failed to send 944 > messages to Netty-Client-eda.local/192.168.137.2:6708: > java.nio.channels.ClosedChannelException > 2015-12-15T21:38:37.510+0800 b.s.m.n.Client [ERROR] failed to send 944 > messages to Netty-Client-eda.local/192.168.137.2:6708: > java.nio.channels.ClosedChannelException > 2015-12-15T21:38:37.516+0800 b.s.m.n.StormClientHandler [INFO] > Connection failed Netty-Client-eda.local/192.168.137.2:6708 > java.lang.OutOfMemoryError: Direct buffer memory at > java.nio.Bits.reserveMemory(Bits.java:658) ~[na:1.7.0_67] at > java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) > ~[na:1.7.0_67] > at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) > ~[na:1.7.0_67] at > org.apache.storm.netty.buffer.ChannelBuffers.directBuffer(ChannelBuffe > rs.java:167) > ~[storm-core-0.9.6.jar:0.9.6] > at > org.apache.storm.netty.buffer.ChannelBuffers.directBuffer(ChannelBuffe > rs.java:151) > ~[storm-core-0.9.6.jar:0.9.6] > at > backtype.storm.messaging.netty.MessageBatch.buffer(MessageBatch.java:8 > 4) > ~[storm-core-0.9.6.jar:0.9.6] > at > backtype.storm.messaging.netty.MessageEncoder.encode(MessageEncoder.ja > va:32) > ~[storm-core-0.9.6.jar:0.9.6] > at > org.apache.storm.netty.handler.codec.oneone.OneToOneEncoder.doEncode(O > neToOneEncoder.java:66) > ~[storm-core-0.9.6.jar:0.9.6] > at > org.apache.storm.netty.handler.codec.oneone.OneToOneEncoder.handleDown > stream(OneToOneEncoder.java:59) > ~[storm-core-0.9.6.jar:0.9.6] > at > org.apache.storm.netty.channel.DefaultChannelPipeline.sendDownstream(D > efaultChannelPipeline.java:591) > ~[storm-core-0.9.6.jar:0.9.6] > at > org.apache.storm.netty.channel.DefaultChannelPipeline.sendDownstream(D > efaultChannelPipeline.java:582) > ~[storm-core-0.9.6.jar:0.9.6] > at org.apache.storm.netty.channel.Channels.write(Channels.java:704) > ~[storm-core-0.9.6.jar:0.9.6] > at org.apache.storm.netty.channel.Channels.write(Channels.java:671) > ~[storm-core-0.9.6.jar:0.9.6] > at > org.apache.storm.netty.channel.AbstractChannel.write(AbstractChannel.j > ava:248) > ~[storm-core-0.9.6.jar:0.9.6] > at > backtype.storm.messaging.netty.Client.flushMessages(Client.java:323) > ~[storm-core-0.9.6.jar:0.9.6] > at backtype.storm.messaging.netty.Client.send(Client.java:250) > ~[storm-core-0.9.6.jar:0.9.6] > at backtype.storm.utils.TransferDrainer.send(TransferDrainer.java:54) > ~[storm-core-0.9.6.jar:0.9.6] > at > backtype.storm.daemon.worker$mk_transfer_tuples_handler$fn__4610$fn__4 > 611.invoke(worker.clj:338) > ~[storm-core-0.9.6.jar:0.9.6] > at > backtype.storm.daemon.worker$mk_transfer_tuples_handler$fn__4610.invok > e(worker.clj:336) > ~[storm-core-0.9.6.jar:0.9.6] > at > backtype.storm.disruptor$clojure_handler$reify__1445.onEvent(disruptor > .clj:58) > ~[storm-core-0.9.6.jar:0.9.6] > at > backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueu > e.java:132) > ~[storm-core-0.9.6.jar:0.9.6] > at > backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(Disrupto > rQueue.java:106) > ~[storm-core-0.9.6.jar:0.9.6] > at > backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor > .clj:80) > ~[storm-core-0.9.6.jar:0.9.6] > at > backtype.storm.disruptor$consume_loop_STAR_$fn__1458.invoke(disruptor. > clj:94) > ~[storm-core-0.9.6.jar:0.9.6] > at backtype.storm.util$async_loop$fn__460.invoke(util.clj:463) > ~[storm-core-0.9.6.jar:0.9.6] > at clojure.lang.AFn.run(AFn.java:24) ~[clojure-1.5.1.jar:na] at > java.lang.Thread.run(Thread.java:745) [na:1.7.0_67] > 2015-12-15T21:38:37.516+0800 b.s.m.n.Client [ERROR] failed to send 945 > messages to Netty-Client-eda.local/192.168.137.2:6708: > java.nio.channels.ClosedChannelException > 2015-12-15T21:38:37.521+0800 b.s.m.n.Client [ERROR] failed to send 944 > messages to Netty-Client-eda.local/192.168.137.2:6708: > java.nio.channels.ClosedChannelException > > and my configure Fragment in storm.yaml as following: > > #topology.skip.missing.kryo.registrations: false > storm.messaging.netty.flush.check.interval.ms: 100 > storm.messaging.transport: "backtype.storm.messaging.netty.Context" > storm.messaging.netty.server_worker_threads: 1 > storm.messaging.netty.client_worker_threads: 1 > #storm.messaging.netty.buffer_size: 5242880 > storm.messaging.netty.transfer.batch.size: 462144 > storm.messaging.netty.buffer_size: 5242880 > storm.messaging.netty.max_retries: 100 > storm.messaging.netty.max_wait_ms: 1000 > storm.messaging.netty.min_wait_ms: 100 > topology.backpressure.enable: false > #storm.scheduler: "backtype.storm.scheduler.DefaultScheduler" > worker.childopts: "-Xmx768m -XX:NewSize=128m -XX:MaxNewSize=128m > -XX:PermSize=64m -XX:MaxPermSize=64m -XX:MaxDirectMemorySize=128m > -Djava.net.preferIPv4Stack=true -XX:+UseParNewGC -XX:+PrintGCDetails > -XX:+PrintGCTimeStamps -verbose:gc > -Xloggc:/ffcs/gc/gc-storm-worker-%ID%.log > -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/ffcs/gc/heapdump" > #worker.childopts: "-Xmx1024m" > supervisor.childopts: "-Xmx256m" > nimbus.childopts: "-Xmx768m" > ui.childopts: "-Xmx768m" > nimbus.thrift.threads: 256 > > > Anyone knows how to solve these 2 problems? > > > > ght230 >
