It could be other issues that trigger these netty connection problems and finally OOM. Could you check your supervisor log and nimbus log as well ?
On Thu, Dec 17, 2015 at 3:46 PM, ght230 <[email protected]> wrote: > Hi : > > I ran a simple wordcount topology on storm0.9.6, found 2 abnormal behaviors > 1. Netty reconnect so many time. > 2. direct memory continuously growing. and finally throw > java.lang.OutOfMemoryError. > > following is the unexpected information: > > 2015-12-15T21:30:18.546+0800 b.s.d.executor [INFO] Activating spout > emitSpout:(1) > 2015-12-15T21:30:18.578+0800 b.s.d.executor [INFO] Preparing bolt > __system:(-1) > 2015-12-15T21:30:18.614+0800 b.s.d.executor [INFO] Prepared bolt > __system:(-1) > 2015-12-15T21:38:37.453+0800 b.s.m.n.Client [ERROR] failed to send 943 > messages to Netty-Client-eda.local/192.168.137.2:6708: > java.lang.OutOfMemoryError: Direct buffer memory > 2015-12-15T21:38:37.457+0800 b.s.m.n.Client [ERROR] failed to send 944 > messages to Netty-Client-eda.local/192.168.137.2:6708: > java.nio.channels.ClosedChannelException > 2015-12-15T21:38:37.464+0800 b.s.m.n.Client [ERROR] failed to send 945 > messages to Netty-Client-eda.local/192.168.137.2:6708: > java.nio.channels.ClosedChannelException > 2015-12-15T21:38:37.472+0800 b.s.m.n.Client [ERROR] failed to send 944 > messages to Netty-Client-eda.local/192.168.137.2:6708: > java.nio.channels.ClosedChannelException > 2015-12-15T21:38:37.477+0800 b.s.m.n.Client [ERROR] failed to send 943 > messages to Netty-Client-eda.local/192.168.137.2:6708: > java.nio.channels.ClosedChannelException > 2015-12-15T21:38:37.479+0800 b.s.m.n.Client [ERROR] failed to send 938 > messages to Netty-Client-eda.local/192.168.137.2:6708: > java.nio.channels.ClosedChannelException > 2015-12-15T21:38:37.481+0800 b.s.m.n.Client [ERROR] failed to send 946 > messages to Netty-Client-eda.local/192.168.137.2:6708: > java.nio.channels.ClosedChannelException > 2015-12-15T21:38:37.482+0800 b.s.m.n.Client [ERROR] failed to send 943 > messages to Netty-Client-eda.local/192.168.137.2:6708: > java.nio.channels.ClosedChannelException > 2015-12-15T21:38:37.484+0800 b.s.m.n.Client [ERROR] failed to send 943 > messages to Netty-Client-eda.local/192.168.137.2:6708: > java.nio.channels.ClosedChannelException > 2015-12-15T21:38:37.485+0800 b.s.m.n.Client [ERROR] failed to send 941 > messages to Netty-Client-eda.local/192.168.137.2:6708: > java.nio.channels.ClosedChannelException > 2015-12-15T21:38:37.490+0800 b.s.m.n.Client [ERROR] failed to send 942 > messages to Netty-Client-eda.local/192.168.137.2:6708: > java.nio.channels.ClosedChannelException > 2015-12-15T21:38:37.504+0800 b.s.m.n.Client [ERROR] failed to send 941 > messages to Netty-Client-eda.local/192.168.137.2:6708: > java.nio.channels.ClosedChannelException > 2015-12-15T21:38:37.505+0800 b.s.m.n.Client [ERROR] failed to send 943 > messages to Netty-Client-eda.local/192.168.137.2:6708: > java.nio.channels.ClosedChannelException > 2015-12-15T21:38:37.507+0800 b.s.m.n.Client [ERROR] failed to send 944 > messages to Netty-Client-eda.local/192.168.137.2:6708: > java.nio.channels.ClosedChannelException > 2015-12-15T21:38:37.510+0800 b.s.m.n.Client [ERROR] failed to send 944 > messages to Netty-Client-eda.local/192.168.137.2:6708: > java.nio.channels.ClosedChannelException > 2015-12-15T21:38:37.516+0800 b.s.m.n.StormClientHandler [INFO] Connection > failed Netty-Client-eda.local/192.168.137.2:6708 > java.lang.OutOfMemoryError: Direct buffer memory > at java.nio.Bits.reserveMemory(Bits.java:658) ~[na:1.7.0_67] > at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) > ~[na:1.7.0_67] > at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) ~[na:1.7.0_67] > at > org.apache.storm.netty.buffer.ChannelBuffers.directBuffer(ChannelBuffers.java:167) > ~[storm-core-0.9.6.jar:0.9.6] > at > org.apache.storm.netty.buffer.ChannelBuffers.directBuffer(ChannelBuffers.java:151) > ~[storm-core-0.9.6.jar:0.9.6] > at > backtype.storm.messaging.netty.MessageBatch.buffer(MessageBatch.java:84) > ~[storm-core-0.9.6.jar:0.9.6] > at > backtype.storm.messaging.netty.MessageEncoder.encode(MessageEncoder.java:32) > ~[storm-core-0.9.6.jar:0.9.6] > at > org.apache.storm.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:66) > ~[storm-core-0.9.6.jar:0.9.6] > at > org.apache.storm.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59) > ~[storm-core-0.9.6.jar:0.9.6] > at > org.apache.storm.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) > ~[storm-core-0.9.6.jar:0.9.6] > at > org.apache.storm.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582) > ~[storm-core-0.9.6.jar:0.9.6] > at org.apache.storm.netty.channel.Channels.write(Channels.java:704) > ~[storm-core-0.9.6.jar:0.9.6] > at org.apache.storm.netty.channel.Channels.write(Channels.java:671) > ~[storm-core-0.9.6.jar:0.9.6] > at > org.apache.storm.netty.channel.AbstractChannel.write(AbstractChannel.java:248) > ~[storm-core-0.9.6.jar:0.9.6] > at backtype.storm.messaging.netty.Client.flushMessages(Client.java:323) > ~[storm-core-0.9.6.jar:0.9.6] > at backtype.storm.messaging.netty.Client.send(Client.java:250) > ~[storm-core-0.9.6.jar:0.9.6] > at backtype.storm.utils.TransferDrainer.send(TransferDrainer.java:54) > ~[storm-core-0.9.6.jar:0.9.6] > at > backtype.storm.daemon.worker$mk_transfer_tuples_handler$fn__4610$fn__4611.invoke(worker.clj:338) > ~[storm-core-0.9.6.jar:0.9.6] > at > backtype.storm.daemon.worker$mk_transfer_tuples_handler$fn__4610.invoke(worker.clj:336) > ~[storm-core-0.9.6.jar:0.9.6] > at > backtype.storm.disruptor$clojure_handler$reify__1445.onEvent(disruptor.clj:58) > ~[storm-core-0.9.6.jar:0.9.6] > at > backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:132) > ~[storm-core-0.9.6.jar:0.9.6] > at > backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:106) > ~[storm-core-0.9.6.jar:0.9.6] > at > backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) > ~[storm-core-0.9.6.jar:0.9.6] > at > backtype.storm.disruptor$consume_loop_STAR_$fn__1458.invoke(disruptor.clj:94) > ~[storm-core-0.9.6.jar:0.9.6] > at backtype.storm.util$async_loop$fn__460.invoke(util.clj:463) > ~[storm-core-0.9.6.jar:0.9.6] > at clojure.lang.AFn.run(AFn.java:24) ~[clojure-1.5.1.jar:na] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67] > 2015-12-15T21:38:37.516+0800 b.s.m.n.Client [ERROR] failed to send 945 > messages to Netty-Client-eda.local/192.168.137.2:6708: > java.nio.channels.ClosedChannelException > 2015-12-15T21:38:37.521+0800 b.s.m.n.Client [ERROR] failed to send 944 > messages to Netty-Client-eda.local/192.168.137.2:6708: > java.nio.channels.ClosedChannelException > > and my configure Fragment in storm.yaml as following: > > #topology.skip.missing.kryo.registrations: false > storm.messaging.netty.flush.check.interval.ms: 100 > storm.messaging.transport: "backtype.storm.messaging.netty.Context" > storm.messaging.netty.server_worker_threads: 1 > storm.messaging.netty.client_worker_threads: 1 > #storm.messaging.netty.buffer_size: 5242880 > storm.messaging.netty.transfer.batch.size: 462144 > storm.messaging.netty.buffer_size: 5242880 > storm.messaging.netty.max_retries: 100 > storm.messaging.netty.max_wait_ms: 1000 > storm.messaging.netty.min_wait_ms: 100 > topology.backpressure.enable: false > #storm.scheduler: "backtype.storm.scheduler.DefaultScheduler" > worker.childopts: "-Xmx768m -XX:NewSize=128m -XX:MaxNewSize=128m > -XX:PermSize=64m -XX:MaxPermSize=64m -XX:MaxDirectMemorySize=128m > -Djava.net.preferIPv4Stack=true -XX:+UseParNewGC -XX:+PrintGCDetails > -XX:+PrintGCTimeStamps -verbose:gc > -Xloggc:/ffcs/gc/gc-storm-worker-%ID%.log -XX:+HeapDumpOnOutOfMemoryError > -XX:HeapDumpPath=/ffcs/gc/heapdump" > #worker.childopts: "-Xmx1024m" > supervisor.childopts: "-Xmx256m" > nimbus.childopts: "-Xmx768m" > ui.childopts: "-Xmx768m" > nimbus.thrift.threads: 256 > > > Anyone knows how to solve these 2 problems? > > > > ght230 >
