HI! For me it seems that the your HBase bolt is loosing the connection with zookeeper and it tries over and over to connect via thread. Please check your zookeeper health. Do your HBase cluster and Storm cluster are using the same Zookeeper? How many Hbase region servers and how many storm workers are you using? Depending on these numbers and your workload you can overkill your zk cluster. I hope that these help. Florin
On Sun, Jun 12, 2016 at 11:11 AM, Satish Duggana <satish.dugg...@gmail.com> wrote: > Hi, > Your message says it is throwing OutOfMemory error. So, you should look > into what is causing that. It may not be really because of storm but it may > be because of application code also. You may want to use ` > -XX:+HeapDumpOnOutOfMemoryError` and `-XX:HeapDumpPath=/worker/dumps` to > dump heap on outofmemory and analyze what is causing OutOfMemory. > > Thanks, > Satish. > > On Sun, Jun 12, 2016 at 11:34 AM, fanxi...@travelsky.com < > fanxi...@travelsky.com> wrote: > >> Hi Wenwei >> >> Actually, I did not new any thread in my bolt. The error reported just >> comes from storm core itself. >> >> ------------------------------ >> >> I t seems you created too many threads, that >> cause no available thread resource. >> >> Sent from my iPhone >> >> On Jun 12, 2016, at 10:18, "fanxi...@travelsky.com" < >> fanxi...@travelsky.com> wrote: >> >> *Hi user,* >> >> *I have a topology to write into HBase. Every time I submitted the >> topology, it runned well. But after a well, for example, one or two days, >> the topology always reports an execption like below:* >> >> java.lang.OutOfMemoryError: unable to create new native thread at >> java.lang.Thread.start0(Native Method) at >> java.lang.Thread.start(Thread.java:714) at >> org.apache.zookeeper.ClientCnxn.start(ClientCnxn.java:406) at >> org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:450) at >> org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:380) at >> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.checkZk(RecoverableZooKeeper.java:140) >> at >> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.<init>(RecoverableZooKeeper.java:127) >> at org.apache.hadoop.hbase.zookeeper.ZKUtil.connect(ZKUtil.java:132) at >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:165) >> at >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:134) >> at >> org.apache.hadoop.hbase.catalog.CatalogTracker.<init>(CatalogTracker.java:179) >> at >> org.apache.hadoop.hbase.catalog.CatalogTracker.<init>(CatalogTracker.java:153) >> at >> org.apache.hadoop.hbase.catalog.CatalogTracker.<init>(CatalogTracker.java:135) >> at >> org.apache.hadoop.hbase.client.HBaseAdmin.getCatalogTracker(HBaseAdmin.java:234) >> at >> org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:306) >> at >> com.travelsky.roc.hbase.utils.HBaseUtils.isTableAvailable(HBaseUtils.java:22) >> at >> com.travelsky.roc.hbase.bolt.HBaseSinkBolt.execute(HBaseSinkBolt.java:279) >> at >> backtype.storm.daemon.executor$fn__5641$tuple_action_fn__5643.invoke(executor.clj:631) >> at >> backtype.storm.daemon.executor$mk_task_receiver$fn__5564.invoke(executor.clj:399) >> at >> backtype.storm.disruptor$clojure_handler$reify__745.onEvent(disruptor.clj:58) >> at >> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125) >> at >> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) >> at >> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) >> at >> backtype.storm.daemon.executor$fn__5641$fn__5653$fn__5700.invoke(executor.clj:746) >> at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431) at >> clojure.lang.AFn.run(AFn.java:24) at >> java.lang.Thread.run(Thread.java:745)*then >> the topology runs very slow.* >> *I took a look at the log, it is full of the information like below:* >> 2016-06-12 10:16:04 o.a.h.h.z.RecoverableZooKeeper [INFO] Process >> identifier=catalogtracker-on-hconnection-0x5ade861c connecting to ZooKeeper >> ensemble=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181 >> 2016-06-12 10:16:04 o.a.z.ZooKeeper [INFO] Initiating client connection, >> connectString=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181 >> sessionTimeout=120000 watcher=catalogtracker-on-hconnection-0x5ade861c, >> quorum=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181, baseZNode=/hbasenew >> 2016-06-12 10:16:04 o.a.z.ClientCnxn [INFO] Opening socket connection to >> server r720m8-hdp/10.6.116.3:2181. Will not attempt to authenticate >> using SASL (unknown error) >> 2016-06-12 10:16:04 o.a.z.ClientCnxn [INFO] Socket connection established >> to r720m8-hdp/10.6.116.3:2181, initiating session >> 2016-06-12 10:16:04 o.a.z.ClientCnxn [INFO] Session establishment >> complete on server r720m8-hdp/10.6.116.3:2181, sessionid = >> 0x15138b0b2df471f, negotiated timeout = 120000 >> 2016-06-12 10:16:04 o.a.z.ZooKeeper [INFO] Session: 0x15138b0b2df471f >> closed >> 2016-06-12 10:16:04 o.a.z.ClientCnxn [INFO] EventThread shut down >> 2016-06-12 10:16:07 o.a.h.h.z.RecoverableZooKeeper [INFO] Process >> identifier=catalogtracker-on-hconnection-0x5ade861c connecting to ZooKeeper >> ensemble=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181 >> 2016-06-12 10:16:07 o.a.z.ZooKeeper [INFO] Initiating client connection, >> connectString=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181 >> sessionTimeout=120000 watcher=catalogtracker-on-hconnection-0x5ade861c, >> quorum=r720m6-hdp:2181,r720m8-hdp:2181,r720n5-hdp:2181, baseZNode=/hbasenew >> 2016-06-12 10:16:07 o.a.z.ClientCnxn [INFO] Opening socket connection to >> server r720m8-hdp/10.6.116.3:2181. Will not attempt to authenticate >> using SASL (unknown error) >> 2016-06-12 10:16:07 o.a.z.ClientCnxn [INFO] Socket connection established >> to r720m8-hdp/10.6.116.3:2181, initiating session >> 2016-06-12 10:16:07 o.a.z.ClientCnxn [INFO] Session establishment >> complete on server r720m8-hdp/10.6.116.3:2181, sessionid = >> 0x15138b0b2df473f, negotiated timeout = 120000 >> 2016-06-12 10:16:07 o.a.z.ZooKeeper [INFO] Session: 0x15138b0b2df473f >> closed >> 2016-06-12 10:16:07 o.a.z.ClientCnxn [INFO] EventThread shut down >> >> *Anyone comes across this problem? Thanks for your hints.* >> >> ------------------------------ >> Joshua >> 2016-06-12 10:12:33 >> >> >> >
