Can you us your zookeeper logs and config? On Dec 13, 2016 2:32 AM, "Joaquin Menchaca" <[email protected]> wrote:
> I am able to connect to the zookeeper: > > # printf "%s\n" $(echo ruok | nc zookeeper-stormcluster.gobalto.com 2181) > imok > > But not always. I looked at the zookeeper server, t3.large now, and the > CPU jumps around, sometimes 8%, sometimes 20%-35%, ocassionally 110%. > > I am not sure at this point where to look, it was working up until > recently. > > > On Mon, Dec 12, 2016 at 3:53 PM, Mostafa Gomaa <[email protected]> wrote: > >> Yes, your nimbus machine can't connect to zookeeper as per the log. Check >> out that zookeeper is running and that there aren't any networking issues >> between the two machines. >> >> On Dec 13, 2016 1:49 AM, "Joaquin Menchaca" <[email protected]> >> wrote: >> >>> Storm.yaml (Storm 1.0.2): >>> >>> storm.zookeeper.servers: >>> - "zookeeper-stormcluster.gobalto.com" >>> >>> nimbus.seeds: ["nimbus-stormcluster.gobalto.com", " >>> nimbus2-stormcluster.gobalto.com"] >>> nimbus.childopts: "-Xmx1024m -Djava.net.preferIPv4Stack=true" >>> ui.childopts: "-Xmx768m -Djava.net.preferIPv4Stack=true" >>> >>> supervisor.childopts: "-Djava.net.preferIPv4Stack=true" >>> worker.childopts: "-Xmx768m -Djava.net.preferIPv4Stack=true" >>> supervisor.slots.ports: >>> - 6700 >>> - 6701 >>> - 6702 >>> - 6703 >>> >>> storm.local.dir: "/app/storm" >>> >>> >>> Maybe my zookeeper is down or corrupt: >>> >>> 2016-12-12 23:46:50.628 o.a.s.s.o.a.c.ConnectionState [ERROR] Connection >>> timed out for connection string (zookeeper-stormcluster.gobalto.com:2181) >>> and timeout (15000) / elapsed (16327) >>> org.apache.storm.shade.org.apache.curator.CuratorConnectionLossException: >>> KeeperErrorCode = ConnectionLoss >>> at org.apache.storm.shade.org.apache.curator.ConnectionState.ch >>> eckTimeouts(ConnectionState.java:197) >>> at org.apache.storm.shade.org.apache.curator.ConnectionState.ge >>> tZooKeeper(ConnectionState.java:88) >>> at org.apache.storm.shade.org.apache.curator.CuratorZookeeperCl >>> ient.getZooKeeper(CuratorZookeeperClient.java:116) >>> at org.apache.storm.shade.org.apache.curator.framework.imps.Cur >>> atorFrameworkImpl.getZooKeeper(CuratorFrameworkImpl.java:489) >>> at org.apache.storm.shade.org.apache.curator.framework.imps.Exi >>> stsBuilderImpl$3.call(ExistsBuilderImpl.java:226) >>> at org.apache.storm.shade.org.apache.curator.framework.imps.Exi >>> stsBuilderImpl$3.call(ExistsBuilderImpl.java:215) >>> at org.apache.storm.shade.org.apache.curator.RetryLoop.callWith >>> Retry(RetryLoop.java:108) >>> at org.apache.storm.shade.org.apache.curator.framework.imps.Exi >>> stsBuilderImpl.pathInForegroundStandard(ExistsBuilderImpl.java:212) >>> at org.apache.storm.shade.org.apache.curator.framework.imps.Exi >>> stsBuilderImpl.pathInForeground(ExistsBuilderImpl.java:205) >>> at org.apache.storm.shade.org.apache.curator.framework.imps.Exi >>> stsBuilderImpl.forPath(ExistsBuilderImpl.java:168) >>> at org.apache.storm.shade.org.apache.curator.framework.imps.Exi >>> stsBuilderImpl.forPath(ExistsBuilderImpl.java:39) >>> at org.apache.storm.zookeeper$exists_node_QMARK_$fn__2090.invok >>> e(zookeeper.clj:109) >>> at org.apache.storm.zookeeper$exists_node_QMARK_.invoke(zookeep >>> er.clj:106) >>> at org.apache.storm.zookeeper$mkdirs.invoke(zookeeper.clj:126) >>> at org.apache.storm.cluster_state.zookeeper_state_factory$_mkSt >>> ate.invoke(zookeeper_state_factory.clj:29) >>> at >>> org.apache.storm.cluster_state.zookeeper_state_factory.mkState(Unknown >>> Source) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce >>> ssorImpl.java:62) >>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe >>> thodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:498) >>> at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) >>> at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) >>> at org.apache.storm.cluster$mk_distributed_cluster_state.doInvo >>> ke(cluster.clj:46) >>> at clojure.lang.RestFn.invoke(RestFn.java:559) >>> at org.apache.storm.cluster$mk_storm_cluster_state.doInvoke(clu >>> ster.clj:250) >>> at clojure.lang.RestFn.invoke(RestFn.java:486) >>> at org.apache.storm.daemon.nimbus$nimbus_data.invoke(nimbus.clj:176) >>> at org.apache.storm.daemon.nimbus$fn__7166$exec_fn__2466__auto_ >>> ___7167.invoke(nimbus.clj:1349) >>> at clojure.lang.AFn.applyToHelper(AFn.java:156) >>> at clojure.lang.AFn.applyTo(AFn.java:144) >>> at clojure.core$apply.invoke(core.clj:630) >>> at org.apache.storm.daemon.nimbus$fn__7166$service_handler__741 >>> 0.doInvoke(nimbus.clj:1346) >>> at clojure.lang.RestFn.invoke(RestFn.java:421) >>> at org.apache.storm.daemon.nimbus$launch_server_BANG_.invoke(ni >>> mbus.clj:2193) >>> at org.apache.storm.daemon.nimbus$_launch.invoke(nimbus.clj:2226) >>> at org.apache.storm.daemon.nimbus$_main.invoke(nimbus.clj:2249) >>> at clojure.lang.AFn.applyToHelper(AFn.java:152) >>> at clojure.lang.AFn.applyTo(AFn.java:144) >>> at org.apache.storm.daemon.nimbus.main(Unknown Source) >>> 2016-12-12 23:46:50.862 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket >>> connection to server zookeeper-stormcluster.gobalto.com/10.110.20.8:2181. >>> Will not attempt to authenticate using SASL (unknown error) >>> 2016-12-12 23:46:50.885 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket >>> connection established to zookeeper-stormcluster.gobalto >>> .com/10.110.20.8:2181, initiating session >>> 2016-12-12 23:46:50.886 o.a.s.s.o.a.z.ClientCnxn [INFO] Unable to read >>> additional data from server sessionid 0x0, likely server has closed socket, >>> closing socket connection and attempting reconnect >>> 2016-12-12 23:46:51.578 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket >>> connection to server zookeeper-stormcluster.gobalto.com/10.110.20.8:2181. >>> Will not attempt to authenticate using SASL (unknown error) >>> 2016-12-12 23:46:51.602 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket >>> connection established to zookeeper-stormcluster.gobalto >>> .com/10.110.20.8:2181, initiating session >>> 2016-12-12 23:46:51.603 o.a.s.s.o.a.z.ClientCnxn [INFO] Unable to read >>> additional data from server sessionid 0x0, likely server has closed socket, >>> closing socket connection and attempting reconnect >>> 2016-12-12 23:46:52.041 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket >>> connection to server zookeeper-stormcluster.gobalto.com/10.110.20.8:2181. >>> Will not attempt to authenticate using SASL (unknown error) >>> 2016-12-12 23:46:52.060 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket >>> connection established to zookeeper-stormcluster.gobalto >>> .com/10.110.20.8:2181, initiating session >>> 2016-12-12 23:46:52.061 o.a.s.s.o.a.z.ClientCnxn [INFO] Unable to read >>> additional data from server sessionid 0x0, likely server has closed socket, >>> closing socket connection and attempting reconnect >>> 2016-12-12 23:46:52.958 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket >>> connection to server zookeeper-stormcluster.gobalto.com/10.110.20.8:2181. >>> Will not attempt to authenticate using SASL (unknown error) >>> 2016-12-12 23:46:52.959 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket >>> connection established to zookeeper-stormcluster.gobalto >>> .com/10.110.20.8:2181, initiating session >>> 2016-12-12 23:46:53.003 o.a.s.s.o.a.z.ClientCnxn [WARN] Session 0x0 for >>> server zookeeper-stormcluster.gobalto.com/10.110.20.8:2181, unexpected >>> error, closing socket connection and attempting reconnect >>> java.io.IOException: Connection reset by peer >>> at sun.nio.ch.FileDispatcherImpl.read0(Native Method) >>> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) >>> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) >>> at sun.nio.ch.IOUtil.read(IOUtil.java:192) >>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) >>> at org.apache.storm.shade.org.apache.zookeeper.ClientCnxnSocket >>> NIO.doIO(ClientCnxnSocketNIO.java:68) >>> at org.apache.storm.shade.org.apache.zookeeper.ClientCnxnSocket >>> NIO.doTransport(ClientCnxnSocketNIO.java:366) >>> at org.apache.storm.shade.org.apache.zookeeper.ClientCnxn$SendT >>> hread.run(ClientCnxn.java:1081) >>> 2016-12-12 23:46:53.130 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket >>> connection to server zookeeper-stormcluster.gobalto.com/10.110.20.8:2181. >>> Will not attempt to authenticate using SASL (unknown error) >>> 2016-12-12 23:46:53.146 o.a.s.s.o.a.z.ClientCnxn [WARN] Session 0x0 for >>> server null, unexpected error, closing socket connection and attempting >>> reconnect >>> java.net.ConnectException: Connection refused >>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >>> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl >>> .java:717) >>> at org.apache.storm.shade.org.apache.zookeeper.ClientCnxnSocket >>> NIO.doTransport(ClientCnxnSocketNIO.java:361) >>> at org.apache.storm.shade.org.apache.zookeeper.ClientCnxn$SendT >>> hread.run(ClientCnxn.java:1081) >>> 2016-12-12 23:46:53.463 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket >>> connection to server zookeeper-stormcluster.gobalto.com/10.110.20.8:2181. >>> Will not attempt to authenticate using SASL (unknown error) >>> 2016-12-12 23:46:53.474 o.a.s.s.o.a.z.ClientCnxn [WARN] Session 0x0 for >>> server null, unexpected error, closing socket connection and attempting >>> reconnect >>> java.net.ConnectException: Connection refused >>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >>> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl >>> .java:717) >>> at org.apache.storm.shade.org.apache.zookeeper.ClientCnxnSocket >>> NIO.doTransport(ClientCnxnSocketNIO.java:361) >>> at org.apache.storm.shade.org.apache.zookeeper.ClientCnxn$SendT >>> hread.run(ClientCnxn.java:1081) >>> 2016-12-12 23:46:54.291 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket >>> connection to server zookeeper-stormcluster.gobalto.com/10.110.20.8:2181. >>> Will not attempt to authenticate using SASL (unknown error) >>> 2016-12-12 23:46:54.292 o.a.s.s.o.a.z.ClientCnxn [WARN] Session 0x0 for >>> server null, unexpected error, closing socket connection and attempting >>> reconnect >>> java.net.ConnectException: Connection refused >>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >>> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl >>> .java:717) >>> at org.apache.storm.shade.org.apache.zookeeper.ClientCnxnSocket >>> NIO.doTransport(ClientCnxnSocketNIO.java:361) >>> at org.apache.storm.shade.org.apache.zookeeper.ClientCnxn$SendT >>> hread.run(ClientCnxn.java:1081) >>> 2016-12-12 23:46:54.440 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket >>> connection to server zookeeper-stormcluster.gobalto.com/10.110.20.8:2181. >>> Will not attempt to authenticate using SASL (unknown error) >>> 2016-12-12 23:46:54.454 o.a.s.s.o.a.z.ClientCnxn [WARN] Session 0x0 for >>> server null, unexpected error, closing socket connection and attempting >>> reconnect >>> java.net.ConnectException: Connection refused >>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >>> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl >>> .java:717) >>> at org.apache.storm.shade.org.apache.zookeeper.ClientCnxnSocket >>> NIO.doTransport(ClientCnxnSocketNIO.java:361) >>> at org.apache.storm.shade.org.apache.zookeeper.ClientCnxn$SendT >>> hread.run(ClientCnxn.java:1081) >>> 2016-12-12 23:46:55.393 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket >>> connection to server zookeeper-stormcluster.gobalto.com/10.110.20.8:2181. >>> Will not attempt to authenticate using SASL (unknown error) >>> 2016-12-12 23:46:55.394 o.a.s.s.o.a.z.ClientCnxn [WARN] Session 0x0 for >>> server null, unexpected error, closing socket connection and attempting >>> reconnect >>> java.net.ConnectException: Connection refused >>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >>> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl >>> .java:717) >>> at org.apache.storm.shade.org.apache.zookeeper.ClientCnxnSocket >>> NIO.doTransport(ClientCnxnSocketNIO.java:361) >>> at org.apache.storm.shade.org.apache.zookeeper.ClientCnxn$SendT >>> hread.run(ClientCnxn.java:1081) >>> 2016-12-12 23:46:55.741 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket >>> connection to server zookeeper-stormcluster.gobalto.com/10.110.20.8:2181. >>> Will not attempt to authenticate using SASL (unknown error) >>> 2016-12-12 23:46:55.763 o.a.s.s.o.a.z.ClientCnxn [WARN] Session 0x0 for >>> server null, unexpected error, closing socket connection and attempting >>> reconnect >>> java.net.ConnectException: Connection refused >>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >>> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl >>> .java:717) >>> at org.apache.storm.shade.org.apache.zookeeper.ClientCnxnSocket >>> NIO.doTransport(ClientCnxnSocketNIO.java:361) >>> at org.apache.storm.shade.org.apache.zookeeper.ClientCnxn$SendT >>> hread.run(ClientCnxn.java:1081) >>> 2016-12-12 23:46:55.959 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket >>> connection to server zookeeper-stormcluster.gobalto.com/10.110.20.8:2181. >>> Will not attempt to authenticate using SASL (unknown error) >>> 2016-12-12 23:46:55.969 o.a.s.s.o.a.z.ClientCnxn [WARN] Session 0x0 for >>> server null, unexpected error, closing socket connection and attempting >>> reconnect >>> java.net.ConnectException: Connection refused >>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >>> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl >>> .java:717) >>> at org.apache.storm.shade.org.apache.zookeeper.ClientCnxnSocket >>> NIO.doTransport(ClientCnxnSocketNIO.java:361) >>> at org.apache.storm.shade.org.apache.zookeeper.ClientCnxn$SendT >>> hread.run(ClientCnxn.java:1081) >>> >>> >>> >>> >>> On Mon, Dec 12, 2016 at 3:45 PM, Mostafa Gomaa <[email protected]> >>> wrote: >>> >>>> Can you show us your config files as well as the nimbus log? >>>> >>>> On Dec 13, 2016 1:44 AM, "Joaquin Menchaca" <[email protected]> >>>> wrote: >>>> >>>>> Also: >>>>> >>>>> $ storm list >>>>> Running: /usr/lib/jvm/java-8-oracle/bin/java -client -Ddaemon.name= >>>>> -Dstorm.options= -Dstorm.home=/usr/lib/apache/storm/1.0.2 >>>>> -Dstorm.log.dir=/usr/lib/apache/storm/1.0.2/logs >>>>> -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib >>>>> -Dstorm.conf.file= -cp /usr/lib/apache/storm/1.0.2/li >>>>> b/clojure-1.7.0.jar:/usr/lib/apache/storm/1.0.2/lib/minlog-1 >>>>> .3.0.jar:/usr/lib/apache/storm/1.0.2/lib/log4j-api-2.1.jar:/ >>>>> usr/lib/apache/storm/1.0.2/lib/storm-core-1.0.2.jar:/usr/lib >>>>> /apache/storm/1.0.2/lib/storm-rename-hack-1.0.2.jar:/usr/lib >>>>> /apache/storm/1.0.2/lib/slf4j-api-1.7.7.jar:/usr/lib/apache/ >>>>> storm/1.0.2/lib/log4j-slf4j-impl-2.1.jar:/usr/lib/apache/ >>>>> storm/1.0.2/lib/kryo-3.0.3.jar:/usr/lib/apache/storm/1.0. >>>>> 2/lib/objenesis-2.1.jar:/usr/lib/apache/storm/1.0.2/lib/ >>>>> log4j-core-2.1.jar:/usr/lib/apache/storm/1.0.2/lib/servlet >>>>> -api-2.5.jar:/usr/lib/apache/storm/1.0.2/lib/log4j-over-slf >>>>> 4j-1.6.6.jar:/usr/lib/apache/storm/1.0.2/lib/disruptor-3.3. >>>>> 2.jar:/usr/lib/apache/storm/1.0.2/lib/reflectasm-1.10.1.jar: >>>>> /usr/lib/apache/storm/1.0.2/lib/asm-5.0.3.jar:/usr/lib/apa >>>>> che/storm/1.0.2/conf:/usr/lib/apache/storm/1.0.2/bin >>>>> org.apache.storm.command.list >>>>> 12035 [main] WARN o.a.s.u.NimbusClient - Ignoring exception while >>>>> trying to get leader nimbus info from nimbus-stormcluster.gobalto.com. >>>>> will retry with a different seed host. >>>>> java.lang.RuntimeException: >>>>> org.apache.storm.thrift.transport.TTransportException: >>>>> java.net.ConnectException: Connection refused >>>>> at >>>>> org.apache.storm.security.auth.TBackoffConnect.retryNext(TBackoffConnect.java:64) >>>>> ~[storm-core-1.0.2.jar:1.0.2] >>>>> at >>>>> org.apache.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:56) >>>>> ~[storm-core-1.0.2.jar:1.0.2] >>>>> at >>>>> org.apache.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:99) >>>>> ~[storm-core-1.0.2.jar:1.0.2] >>>>> at >>>>> org.apache.storm.security.auth.ThriftClient.<init>(ThriftClient.java:69) >>>>> ~[storm-core-1.0.2.jar:1.0.2] >>>>> at org.apache.storm.utils.NimbusClient.<init>(NimbusClient.java:106) >>>>> ~[storm-core-1.0.2.jar:1.0.2] >>>>> at >>>>> org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:66) >>>>> [storm-core-1.0.2.jar:1.0.2] >>>>> at org.apache.storm.command.list$_main.invoke(list.clj:22) >>>>> [storm-core-1.0.2.jar:1.0.2] >>>>> at clojure.lang.AFn.applyToHelper(AFn.java:152) >>>>> [clojure-1.7.0.jar:?] >>>>> at clojure.lang.AFn.applyTo(AFn.java:144) [clojure-1.7.0.jar:?] >>>>> at org.apache.storm.command.list.main(Unknown Source) >>>>> [storm-core-1.0.2.jar:1.0.2] >>>>> Caused by: org.apache.storm.thrift.transport.TTransportException: >>>>> java.net.ConnectException: Connection refused >>>>> at org.apache.storm.thrift.transport.TSocket.open(TSocket.java:226) >>>>> ~[storm-core-1.0.2.jar:1.0.2] >>>>> at >>>>> org.apache.storm.thrift.transport.TFramedTransport.open(TFramedTransport.java:81) >>>>> ~[storm-core-1.0.2.jar:1.0.2] >>>>> at org.apache.storm.security.auth.SimpleTransportPlugin.connect >>>>> (SimpleTransportPlugin.java:103) ~[storm-core-1.0.2.jar:1.0.2] >>>>> at >>>>> org.apache.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:53) >>>>> ~[storm-core-1.0.2.jar:1.0.2] >>>>> ... 8 more >>>>> Caused by: java.net.ConnectException: Connection refused >>>>> at java.net.PlainSocketImpl.socketConnect(Native Method) >>>>> ~[?:1.8.0_91] >>>>> at >>>>> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) >>>>> ~[?:1.8.0_91] >>>>> at >>>>> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) >>>>> ~[?:1.8.0_91] >>>>> at >>>>> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) >>>>> ~[?:1.8.0_91] >>>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) >>>>> ~[?:1.8.0_91] >>>>> at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_91] >>>>> at org.apache.storm.thrift.transport.TSocket.open(TSocket.java:221) >>>>> ~[storm-core-1.0.2.jar:1.0.2] >>>>> at >>>>> org.apache.storm.thrift.transport.TFramedTransport.open(TFramedTransport.java:81) >>>>> ~[storm-core-1.0.2.jar:1.0.2] >>>>> at org.apache.storm.security.auth.SimpleTransportPlugin.connect >>>>> (SimpleTransportPlugin.java:103) ~[storm-core-1.0.2.jar:1.0.2] >>>>> at >>>>> org.apache.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:53) >>>>> ~[storm-core-1.0.2.jar:1.0.2] >>>>> ... 8 more >>>>> 22095 [main] WARN o.a.s.u.NimbusClient - Ignoring exception while >>>>> trying to get leader nimbus info from nimbus2-stormcluster.gobalto.com. >>>>> will retry with a different seed host. >>>>> java.lang.RuntimeException: >>>>> org.apache.storm.thrift.transport.TTransportException: >>>>> java.net.ConnectException: Connection refused >>>>> at >>>>> org.apache.storm.security.auth.TBackoffConnect.retryNext(TBackoffConnect.java:64) >>>>> ~[storm-core-1.0.2.jar:1.0.2] >>>>> at >>>>> org.apache.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:56) >>>>> ~[storm-core-1.0.2.jar:1.0.2] >>>>> at >>>>> org.apache.storm.security.auth.ThriftClient.reconnect(ThriftClient.java:99) >>>>> ~[storm-core-1.0.2.jar:1.0.2] >>>>> at >>>>> org.apache.storm.security.auth.ThriftClient.<init>(ThriftClient.java:69) >>>>> ~[storm-core-1.0.2.jar:1.0.2] >>>>> at org.apache.storm.utils.NimbusClient.<init>(NimbusClient.java:106) >>>>> ~[storm-core-1.0.2.jar:1.0.2] >>>>> at >>>>> org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:66) >>>>> [storm-core-1.0.2.jar:1.0.2] >>>>> at org.apache.storm.command.list$_main.invoke(list.clj:22) >>>>> [storm-core-1.0.2.jar:1.0.2] >>>>> at clojure.lang.AFn.applyToHelper(AFn.java:152) >>>>> [clojure-1.7.0.jar:?] >>>>> at clojure.lang.AFn.applyTo(AFn.java:144) [clojure-1.7.0.jar:?] >>>>> at org.apache.storm.command.list.main(Unknown Source) >>>>> [storm-core-1.0.2.jar:1.0.2] >>>>> Caused by: org.apache.storm.thrift.transport.TTransportException: >>>>> java.net.ConnectException: Connection refused >>>>> at org.apache.storm.thrift.transport.TSocket.open(TSocket.java:226) >>>>> ~[storm-core-1.0.2.jar:1.0.2] >>>>> at >>>>> org.apache.storm.thrift.transport.TFramedTransport.open(TFramedTransport.java:81) >>>>> ~[storm-core-1.0.2.jar:1.0.2] >>>>> at org.apache.storm.security.auth.SimpleTransportPlugin.connect >>>>> (SimpleTransportPlugin.java:103) ~[storm-core-1.0.2.jar:1.0.2] >>>>> at >>>>> org.apache.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:53) >>>>> ~[storm-core-1.0.2.jar:1.0.2] >>>>> ... 8 more >>>>> Caused by: java.net.ConnectException: Connection refused >>>>> at java.net.PlainSocketImpl.socketConnect(Native Method) >>>>> ~[?:1.8.0_91] >>>>> at >>>>> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) >>>>> ~[?:1.8.0_91] >>>>> at >>>>> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) >>>>> ~[?:1.8.0_91] >>>>> at >>>>> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) >>>>> ~[?:1.8.0_91] >>>>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) >>>>> ~[?:1.8.0_91] >>>>> at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_91] >>>>> at org.apache.storm.thrift.transport.TSocket.open(TSocket.java:221) >>>>> ~[storm-core-1.0.2.jar:1.0.2] >>>>> at >>>>> org.apache.storm.thrift.transport.TFramedTransport.open(TFramedTransport.java:81) >>>>> ~[storm-core-1.0.2.jar:1.0.2] >>>>> at org.apache.storm.security.auth.SimpleTransportPlugin.connect >>>>> (SimpleTransportPlugin.java:103) ~[storm-core-1.0.2.jar:1.0.2] >>>>> at >>>>> org.apache.storm.security.auth.TBackoffConnect.doConnectWithRetry(TBackoffConnect.java:53) >>>>> ~[storm-core-1.0.2.jar:1.0.2] >>>>> ... 8 more >>>>> Exception in thread "main" >>>>> org.apache.storm.utils.NimbusLeaderNotFoundException: >>>>> Could not find leader nimbus from seed hosts [" >>>>> nimbus-stormcluster.gobalto.com" "nimbus2-stormcluster.gobalto.com"]. >>>>> Did you specify a valid list of nimbus hosts for config nimbus.seeds? >>>>> at org.apache.storm.utils.NimbusClient.getConfiguredClientAs(Ni >>>>> mbusClient.java:90) >>>>> at org.apache.storm.command.list$_main.invoke(list.clj:22) >>>>> at clojure.lang.AFn.applyToHelper(AFn.java:152) >>>>> at clojure.lang.AFn.applyTo(AFn.java:144) >>>>> at org.apache.storm.command.list.main(Unknown Source) >>>>> >>>>> >>>>> On Mon, Dec 12, 2016 at 3:39 PM, Joaquin Menchaca < >>>>> [email protected]> wrote: >>>>> >>>>>> >>>>>> *Ideas?* >>>>>> org.apache.storm.utils.NimbusLeaderNotFoundException: Could not find >>>>>> leader nimbus from seed hosts ["nimbus-stormcluster.gobalto.com" >>>>>> "nimbus2-stormcluster.gobalto.com"]. Did you specify a valid list of >>>>>> nimbus hosts for config nimbus.seeds? >>>>>> at >>>>>> org.apache.storm.utils.NimbusClient.getConfiguredClientAs(NimbusClient.java:90) >>>>>> at org.apache.storm.ui.core$cluster_summary.invoke(core.clj:351) >>>>>> at org.apache.storm.ui.core$fn__12593.invoke(core.clj:931) >>>>>> at >>>>>> org.apache.storm.shade.compojure.core$make_route$fn__4631.invoke(core.clj:100) >>>>>> at >>>>>> org.apache.storm.shade.compojure.core$if_route$fn__4619.invoke(core.clj:46) >>>>>> at >>>>>> org.apache.storm.shade.compojure.core$if_method$fn__4612.invoke(core.clj:31) >>>>>> at >>>>>> org.apache.storm.shade.compojure.core$routing$fn__4637.invoke(core.clj:113) >>>>>> at clojure.core$some.invoke(core.clj:2570) >>>>>> at org.apache.storm.shade.compojure.core$routing.doInvoke(core.clj:113) >>>>>> at clojure.lang.RestFn.applyTo(RestFn.java:139) >>>>>> at clojure.core$apply.invoke(core.clj:632) >>>>>> at >>>>>> org.apache.storm.shade.compojure.core$routes$fn__4641.invoke(core.clj:118) >>>>>> at >>>>>> org.apache.storm.shade.ring.middleware.json$wrap_json_params$fn__12065.invoke(json.clj:56) >>>>>> at >>>>>> org.apache.storm.shade.ring.middleware.multipart_params$wrap_multipart_params$fn__5766.invoke(multipart_params.clj:118) >>>>>> at >>>>>> org.apache.storm.shade.ring.middleware.reload$wrap_reload$fn__11217.invoke(reload.clj:22) >>>>>> at >>>>>> org.apache.storm.ui.helpers$requests_middleware$fn__6019.invoke(helpers.clj:50) >>>>>> at org.apache.storm.ui.core$catch_errors$fn__12786.invoke(core.clj:1225) >>>>>> at >>>>>> org.apache.storm.shade.ring.middleware.keyword_params$wrap_keyword_params$fn__5686.invoke(keyword_params.clj:35) >>>>>> at >>>>>> org.apache.storm.shade.ring.middleware.nested_params$wrap_nested_params$fn__5729.invoke(nested_params.clj:84) >>>>>> at >>>>>> org.apache.storm.shade.ring.middleware.params$wrap_params$fn__5658.invoke(params.clj:64) >>>>>> at >>>>>> org.apache.storm.shade.ring.middleware.multipart_params$wrap_multipart_params$fn__5766.invoke(multipart_params.clj:118) >>>>>> at >>>>>> org.apache.storm.shade.ring.middleware.flash$wrap_flash$fn__5981.invoke(flash.clj:35) >>>>>> at >>>>>> org.apache.storm.shade.ring.middleware.session$wrap_session$fn__5967.invoke(session.clj:98) >>>>>> at >>>>>> org.apache.storm.shade.ring.util.servlet$make_service_method$fn__5516.invoke(servlet.clj:127) >>>>>> at >>>>>> org.apache.storm.shade.ring.util.servlet$servlet$fn__5520.invoke(servlet.clj:136) >>>>>> at >>>>>> org.apache.storm.shade.ring.util.servlet.proxy$javax.servlet.http.HttpServlet$ff19274a.service(Unknown >>>>>> Source) >>>>>> at >>>>>> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:654) >>>>>> at >>>>>> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1320) >>>>>> at >>>>>> org.apache.storm.logging.filters.AccessLoggingFilter.handle(AccessLoggingFilter.java:47) >>>>>> at >>>>>> org.apache.storm.logging.filters.AccessLoggingFilter.doFilter(AccessLoggingFilter.java:39) >>>>>> at >>>>>> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291) >>>>>> at sun.reflect.GeneratedMethodAccessor43.invoke(Unknown Source) >>>>>> at >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>>> at java.lang.reflect.Method.invoke(Method.java:498) >>>>>> at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) >>>>>> at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) >>>>>> at >>>>>> org.apache.storm.ui.helpers$x_frame_options_filter_handler$fn__6112.invoke(helpers.clj:189) >>>>>> at >>>>>> org.apache.storm.ui.helpers.proxy$java.lang.Object$Filter$abec9a8f.doFilter(Unknown >>>>>> Source) >>>>>> at >>>>>> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291) >>>>>> at >>>>>> org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:247) >>>>>> at >>>>>> org.apache.storm.shade.org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:210) >>>>>> at >>>>>> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1291) >>>>>> at >>>>>> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:443) >>>>>> at >>>>>> org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1044) >>>>>> at >>>>>> org.apache.storm.shade.org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:372) >>>>>> at >>>>>> org.apache.storm.shade.org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:978) >>>>>> at >>>>>> org.apache.storm.shade.org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) >>>>>> at >>>>>> org.apache.storm.shade.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) >>>>>> at >>>>>> org.apache.storm.shade.org.eclipse.jetty.server.Server.handle(Server.java:369) >>>>>> at >>>>>> org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:486) >>>>>> at >>>>>> org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:933) >>>>>> at >>>>>> org.apache.storm.shade.org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:995) >>>>>> at >>>>>> org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) >>>>>> at >>>>>> org.apache.storm.shade.org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) >>>>>> at >>>>>> org.apache.storm.shade.org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) >>>>>> at >>>>>> org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668) >>>>>> at >>>>>> org.apache.storm.shade.org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) >>>>>> at >>>>>> org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) >>>>>> at >>>>>> org.apache.storm.shade.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) >>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> 是故勝兵先勝而後求戰，敗兵先戰而後求勝。 >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> 是故勝兵先勝而後求戰，敗兵先戰而後求勝。 >>>>> >>>> >>> >>> >>> -- >>> >>> 是故勝兵先勝而後求戰，敗兵先戰而後求勝。 >>> >> > > > -- > > 是故勝兵先勝而後求戰，敗兵先戰而後求勝。 >
