That is a little hard to tell. It looks like your ZK server crashed and was
not relaunched.
- Bobby
On Friday, April 7, 2017, 1:35:43 AM CDT, xx <whzq2...@126.com> wrote:hi,
everyone! Lylian for your help!I'm running a storm cluster.I have a
nimbus,zookeeper,kafka server, and a supervisor in one node.Furthermore, I
deploy all of them in this node.so the zookeeper status is standalone.
version information list:Storm 0.10.0Zookeeper-3.4.6Kafka_2.9.2-0.8.2.0JDK 1.7
when I deploy three topology in this node, it seems like work well .but two
days later, it suddenly stop with a crash. nimbus, supervisor, worker ,storm
ui, are killed.
2017-04-05 11:13:11.627 b.s.d.nimbus [INFO] [req 63] Access from: principal:
op:fileDownload2017-04-05 11:13:11.629 b.s.d.nimbus [INFO] [req 64] Access
from: principal: op:fileDownload2017-04-05 11:13:11.631 b.s.d.nimbus [INFO]
[req 1] Access from: principal: op:fileDownload2017-04-05 11:13:11.632
b.s.d.nimbus [INFO] [req 2] Access from: principal: op:fileDownload2017-04-05
11:13:11.633 b.s.d.nimbus [INFO] [req 3] Access from: principal:
op:fileDownload2017-04-05 11:13:11.633 b.s.d.nimbus [INFO] [req 4] Access from:
principal: op:fileDownload2017-04-05 11:13:11.635 b.s.d.nimbus [INFO] [req 5]
Access from: principal: op:fileDownload2017-04-05 11:13:40.660 b.s.d.nimbus
[INFO] [req 6] Access from: principal: op:getClusterInfo2017-04-05
11:14:03.901 b.s.d.nimbus [INFO] [req 7] Access from: principal:
op:getClusterInfo2017-04-05 11:14:35.821 b.s.d.nimbus [INFO] [req 8] Access
from: principal: op:getClusterInfo2017-04-05 19:30:32.541
o.a.s.s.o.a.z.ClientCnxn [INFO] Client session timed out, have not heard from
server in 13337ms for sessionid 0x15b23c37c6400c6, closing socket connection
and attempting reconnect2017-04-05 19:30:32.644
o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State change:
SUSPENDED2017-04-05 19:30:32.651 b.s.cluster [WARN] Received event
:disconnected::none: with disconnected Zookeeper.2017-04-05 19:30:33.138
o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to server
xiang/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown
error)2017-04-05 19:30:33.139 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection
established to xiang/127.0.0.1:2181, initiating session2017-04-05 19:30:33.141
o.a.s.s.o.a.z.ClientCnxn [INFO] Session establishment complete on server
xiang/127.0.0.1:2181, sessionid = 0x15b23c37c6400c6, negotiated timeout =
200002017-04-05 19:30:33.141 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO]
State change: RECONNECTED2017-04-05 19:30:59.303 o.a.s.s.o.a.z.ClientCnxn
[INFO] Client session timed out, have not heard from server in 13337ms for
sessionid 0x15b23c37c6400c6, closing socket connection and attempting
reconnect2017-04-05 19:30:59.404 o.a.s.s.o.a.c.f.s.ConnectionStateManager
[INFO] State change: SUSPENDED2017-04-05 19:30:59.405 b.s.cluster [WARN]
Received event :disconnected::none: with disconnected Zookeeper.2017-04-05
19:31:00.145 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to
server xiang/10.240.47.205:2181. Will not attempt to authenticate using SASL
(unknown error)2017-04-05 19:31:00.146 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket
connection established to xiang/10.240.47.205:2181, initiating
session2017-04-05 19:31:00.147 o.a.s.s.o.a.z.ClientCnxn [INFO] Session
establishment complete on server xiang/10.240.47.205:2181, sessionid =
0x15b23c37c6400c6, negotiated timeout = 200002017-04-05 19:31:00.148
o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State change:
RECONNECTED2017-04-05 19:31:13.487 o.a.s.s.o.a.z.ClientCnxn [INFO] Client
session timed out, have not heard from server in 13340ms for sessionid
0x15b23c37c6400c6, closing socket connection and attempting reconnect2017-04-05
19:31:13.588 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State change:
SUSPENDED2017-04-05 19:31:13.588 b.s.cluster [WARN] Received event
:disconnected::none: with disconnected Zookeeper.2017-04-05 19:31:14.436
o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to server
xiang/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown
error)2017-04-05 19:31:14.437 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection
established to xiang/127.0.0.1:2181, initiating session2017-04-05 19:31:14.438
o.a.s.s.o.a.z.ClientCnxn [INFO] Session establishment complete on server
xiang/127.0.0.1:2181, sessionid = 0x15b23c37c6400c6, negotiated timeout =
200002017-04-05 19:31:14.439 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO]
State change: RECONNECTED2017-04-05 19:31:27.772 o.a.s.s.o.a.z.ClientCnxn
[INFO] Client session timed out, have not heard from server in 13334ms for
sessionid 0x15b23c37c6400c6, closing socket connection and attempting
reconnect2017-04-05 19:31:27.873 o.a.s.s.o.a.c.f.s.ConnectionStateManager
[INFO] State change: SUSPENDED2017-04-05 19:31:27.874 b.s.cluster [WARN]
Received event :disconnected::none: with disconnected Zookeeper.2017-04-05
19:31:28.572 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to
server xiang/10.240.47.205:2181. Will not attempt to authenticate using SASL
(unknown error)2017-04-05 19:31:28.573 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket
connection established to xiang/10.240.47.205:2181, initiating
session2017-04-05 19:31:28.575 o.a.s.s.o.a.z.ClientCnxn [INFO] Session
establishment complete on server xiang/10.240.47.205:2181, sessionid =
0x15b23c37c6400c6, negotiated timeout = 200002017-04-05 19:31:28.575
o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State change:
RECONNECTED2017-04-05 19:31:41.914 o.a.s.s.o.a.z.ClientCnxn [INFO] Client
session timed out, have not heard from server in 13339ms for sessionid
0x15b23c37c6400c6, closing socket connection and attempting reconnect2017-04-05
19:31:42.015 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State change:
SUSPENDED2017-04-05 19:31:42.015 b.s.cluster [WARN] Received event
:disconnected::none: with disconnected Zookeeper.2017-04-05 19:31:42.361
o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to server
xiang/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown
error)2017-04-05 19:31:42.362 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection
established to xiang/127.0.0.1:2181, initiating session2017-04-05 19:31:42.384
o.a.s.s.o.a.z.ClientCnxn [INFO] Session establishment complete on server
xiang/127.0.0.1:2181, sessionid = 0x15b23c37c6400c6, negotiated timeout =
200002017-04-05 19:31:42.384 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO]
State change: RECONNECTED2017-04-05 19:31:55.718 o.a.s.s.o.a.z.ClientCnxn
[INFO] Client session timed out, have not heard from server in 13334ms for
sessionid 0x15b23c37c6400c6, closing socket connection and attempting
reconnect2017-04-05 19:31:55.819 o.a.s.s.o.a.c.f.s.ConnectionStateManager
[INFO] State change: SUSPENDED2017-04-05 19:31:55.819 b.s.cluster [WARN]
Received event :disconnected::none: with disconnected Zookeeper.2017-04-05
19:31:56.767 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket connection to
server xiang/10.240.47.205:2181. Will not attempt to authenticate using SASL
(unknown error)2017-04-05 19:31:56.767 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket
connection established to xiang/10.240.47.205:2181, initiating
session2017-04-05 19:31:56.769 o.a.s.s.o.a.z.ClientCnxn [INFO] Session
establishment complete on server xiang/10.240.47.205:2181, sessionid =
0x15b23c37c6400c6, negotiated timeout = 200002017-04-05 19:31:56.769
o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State change:
RECONNECTED2017-04-05 19:32:10.109 o.a.s.s.o.a.z.ClientCnxn [INFO] Client
session timed out, have not heard from server in 13340ms for sessionid
0x15b23c37c6400c6, closing socket connection and attempting reconnect2017-04-05
19:32:10.210 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State change:
SUSPENDED2017-04-05 19:32:10.211 b.s.cluster [WARN] Received event
:disconnected::none: with disconnected Zookeeper.2017-04-05 19:32:10.212
b.s.d.nimbus [ERROR] Error when processing eventjava.lang.RuntimeException:
org.apache.storm.shade.org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /storms/apm_source_topology-11-1491038533
at backtype.storm.util$wrap_in_runtime.invoke(util.clj:49)
~[storm-core-0.10.0.jar:0.10.0] at
backtype.storm.zookeeper$exists_node_QMARK_$fn__4306.invoke(zookeeper.clj:103)
~[storm-core-0.10.0.jar:0.10.0] at
backtype.storm.zookeeper$exists_node_QMARK_.invoke(zookeeper.clj:99)
~[storm-core-0.10.0.jar:0.10.0] at
backtype.storm.zookeeper$get_data.invoke(zookeeper.clj:128)
~[storm-core-0.10.0.jar:0.10.0] at
backtype.storm.cluster$mk_distributed_cluster_state$reify__4580.get_data(cluster.clj:114)
~[storm-core-0.10.0.jar:0.10.0] at
backtype.storm.cluster$mk_storm_cluster_state$reify__5120.storm_base(cluster.clj:426)
~[storm-core-0.10.0.jar:0.10.0] at
sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source) ~[?:?] at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.7.0_21] at java.lang.reflect.Method.invoke(Method.java:601)
~[?:1.7.0_21] at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
~[clojure-1.6.0.jar:?] at
clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
~[clojure-1.6.0.jar:?] at
backtype.storm.daemon.nimbus$read_topology_details.invoke(nimbus.clj:344)
~[storm-core-0.10.0.jar:0.10.0] at
backtype.storm.daemon.nimbus$mk_assignments$iter__6344__6348$fn__6349.invoke(nimbus.clj:674)
~[storm-core-0.10.0.jar:0.10.0] at clojure.lang.LazySeq.sval(LazySeq.java:40)
~[clojure-1.6.0.jar:?] at clojure.lang.LazySeq.seq(LazySeq.java:49)
~[clojure-1.6.0.jar:?] at clojure.lang.RT.seq(RT.java:484)
~[clojure-1.6.0.jar:?] at clojure.core$seq.invoke(core.clj:133)
~[clojure-1.6.0.jar:?] at
clojure.core.protocols$seq_reduce.invoke(protocols.clj:30)
~[clojure-1.6.0.jar:?] at
clojure.core.protocols$fn__6078.invoke(protocols.clj:54) ~[clojure-1.6.0.jar:?]
at clojure.core.protocols$fn__6031$G__6026__6044.invoke(protocols.clj:13)
~[clojure-1.6.0.jar:?] at clojure.core$reduce.invoke(core.clj:6289)
~[clojure-1.6.0.jar:?] at clojure.core$into.invoke(core.clj:6341)
~[clojure-1.6.0.jar:?] at
backtype.storm.daemon.nimbus$mk_assignments.doInvoke(nimbus.clj:673)
~[storm-core-0.10.0.jar:0.10.0] at clojure.lang.RestFn.invoke(RestFn.java:410)
~[clojure-1.6.0.jar:?] at
backtype.storm.daemon.nimbus$fn__6583$exec_fn__1236__auto____6584$fn__6589$fn__6590.invoke(nimbus.clj:1025)
~[storm-core-0.10.0.jar:0.10.0] at
backtype.storm.daemon.nimbus$fn__6583$exec_fn__1236__auto____6584$fn__6589.invoke(nimbus.clj:1024)
~[storm-core-0.10.0.jar:0.10.0] at
backtype.storm.timer$schedule_recurring$this__3721.invoke(timer.clj:102)
~[storm-core-0.10.0.jar:0.10.0] at
backtype.storm.timer$mk_timer$fn__3704$fn__3705.invoke(timer.clj:50)
[storm-core-0.10.0.jar:0.10.0] at
backtype.storm.timer$mk_timer$fn__3704.invoke(timer.clj:42)
[storm-core-0.10.0.jar:0.10.0] at clojure.lang.AFn.run(AFn.java:22)
[clojure-1.6.0.jar:?] at java.lang.Thread.run(Thread.java:722)
[?:1.7.0_21]Caused by:
org.apache.storm.shade.org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /storms/apm_source_topology-11-1491038533
at
org.apache.storm.shade.org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
~[storm-core-0.10.0.jar:0.10.0] at
org.apache.storm.shade.org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
~[storm-core-0.10.0.jar:0.10.0] at
org.apache.storm.shade.org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
~[storm-core-0.10.0.jar:0.10.0] at
org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl$2.call(ExistsBuilderImpl.java:172)
~[storm-core-0.10.0.jar:0.10.0] at
org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl$2.call(ExistsBuilderImpl.java:161)
~[storm-core-0.10.0.jar:0.10.0] at
org.apache.storm.shade.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
~[storm-core-0.10.0.jar:0.10.0] at
org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl.pathInForeground(ExistsBuilderImpl.java:157)
~[storm-core-0.10.0.jar:0.10.0] at
org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:148)
~[storm-core-0.10.0.jar:0.10.0] at
org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:36)
~[storm-core-0.10.0.jar:0.10.0] at
backtype.storm.zookeeper$exists_node_QMARK_$fn__4306.invoke(zookeeper.clj:102)
~[storm-core-0.10.0.jar:0.10.0] ... 29 more2017-04-05 19:32:10.238 b.s.util
[ERROR] Halting process: ("Error when processing an
event")java.lang.RuntimeException: ("Error when processing an event") at
backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:336)
[storm-core-0.10.0.jar:0.10.0] at clojure.lang.RestFn.invoke(RestFn.java:423)
[clojure-1.6.0.jar:?] at
backtype.storm.daemon.nimbus$nimbus_data$fn__5953.invoke(nimbus.clj:97)
[storm-core-0.10.0.jar:0.10.0] at
backtype.storm.timer$mk_timer$fn__3704$fn__3705.invoke(timer.clj:71)
[storm-core-0.10.0.jar:0.10.0] at
backtype.storm.timer$mk_timer$fn__3704.invoke(timer.clj:42)
[storm-core-0.10.0.jar:0.10.0] at clojure.lang.AFn.run(AFn.java:22)
[clojure-1.6.0.jar:?] at java.lang.Thread.run(Thread.java:722)
[?:1.7.0_21]2017-04-05 19:32:10.241 b.s.d.nimbus [INFO] Shutting down master
I don't really know a lot about storm and this is for my first project so
please help!
