hi, everyone! Lylian for your help!
I'm running a storm cluster.I have a nimbus,zookeeper,kafka server, and a
supervisor in one node.
Furthermore, I deploy all of them in this node.so the zookeeper status is
standalone.
version information list:
Storm 0.10.0
Zookeeper-3.4.6
Kafka_2.9.2-0.8.2.0
JDK 1.7
when I deploy three topology in this node, it seems like work well .
but two days later, it suddenly stop with a crash.
nimbus, supervisor, worker ,storm ui, are killed.
2017-04-05 11:13:11.627 b.s.d.nimbus [INFO] [req 63] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.629 b.s.d.nimbus [INFO] [req 64] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.631 b.s.d.nimbus [INFO] [req 1] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.632 b.s.d.nimbus [INFO] [req 2] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.633 b.s.d.nimbus [INFO] [req 3] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.633 b.s.d.nimbus [INFO] [req 4] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.635 b.s.d.nimbus [INFO] [req 5] Access from: principal:
op:fileDownload
2017-04-05 11:13:40.660 b.s.d.nimbus [INFO] [req 6] Access from: principal:
op:getClusterInfo
2017-04-05 11:14:03.901 b.s.d.nimbus [INFO] [req 7] Access from: principal:
op:getClusterInfo
2017-04-05 11:14:35.821 b.s.d.nimbus [INFO] [req 8] Access from: principal:
op:getClusterInfo
2017-04-05 19:30:32.541 o.a.s.s.o.a.z.ClientCnxn [INFO] Client session timed
out, have not heard from server in 13337ms for sessionid 0x15b23c37c6400c6,
closing socket connection and attempting reconnect
2017-04-05 19:30:32.644 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State
change: SUSPENDED
2017-04-05 19:30:32.651 b.s.cluster [WARN] Received event :disconnected::none:
with disconnected Zookeeper.
2017-04-05 19:30:33.138 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket
connection to server xiang/127.0.0.1:2181. Will not attempt to authenticate
using SASL (unknown error)
2017-04-05 19:30:33.139 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection
established to xiang/127.0.0.1:2181, initiating session
2017-04-05 19:30:33.141 o.a.s.s.o.a.z.ClientCnxn [INFO] Session establishment
complete on server xiang/127.0.0.1:2181, sessionid = 0x15b23c37c6400c6,
negotiated timeout = 20000
2017-04-05 19:30:33.141 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State
change: RECONNECTED
2017-04-05 19:30:59.303 o.a.s.s.o.a.z.ClientCnxn [INFO] Client session timed
out, have not heard from server in 13337ms for sessionid 0x15b23c37c6400c6,
closing socket connection and attempting reconnect
2017-04-05 19:30:59.404 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State
change: SUSPENDED
2017-04-05 19:30:59.405 b.s.cluster [WARN] Received event :disconnected::none:
with disconnected Zookeeper.
2017-04-05 19:31:00.145 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket
connection to server xiang/10.240.47.205:2181. Will not attempt to authenticate
using SASL (unknown error)
2017-04-05 19:31:00.146 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection
established to xiang/10.240.47.205:2181, initiating session
2017-04-05 19:31:00.147 o.a.s.s.o.a.z.ClientCnxn [INFO] Session establishment
complete on server xiang/10.240.47.205:2181, sessionid = 0x15b23c37c6400c6,
negotiated timeout = 20000
2017-04-05 19:31:00.148 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State
change: RECONNECTED
2017-04-05 19:31:13.487 o.a.s.s.o.a.z.ClientCnxn [INFO] Client session timed
out, have not heard from server in 13340ms for sessionid 0x15b23c37c6400c6,
closing socket connection and attempting reconnect
2017-04-05 19:31:13.588 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State
change: SUSPENDED
2017-04-05 19:31:13.588 b.s.cluster [WARN] Received event :disconnected::none:
with disconnected Zookeeper.
2017-04-05 19:31:14.436 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket
connection to server xiang/127.0.0.1:2181. Will not attempt to authenticate
using SASL (unknown error)
2017-04-05 19:31:14.437 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection
established to xiang/127.0.0.1:2181, initiating session
2017-04-05 19:31:14.438 o.a.s.s.o.a.z.ClientCnxn [INFO] Session establishment
complete on server xiang/127.0.0.1:2181, sessionid = 0x15b23c37c6400c6,
negotiated timeout = 20000
2017-04-05 19:31:14.439 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State
change: RECONNECTED
2017-04-05 19:31:27.772 o.a.s.s.o.a.z.ClientCnxn [INFO] Client session timed
out, have not heard from server in 13334ms for sessionid 0x15b23c37c6400c6,
closing socket connection and attempting reconnect
2017-04-05 19:31:27.873 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State
change: SUSPENDED
2017-04-05 19:31:27.874 b.s.cluster [WARN] Received event :disconnected::none:
with disconnected Zookeeper.
2017-04-05 19:31:28.572 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket
connection to server xiang/10.240.47.205:2181. Will not attempt to authenticate
using SASL (unknown error)
2017-04-05 19:31:28.573 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection
established to xiang/10.240.47.205:2181, initiating session
2017-04-05 19:31:28.575 o.a.s.s.o.a.z.ClientCnxn [INFO] Session establishment
complete on server xiang/10.240.47.205:2181, sessionid = 0x15b23c37c6400c6,
negotiated timeout = 20000
2017-04-05 19:31:28.575 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State
change: RECONNECTED
2017-04-05 19:31:41.914 o.a.s.s.o.a.z.ClientCnxn [INFO] Client session timed
out, have not heard from server in 13339ms for sessionid 0x15b23c37c6400c6,
closing socket connection and attempting reconnect
2017-04-05 19:31:42.015 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State
change: SUSPENDED
2017-04-05 19:31:42.015 b.s.cluster [WARN] Received event :disconnected::none:
with disconnected Zookeeper.
2017-04-05 19:31:42.361 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket
connection to server xiang/127.0.0.1:2181. Will not attempt to authenticate
using SASL (unknown error)
2017-04-05 19:31:42.362 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection
established to xiang/127.0.0.1:2181, initiating session
2017-04-05 19:31:42.384 o.a.s.s.o.a.z.ClientCnxn [INFO] Session establishment
complete on server xiang/127.0.0.1:2181, sessionid = 0x15b23c37c6400c6,
negotiated timeout = 20000
2017-04-05 19:31:42.384 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State
change: RECONNECTED
2017-04-05 19:31:55.718 o.a.s.s.o.a.z.ClientCnxn [INFO] Client session timed
out, have not heard from server in 13334ms for sessionid 0x15b23c37c6400c6,
closing socket connection and attempting reconnect
2017-04-05 19:31:55.819 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State
change: SUSPENDED
2017-04-05 19:31:55.819 b.s.cluster [WARN] Received event :disconnected::none:
with disconnected Zookeeper.
2017-04-05 19:31:56.767 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket
connection to server xiang/10.240.47.205:2181. Will not attempt to authenticate
using SASL (unknown error)
2017-04-05 19:31:56.767 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection
established to xiang/10.240.47.205:2181, initiating session
2017-04-05 19:31:56.769 o.a.s.s.o.a.z.ClientCnxn [INFO] Session establishment
complete on server xiang/10.240.47.205:2181, sessionid = 0x15b23c37c6400c6,
negotiated timeout = 20000
2017-04-05 19:31:56.769 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State
change: RECONNECTED
2017-04-05 19:32:10.109 o.a.s.s.o.a.z.ClientCnxn [INFO] Client session timed
out, have not heard from server in 13340ms for sessionid 0x15b23c37c6400c6,
closing socket connection and attempting reconnect
2017-04-05 19:32:10.210 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State
change: SUSPENDED
2017-04-05 19:32:10.211 b.s.cluster [WARN] Received event :disconnected::none:
with disconnected Zookeeper.
2017-04-05 19:32:10.212 b.s.d.nimbus [ERROR] Error when processing event
java.lang.RuntimeException:
org.apache.storm.shade.org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /storms/apm_source_topology-11-1491038533
at backtype.storm.util$wrap_in_runtime.invoke(util.clj:49)
~[storm-core-0.10.0.jar:0.10.0]
at
backtype.storm.zookeeper$exists_node_QMARK_$fn__4306.invoke(zookeeper.clj:103)
~[storm-core-0.10.0.jar:0.10.0]
at backtype.storm.zookeeper$exists_node_QMARK_.invoke(zookeeper.clj:99)
~[storm-core-0.10.0.jar:0.10.0]
at backtype.storm.zookeeper$get_data.invoke(zookeeper.clj:128)
~[storm-core-0.10.0.jar:0.10.0]
at
backtype.storm.cluster$mk_distributed_cluster_state$reify__4580.get_data(cluster.clj:114)
~[storm-core-0.10.0.jar:0.10.0]
at
backtype.storm.cluster$mk_storm_cluster_state$reify__5120.storm_base(cluster.clj:426)
~[storm-core-0.10.0.jar:0.10.0]
at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source) ~[?:?]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.7.0_21]
at java.lang.reflect.Method.invoke(Method.java:601) ~[?:1.7.0_21]
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
~[clojure-1.6.0.jar:?]
at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
~[clojure-1.6.0.jar:?]
at backtype.storm.daemon.nimbus$read_topology_details.invoke(nimbus.clj:344)
~[storm-core-0.10.0.jar:0.10.0]
at
backtype.storm.daemon.nimbus$mk_assignments$iter__6344__6348$fn__6349.invoke(nimbus.clj:674)
~[storm-core-0.10.0.jar:0.10.0]
at clojure.lang.LazySeq.sval(LazySeq.java:40) ~[clojure-1.6.0.jar:?]
at clojure.lang.LazySeq.seq(LazySeq.java:49) ~[clojure-1.6.0.jar:?]
at clojure.lang.RT.seq(RT.java:484) ~[clojure-1.6.0.jar:?]
at clojure.core$seq.invoke(core.clj:133) ~[clojure-1.6.0.jar:?]
at clojure.core.protocols$seq_reduce.invoke(protocols.clj:30)
~[clojure-1.6.0.jar:?]
at clojure.core.protocols$fn__6078.invoke(protocols.clj:54)
~[clojure-1.6.0.jar:?]
at clojure.core.protocols$fn__6031$G__6026__6044.invoke(protocols.clj:13)
~[clojure-1.6.0.jar:?]
at clojure.core$reduce.invoke(core.clj:6289) ~[clojure-1.6.0.jar:?]
at clojure.core$into.invoke(core.clj:6341) ~[clojure-1.6.0.jar:?]
at backtype.storm.daemon.nimbus$mk_assignments.doInvoke(nimbus.clj:673)
~[storm-core-0.10.0.jar:0.10.0]
at clojure.lang.RestFn.invoke(RestFn.java:410) ~[clojure-1.6.0.jar:?]
at
backtype.storm.daemon.nimbus$fn__6583$exec_fn__1236__auto____6584$fn__6589$fn__6590.invoke(nimbus.clj:1025)
~[storm-core-0.10.0.jar:0.10.0]
at
backtype.storm.daemon.nimbus$fn__6583$exec_fn__1236__auto____6584$fn__6589.invoke(nimbus.clj:1024)
~[storm-core-0.10.0.jar:0.10.0]
at backtype.storm.timer$schedule_recurring$this__3721.invoke(timer.clj:102)
~[storm-core-0.10.0.jar:0.10.0]
at backtype.storm.timer$mk_timer$fn__3704$fn__3705.invoke(timer.clj:50)
[storm-core-0.10.0.jar:0.10.0]
at backtype.storm.timer$mk_timer$fn__3704.invoke(timer.clj:42)
[storm-core-0.10.0.jar:0.10.0]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
at java.lang.Thread.run(Thread.java:722) [?:1.7.0_21]
Caused by:
org.apache.storm.shade.org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /storms/apm_source_topology-11-1491038533
at
org.apache.storm.shade.org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
~[storm-core-0.10.0.jar:0.10.0]
at
org.apache.storm.shade.org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
~[storm-core-0.10.0.jar:0.10.0]
at
org.apache.storm.shade.org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
~[storm-core-0.10.0.jar:0.10.0]
at
org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl$2.call(ExistsBuilderImpl.java:172)
~[storm-core-0.10.0.jar:0.10.0]
at
org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl$2.call(ExistsBuilderImpl.java:161)
~[storm-core-0.10.0.jar:0.10.0]
at
org.apache.storm.shade.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
~[storm-core-0.10.0.jar:0.10.0]
at
org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl.pathInForeground(ExistsBuilderImpl.java:157)
~[storm-core-0.10.0.jar:0.10.0]
at
org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:148)
~[storm-core-0.10.0.jar:0.10.0]
at
org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:36)
~[storm-core-0.10.0.jar:0.10.0]
at
backtype.storm.zookeeper$exists_node_QMARK_$fn__4306.invoke(zookeeper.clj:102)
~[storm-core-0.10.0.jar:0.10.0]
... 29 more
2017-04-05 19:32:10.238 b.s.util [ERROR] Halting process: ("Error when
processing an event")
java.lang.RuntimeException: ("Error when processing an event")
at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:336)
[storm-core-0.10.0.jar:0.10.0]
at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.6.0.jar:?]
at backtype.storm.daemon.nimbus$nimbus_data$fn__5953.invoke(nimbus.clj:97)
[storm-core-0.10.0.jar:0.10.0]
at backtype.storm.timer$mk_timer$fn__3704$fn__3705.invoke(timer.clj:71)
[storm-core-0.10.0.jar:0.10.0]
at backtype.storm.timer$mk_timer$fn__3704.invoke(timer.clj:42)
[storm-core-0.10.0.jar:0.10.0]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
at java.lang.Thread.run(Thread.java:722) [?:1.7.0_21]
2017-04-05 19:32:10.241 b.s.d.nimbus [INFO] Shutting down master
I don't really know a lot about storm and this is for my first project so
please help!
2017-04-05 11:13:11.564 b.s.d.nimbus [INFO] [req 11] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.565 b.s.d.nimbus [INFO] [req 12] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.565 b.s.d.nimbus [INFO] [req 13] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.566 b.s.d.nimbus [INFO] [req 14] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.566 b.s.d.nimbus [INFO] [req 15] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.567 b.s.d.nimbus [INFO] [req 16] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.567 b.s.d.nimbus [INFO] [req 17] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.567 b.s.d.nimbus [INFO] [req 18] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.568 b.s.d.nimbus [INFO] [req 19] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.568 b.s.d.nimbus [INFO] [req 20] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.569 b.s.d.nimbus [INFO] [req 21] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.569 b.s.d.nimbus [INFO] [req 22] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.569 b.s.d.nimbus [INFO] [req 23] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.570 b.s.d.nimbus [INFO] [req 24] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.570 b.s.d.nimbus [INFO] [req 25] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.571 b.s.d.nimbus [INFO] [req 26] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.571 b.s.d.nimbus [INFO] [req 27] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.571 b.s.d.nimbus [INFO] [req 28] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.572 b.s.d.nimbus [INFO] [req 29] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.572 b.s.d.nimbus [INFO] [req 30] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.573 b.s.d.nimbus [INFO] [req 31] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.573 b.s.d.nimbus [INFO] [req 32] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.573 b.s.d.nimbus [INFO] [req 33] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.574 b.s.d.nimbus [INFO] [req 34] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.574 b.s.d.nimbus [INFO] [req 35] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.575 b.s.d.nimbus [INFO] [req 36] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.575 b.s.d.nimbus [INFO] [req 37] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.576 b.s.d.nimbus [INFO] [req 38] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.576 b.s.d.nimbus [INFO] [req 39] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.577 b.s.d.nimbus [INFO] [req 40] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.577 b.s.d.nimbus [INFO] [req 41] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.577 b.s.d.nimbus [INFO] [req 42] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.578 b.s.d.nimbus [INFO] [req 43] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.578 b.s.d.nimbus [INFO] [req 44] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.579 b.s.d.nimbus [INFO] [req 45] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.579 b.s.d.nimbus [INFO] [req 46] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.581 b.s.d.nimbus [INFO] [req 47] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.584 b.s.d.nimbus [INFO] [req 48] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.585 b.s.d.nimbus [INFO] [req 49] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.585 b.s.d.nimbus [INFO] [req 50] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.585 b.s.d.nimbus [INFO] [req 51] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.586 b.s.d.nimbus [INFO] [req 52] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.586 b.s.d.nimbus [INFO] [req 53] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.587 b.s.d.nimbus [INFO] [req 54] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.587 b.s.d.nimbus [INFO] [req 55] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.588 b.s.d.nimbus [INFO] [req 56] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.588 b.s.d.nimbus [INFO] [req 57] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.588 b.s.d.nimbus [INFO] [req 58] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.589 b.s.d.nimbus [INFO] [req 59] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.589 b.s.d.nimbus [INFO] [req 60] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.590 b.s.d.nimbus [INFO] [req 61] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.590 b.s.d.nimbus [INFO] [req 62] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.590 b.s.d.nimbus [INFO] [req 63] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.591 b.s.d.nimbus [INFO] [req 64] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.591 b.s.d.nimbus [INFO] [req 1] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.592 b.s.d.nimbus [INFO] [req 2] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.592 b.s.d.nimbus [INFO] [req 3] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.593 b.s.d.nimbus [INFO] [req 4] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.593 b.s.d.nimbus [INFO] [req 5] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.593 b.s.d.nimbus [INFO] [req 6] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.594 b.s.d.nimbus [INFO] [req 7] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.594 b.s.d.nimbus [INFO] [req 8] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.595 b.s.d.nimbus [INFO] [req 9] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.595 b.s.d.nimbus [INFO] [req 10] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.595 b.s.d.nimbus [INFO] [req 11] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.596 b.s.d.nimbus [INFO] [req 12] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.602 b.s.d.nimbus [INFO] [req 13] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.602 b.s.d.nimbus [INFO] [req 14] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.602 b.s.d.nimbus [INFO] [req 15] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.603 b.s.d.nimbus [INFO] [req 16] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.603 b.s.d.nimbus [INFO] [req 17] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.604 b.s.d.nimbus [INFO] [req 18] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.604 b.s.d.nimbus [INFO] [req 19] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.605 b.s.d.nimbus [INFO] [req 20] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.605 b.s.d.nimbus [INFO] [req 21] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.605 b.s.d.nimbus [INFO] [req 22] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.606 b.s.d.nimbus [INFO] [req 23] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.606 b.s.d.nimbus [INFO] [req 24] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.607 b.s.d.nimbus [INFO] [req 25] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.607 b.s.d.nimbus [INFO] [req 26] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.607 b.s.d.nimbus [INFO] [req 27] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.608 b.s.d.nimbus [INFO] [req 28] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.608 b.s.d.nimbus [INFO] [req 29] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.609 b.s.d.nimbus [INFO] [req 30] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.609 b.s.d.nimbus [INFO] [req 31] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.609 b.s.d.nimbus [INFO] [req 32] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.610 b.s.d.nimbus [INFO] [req 33] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.610 b.s.d.nimbus [INFO] [req 34] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.611 b.s.d.nimbus [INFO] [req 35] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.611 b.s.d.nimbus [INFO] [req 36] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.611 b.s.d.nimbus [INFO] [req 37] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.612 b.s.d.nimbus [INFO] [req 38] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.612 b.s.d.nimbus [INFO] [req 39] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.613 b.s.d.nimbus [INFO] [req 40] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.613 b.s.d.nimbus [INFO] [req 41] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.613 b.s.d.nimbus [INFO] [req 42] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.614 b.s.d.nimbus [INFO] [req 43] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.614 b.s.d.nimbus [INFO] [req 44] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.615 b.s.d.nimbus [INFO] [req 45] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.615 b.s.d.nimbus [INFO] [req 46] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.615 b.s.d.nimbus [INFO] [req 47] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.619 b.s.d.nimbus [INFO] [req 48] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.620 b.s.d.nimbus [INFO] [req 49] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.620 b.s.d.nimbus [INFO] [req 50] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.620 b.s.d.nimbus [INFO] [req 51] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.622 b.s.d.nimbus [INFO] [req 52] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.622 b.s.d.nimbus [INFO] [req 53] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.623 b.s.d.nimbus [INFO] [req 54] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.623 b.s.d.nimbus [INFO] [req 55] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.623 b.s.d.nimbus [INFO] [req 56] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.624 b.s.d.nimbus [INFO] [req 57] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.624 b.s.d.nimbus [INFO] [req 58] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.625 b.s.d.nimbus [INFO] [req 59] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.626 b.s.d.nimbus [INFO] [req 60] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.626 b.s.d.nimbus [INFO] [req 61] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.627 b.s.d.nimbus [INFO] [req 62] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.627 b.s.d.nimbus [INFO] [req 63] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.629 b.s.d.nimbus [INFO] [req 64] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.631 b.s.d.nimbus [INFO] [req 1] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.632 b.s.d.nimbus [INFO] [req 2] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.633 b.s.d.nimbus [INFO] [req 3] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.633 b.s.d.nimbus [INFO] [req 4] Access from: principal:
op:fileDownload
2017-04-05 11:13:11.635 b.s.d.nimbus [INFO] [req 5] Access from: principal:
op:fileDownload
2017-04-05 11:13:40.660 b.s.d.nimbus [INFO] [req 6] Access from: principal:
op:getClusterInfo
2017-04-05 11:14:03.901 b.s.d.nimbus [INFO] [req 7] Access from: principal:
op:getClusterInfo
2017-04-05 11:14:35.821 b.s.d.nimbus [INFO] [req 8] Access from: principal:
op:getClusterInfo
2017-04-05 19:30:32.541 o.a.s.s.o.a.z.ClientCnxn [INFO] Client session timed
out, have not heard from server in 13337ms for sessionid 0x15b23c37c6400c6,
closing socket connection and attempting reconnect
2017-04-05 19:30:32.644 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State
change: SUSPENDED
2017-04-05 19:30:32.651 b.s.cluster [WARN] Received event :disconnected::none:
with disconnected Zookeeper.
2017-04-05 19:30:33.138 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket
connection to server xiang/127.0.0.1:2181. Will not attempt to authenticate
using SASL (unknown error)
2017-04-05 19:30:33.139 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection
established to xiang/127.0.0.1:2181, initiating session
2017-04-05 19:30:33.141 o.a.s.s.o.a.z.ClientCnxn [INFO] Session establishment
complete on server xiang/127.0.0.1:2181, sessionid = 0x15b23c37c6400c6,
negotiated timeout = 20000
2017-04-05 19:30:33.141 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State
change: RECONNECTED
2017-04-05 19:30:59.303 o.a.s.s.o.a.z.ClientCnxn [INFO] Client session timed
out, have not heard from server in 13337ms for sessionid 0x15b23c37c6400c6,
closing socket connection and attempting reconnect
2017-04-05 19:30:59.404 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State
change: SUSPENDED
2017-04-05 19:30:59.405 b.s.cluster [WARN] Received event :disconnected::none:
with disconnected Zookeeper.
2017-04-05 19:31:00.145 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket
connection to server xiang/10.240.47.205:2181. Will not attempt to authenticate
using SASL (unknown error)
2017-04-05 19:31:00.146 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection
established to xiang/10.240.47.205:2181, initiating session
2017-04-05 19:31:00.147 o.a.s.s.o.a.z.ClientCnxn [INFO] Session establishment
complete on server xiang/10.240.47.205:2181, sessionid = 0x15b23c37c6400c6,
negotiated timeout = 20000
2017-04-05 19:31:00.148 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State
change: RECONNECTED
2017-04-05 19:31:13.487 o.a.s.s.o.a.z.ClientCnxn [INFO] Client session timed
out, have not heard from server in 13340ms for sessionid 0x15b23c37c6400c6,
closing socket connection and attempting reconnect
2017-04-05 19:31:13.588 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State
change: SUSPENDED
2017-04-05 19:31:13.588 b.s.cluster [WARN] Received event :disconnected::none:
with disconnected Zookeeper.
2017-04-05 19:31:14.436 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket
connection to server xiang/127.0.0.1:2181. Will not attempt to authenticate
using SASL (unknown error)
2017-04-05 19:31:14.437 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection
established to xiang/127.0.0.1:2181, initiating session
2017-04-05 19:31:14.438 o.a.s.s.o.a.z.ClientCnxn [INFO] Session establishment
complete on server xiang/127.0.0.1:2181, sessionid = 0x15b23c37c6400c6,
negotiated timeout = 20000
2017-04-05 19:31:14.439 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State
change: RECONNECTED
2017-04-05 19:31:27.772 o.a.s.s.o.a.z.ClientCnxn [INFO] Client session timed
out, have not heard from server in 13334ms for sessionid 0x15b23c37c6400c6,
closing socket connection and attempting reconnect
2017-04-05 19:31:27.873 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State
change: SUSPENDED
2017-04-05 19:31:27.874 b.s.cluster [WARN] Received event :disconnected::none:
with disconnected Zookeeper.
2017-04-05 19:31:28.572 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket
connection to server xiang/10.240.47.205:2181. Will not attempt to authenticate
using SASL (unknown error)
2017-04-05 19:31:28.573 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection
established to xiang/10.240.47.205:2181, initiating session
2017-04-05 19:31:28.575 o.a.s.s.o.a.z.ClientCnxn [INFO] Session establishment
complete on server xiang/10.240.47.205:2181, sessionid = 0x15b23c37c6400c6,
negotiated timeout = 20000
2017-04-05 19:31:28.575 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State
change: RECONNECTED
2017-04-05 19:31:41.914 o.a.s.s.o.a.z.ClientCnxn [INFO] Client session timed
out, have not heard from server in 13339ms for sessionid 0x15b23c37c6400c6,
closing socket connection and attempting reconnect
2017-04-05 19:31:42.015 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State
change: SUSPENDED
2017-04-05 19:31:42.015 b.s.cluster [WARN] Received event :disconnected::none:
with disconnected Zookeeper.
2017-04-05 19:31:42.361 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket
connection to server xiang/127.0.0.1:2181. Will not attempt to authenticate
using SASL (unknown error)
2017-04-05 19:31:42.362 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection
established to xiang/127.0.0.1:2181, initiating session
2017-04-05 19:31:42.384 o.a.s.s.o.a.z.ClientCnxn [INFO] Session establishment
complete on server xiang/127.0.0.1:2181, sessionid = 0x15b23c37c6400c6,
negotiated timeout = 20000
2017-04-05 19:31:42.384 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State
change: RECONNECTED
2017-04-05 19:31:55.718 o.a.s.s.o.a.z.ClientCnxn [INFO] Client session timed
out, have not heard from server in 13334ms for sessionid 0x15b23c37c6400c6,
closing socket connection and attempting reconnect
2017-04-05 19:31:55.819 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State
change: SUSPENDED
2017-04-05 19:31:55.819 b.s.cluster [WARN] Received event :disconnected::none:
with disconnected Zookeeper.
2017-04-05 19:31:56.767 o.a.s.s.o.a.z.ClientCnxn [INFO] Opening socket
connection to server xiang/10.240.47.205:2181. Will not attempt to authenticate
using SASL (unknown error)
2017-04-05 19:31:56.767 o.a.s.s.o.a.z.ClientCnxn [INFO] Socket connection
established to xiang/10.240.47.205:2181, initiating session
2017-04-05 19:31:56.769 o.a.s.s.o.a.z.ClientCnxn [INFO] Session establishment
complete on server xiang/10.240.47.205:2181, sessionid = 0x15b23c37c6400c6,
negotiated timeout = 20000
2017-04-05 19:31:56.769 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State
change: RECONNECTED
2017-04-05 19:32:10.109 o.a.s.s.o.a.z.ClientCnxn [INFO] Client session timed
out, have not heard from server in 13340ms for sessionid 0x15b23c37c6400c6,
closing socket connection and attempting reconnect
2017-04-05 19:32:10.210 o.a.s.s.o.a.c.f.s.ConnectionStateManager [INFO] State
change: SUSPENDED
2017-04-05 19:32:10.211 b.s.cluster [WARN] Received event :disconnected::none:
with disconnected Zookeeper.
2017-04-05 19:32:10.212 b.s.d.nimbus [ERROR] Error when processing event
java.lang.RuntimeException:
org.apache.storm.shade.org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /storms/apm_source_topology-11-1491038533
at backtype.storm.util$wrap_in_runtime.invoke(util.clj:49)
~[storm-core-0.10.0.jar:0.10.0]
at
backtype.storm.zookeeper$exists_node_QMARK_$fn__4306.invoke(zookeeper.clj:103)
~[storm-core-0.10.0.jar:0.10.0]
at backtype.storm.zookeeper$exists_node_QMARK_.invoke(zookeeper.clj:99)
~[storm-core-0.10.0.jar:0.10.0]
at backtype.storm.zookeeper$get_data.invoke(zookeeper.clj:128)
~[storm-core-0.10.0.jar:0.10.0]
at
backtype.storm.cluster$mk_distributed_cluster_state$reify__4580.get_data(cluster.clj:114)
~[storm-core-0.10.0.jar:0.10.0]
at
backtype.storm.cluster$mk_storm_cluster_state$reify__5120.storm_base(cluster.clj:426)
~[storm-core-0.10.0.jar:0.10.0]
at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source) ~[?:?]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.7.0_21]
at java.lang.reflect.Method.invoke(Method.java:601) ~[?:1.7.0_21]
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
~[clojure-1.6.0.jar:?]
at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
~[clojure-1.6.0.jar:?]
at
backtype.storm.daemon.nimbus$read_topology_details.invoke(nimbus.clj:344)
~[storm-core-0.10.0.jar:0.10.0]
at
backtype.storm.daemon.nimbus$mk_assignments$iter__6344__6348$fn__6349.invoke(nimbus.clj:674)
~[storm-core-0.10.0.jar:0.10.0]
at clojure.lang.LazySeq.sval(LazySeq.java:40) ~[clojure-1.6.0.jar:?]
at clojure.lang.LazySeq.seq(LazySeq.java:49) ~[clojure-1.6.0.jar:?]
at clojure.lang.RT.seq(RT.java:484) ~[clojure-1.6.0.jar:?]
at clojure.core$seq.invoke(core.clj:133) ~[clojure-1.6.0.jar:?]
at clojure.core.protocols$seq_reduce.invoke(protocols.clj:30)
~[clojure-1.6.0.jar:?]
at clojure.core.protocols$fn__6078.invoke(protocols.clj:54)
~[clojure-1.6.0.jar:?]
at
clojure.core.protocols$fn__6031$G__6026__6044.invoke(protocols.clj:13)
~[clojure-1.6.0.jar:?]
at clojure.core$reduce.invoke(core.clj:6289) ~[clojure-1.6.0.jar:?]
at clojure.core$into.invoke(core.clj:6341) ~[clojure-1.6.0.jar:?]
at backtype.storm.daemon.nimbus$mk_assignments.doInvoke(nimbus.clj:673)
~[storm-core-0.10.0.jar:0.10.0]
at clojure.lang.RestFn.invoke(RestFn.java:410) ~[clojure-1.6.0.jar:?]
at
backtype.storm.daemon.nimbus$fn__6583$exec_fn__1236__auto____6584$fn__6589$fn__6590.invoke(nimbus.clj:1025)
~[storm-core-0.10.0.jar:0.10.0]
at
backtype.storm.daemon.nimbus$fn__6583$exec_fn__1236__auto____6584$fn__6589.invoke(nimbus.clj:1024)
~[storm-core-0.10.0.jar:0.10.0]
at
backtype.storm.timer$schedule_recurring$this__3721.invoke(timer.clj:102)
~[storm-core-0.10.0.jar:0.10.0]
at backtype.storm.timer$mk_timer$fn__3704$fn__3705.invoke(timer.clj:50)
[storm-core-0.10.0.jar:0.10.0]
at backtype.storm.timer$mk_timer$fn__3704.invoke(timer.clj:42)
[storm-core-0.10.0.jar:0.10.0]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
at java.lang.Thread.run(Thread.java:722) [?:1.7.0_21]
Caused by:
org.apache.storm.shade.org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /storms/apm_source_topology-11-1491038533
at
org.apache.storm.shade.org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
~[storm-core-0.10.0.jar:0.10.0]
at
org.apache.storm.shade.org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
~[storm-core-0.10.0.jar:0.10.0]
at
org.apache.storm.shade.org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
~[storm-core-0.10.0.jar:0.10.0]
at
org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl$2.call(ExistsBuilderImpl.java:172)
~[storm-core-0.10.0.jar:0.10.0]
at
org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl$2.call(ExistsBuilderImpl.java:161)
~[storm-core-0.10.0.jar:0.10.0]
at
org.apache.storm.shade.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
~[storm-core-0.10.0.jar:0.10.0]
at
org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl.pathInForeground(ExistsBuilderImpl.java:157)
~[storm-core-0.10.0.jar:0.10.0]
at
org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:148)
~[storm-core-0.10.0.jar:0.10.0]
at
org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:36)
~[storm-core-0.10.0.jar:0.10.0]
at
backtype.storm.zookeeper$exists_node_QMARK_$fn__4306.invoke(zookeeper.clj:102)
~[storm-core-0.10.0.jar:0.10.0]
... 29 more
2017-04-05 19:32:10.238 b.s.util [ERROR] Halting process: ("Error when
processing an event")
java.lang.RuntimeException: ("Error when processing an event")
at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:336)
[storm-core-0.10.0.jar:0.10.0]
at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.6.0.jar:?]
at
backtype.storm.daemon.nimbus$nimbus_data$fn__5953.invoke(nimbus.clj:97)
[storm-core-0.10.0.jar:0.10.0]
at backtype.storm.timer$mk_timer$fn__3704$fn__3705.invoke(timer.clj:71)
[storm-core-0.10.0.jar:0.10.0]
at backtype.storm.timer$mk_timer$fn__3704.invoke(timer.clj:42)
[storm-core-0.10.0.jar:0.10.0]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
at java.lang.Thread.run(Thread.java:722) [?:1.7.0_21]
2017-04-05 19:32:10.241 b.s.d.nimbus [INFO] Shutting down master
