Yeah, I found lot of errors, the error in supervisor which is same as node
of nimbus:
at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.5.1.jar:na]
at clojure.core$apply.invoke(core.clj:619) ~[clojure-1.5.1.jar:na]
at clojure.core$partial$fn__4190.doInvoke(core.clj:2396)
~[clojure-1.5.1.jar:na]
at clojure.lang.RestFn.invoke(RestFn.java:397)
~[clojure-1.5.1.jar:na]
at backtype.storm.event$event_manager$fn__2378.invoke(event.clj:39)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_55]
Caused by: java.io.InvalidClassException: clojure.lang.APersistentMap;
local class incompatible: stream classdesc serialVersionUID =
270281984708184947, local class serialVersionUID = 8648225932767613808
at
java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:617)
~[na:1.7.0_55]
at
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
~[na:1.7.0_55]
at
java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
~[na:1.7.0_55]
at
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
~[na:1.7.0_55]
at
java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
~[na:1.7.0_55]
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
~[na:1.7.0_55]
at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
~[na:1.7.0_55]
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
~[na:1.7.0_55]
at java.util.HashMap.readObject(HashMap.java:1184) ~[na:1.7.0_55]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[na:1.7.0_55]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
~[na:1.7.0_55]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[na:1.7.0_55]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_55]
at
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
~[na:1.7.0_55]
at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
~[na:1.7.0_55]
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
~[na:1.7.0_55]
at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
~[na:1.7.0_55]
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
~[na:1.7.0_55]
at backtype.storm.utils.Utils.deserialize(Utils.java:89)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
... 11 common frames omitted
2014-10-28 15:45:14 b.s.event [ERROR] Error when processing event
java.lang.RuntimeException: java.io.InvalidClassException:
clojure.lang.APersistentMap; local class incompatible: stream classdesc
serialVersionUID = 270281984708184947, local class serialVersionUID =
8648225932767613808
at backtype.storm.utils.Utils.deserialize(Utils.java:93)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.utils.LocalState.snapshot(LocalState.java:45)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.utils.LocalState.get(LocalState.java:56)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at
backtype.storm.daemon.supervisor$mk_synchronize_supervisor$this__6330.invoke(supervisor.clj:307)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at backtype.storm.event$event_manager$fn__2378.invoke(event.clj:39)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_55]
Caused by: java.io.InvalidClassException: clojure.lang.APersistentMap;
local class incompatible: stream classdesc serialVersionUID =
270281984708184947, local class serialVersionUID = 8648225932767613808
at
java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:617)
~[na:1.7.0_55]
at
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
~[na:1.7.0_55]
at
java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
~[na:1.7.0_55]
at
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
~[na:1.7.0_55]
at
java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
~[na:1.7.0_55]
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
~[na:1.7.0_55]
at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
~[na:1.7.0_55]
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
~[na:1.7.0_55]
at java.util.HashMap.readObject(HashMap.java:1184) ~[na:1.7.0_55]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[na:1.7.0_55]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
~[na:1.7.0_55]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[na:1.7.0_55]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_55]
at
java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
~[na:1.7.0_55]
at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
~[na:1.7.0_55]
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
~[na:1.7.0_55]
at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
~[na:1.7.0_55]
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
~[na:1.7.0_55]
at backtype.storm.utils.Utils.deserialize(Utils.java:89)
~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
... 6 common frames omitted
2014-10-28 15:45:14 b.s.util [INFO] Halting process: ("Error when
processing an event")
2014-10-28 15:45:14 b.s.util [INFO] Halting process: ("Error when
processing an event")
This is the errors from supervisor node:
2014-11-26 13:22:34 b.s.d.supervisor [INFO] Starting Supervisor with conf
{"dev.zookeeper.path" "/tmp/dev-storm-zookeeper",
"topology.tick.tuple.freq.secs" nil,
"topology.builtin.metrics.bucket.size.secs" 60,
"topology.fall.back.on.java.serialization" true,
"topology.max.error.report.per.interval" 5, "zmq.linger.millis" 5000,
"topology.skip.missing.kryo.registrations" false,
"storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xms1024m
-Djava.net.preferIPv4Stack=true", "storm.zookeeper.session.timeout" 20000,
"nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 100,
"nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx512m
-Djava.net.preferIPv4Stack=true", "java.library.path"
"/usr/lib/jvm/java-7-openjdk-amd64", "topology.executor.send.buffer.size"
1024, "storm.local.dir" "/app/storm", "storm.messaging.netty.buffer_size"
5242880, "supervisor.worker.start.timeout.secs" 120,
"topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs"
600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64,
"topology.worker.shared.thread.pool.size" 4, "nimbus.host" "10.100.70.128",
"storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2181,
"transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size"
1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root"
"/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000,
"supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1,
"storm.zookeeper.servers" ["10.100.70.128" "10.100.70.28" "10.100.70.29"],
"transactional.zookeeper.root" "/transactional", "topology.acker.executors"
nil, "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil,
"drpc.queue.size" 128, "worker.childopts" "-Xmx768m
-Djava.net.preferIPv4Stack=true", "supervisor.heartbeat.frequency.secs" 5,
"topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772,
"supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m",
"topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3,
"topology.tasks" nil, "storm.messaging.netty.max_retries" 30,
"topology.spout.wait.strategy"
"backtype.storm.spout.SleepSpoutWaitStrategy", "topology.max.spout.pending"
nil, "storm.zookeeper.retry.interval" 1000, "
topology.sleep.spout.wait.strategy.time.ms" 1, "nimbus.topology.validator"
"backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports"
[6700 6701 6702 6703], "topology.debug" false, "nimbus.task.launch.secs"
120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs"
300, "task.refresh.poll.secs" 10, "topology.workers" 1,
"supervisor.childopts" "-Xms1024m -Djava.net.preferIPv4Stack=true",
"nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05,
"worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer"
"backtype.storm.serialization.types.ListDelegateSerializer",
"topology.disruptor.wait.strategy"
"com.lmax.disruptor.BlockingWaitStrategy", "nimbus.task.timeout.secs" 30,
"storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory"
"backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port"
3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times"
5, "storm.thrift.transport"
"backtype.storm.security.auth.SimpleTransportPlugin",
"topology.state.synchronization.timeout.secs" 60,
"supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs"
600, "storm.messaging.transport" "backtype.storm.messaging.zmq", "
logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000,
"drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port"
8080, "nimbus.childopts" "-Xms2048m -Djava.net.preferIPv4Stack=true",
"storm.cluster.mode" "distributed", "topology.optimize" true,
"topology.max.task.parallelism" nil}
2014-11-26 13:22:34 c.n.c.f.i.CuratorFrameworkImpl [INFO] Starting
2014-11-26 13:22:34 o.a.z.ZooKeeper [INFO] Initiating client connection,
connectString=10.100.70.128:2181,10.100.70.28:2181,10.100.70.29:2181
sessionTimeout=20000 watcher=com.netflix.curator.ConnectionState@782831bc
2014-11-26 13:22:34 o.a.z.ClientCnxn [INFO] Opening socket connection to
server /10.100.70.28:2181
2014-11-26 13:22:34 o.a.z.ClientCnxn [INFO] Socket connection established
to pof-kstorm-dev1.pof.local/10.100.70.28:2181, initiating session
2014-11-26 13:22:34 o.a.z.ClientCnxn [INFO] Session establishment complete
on server pof-kstorm-dev1.pof.local/10.100.70.28:2181, sessionid =
0x249edccb5f00047, negotiated timeout = 20000
2014-11-26 13:22:34 b.s.zookeeper [INFO] Zookeeper state update:
:connected:none
2014-11-26 13:22:35 o.a.z.ZooKeeper [INFO] Session: 0x249edccb5f00047 closed
2014-11-26 13:22:35 o.a.z.ClientCnxn [INFO] EventThread shut down
2014-11-26 13:22:35 c.n.c.f.i.CuratorFrameworkImpl [INFO] Starting
2014-11-26 13:22:35 o.a.z.ZooKeeper [INFO] Initiating client connection,
connectString=10.100.70.128:2181,10.100.70.28:2181,10.100.70.29:2181/storm
sessionTimeout=20000 watcher=com.netflix.curator.ConnectionState@478b7093
2014-11-26 13:22:35 o.a.z.ClientCnxn [INFO] Opening socket connection to
server /10.100.70.29:2181
2014-11-26 13:22:35 o.a.z.ClientCnxn [INFO] Socket connection established
to pof-kstorm-dev2/10.100.70.29:2181, initiating session
2014-11-26 13:22:35 o.a.z.ClientCnxn [INFO] Session establishment complete
on server pof-kstorm-dev2/10.100.70.29:2181, sessionid = 0x349edccb6370045,
negotiated timeout = 20000
2014-11-26 13:22:35 b.s.d.supervisor [INFO] Starting supervisor with id
094ce243-f422-434e-81c0-7361d9fbb606 at host pof-kstorm-dev1.pof.local
2014-11-26 13:22:35 b.s.event [ERROR] Error when processing event
java.lang.RuntimeException: java.io.InvalidClassException:
backtype.storm.daemon.common.Assignment; local class incompatible: stream
classdesc serialVersionUID = 1582431921447335237, local class
serialVersionUID = -5102131895282047148
at backtype.storm.utils.Utils.deserialize(Utils.java:69)
~[storm-core-0.9.0.1.jar:na]
at backtype.storm.cluster$maybe_deserialize.invoke(cluster.clj:178)
~[storm-core-0.9.0.1.jar:na]
at
backtype.storm.cluster$mk_storm_cluster_state$reify__2115.assignment_info(cluster.clj:233)
~[storm-core-0.9.0.1.jar:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[na:1.7.0_67]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
~[na:1.7.0_67]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[na:1.7.0_67]
at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_67]
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
~[clojure-1.4.0.jar:na]
at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
~[clojure-1.4.0.jar:na]
at
backtype.storm.daemon.supervisor$assignments_snapshot$iter__6028__6032$fn__6033.invoke(supervisor.clj:25)
~[storm-core-0.9.0.1.jar:na]
at clojure.lang.LazySeq.sval(LazySeq.java:42)
~[clojure-1.4.0.jar:na]
at clojure.lang.LazySeq.seq(LazySeq.java:60) ~[clojure-1.4.0.jar:na]
at clojure.lang.RT.seq(RT.java:473) ~[clojure-1.4.0.jar:na]
at clojure.core$seq.invoke(core.clj:133) ~[clojure-1.4.0.jar:na]
at clojure.core$dorun.invoke(core.clj:2725) ~[clojure-1.4.0.jar:na]
at clojure.core$doall.invoke(core.clj:2741) ~[clojure-1.4.0.jar:na]
at
backtype.storm.daemon.supervisor$assignments_snapshot.invoke(supervisor.clj:25)
~[storm-core-0.9.0.1.jar:na]
at
backtype.storm.daemon.supervisor$mk_synchronize_supervisor$this__6251.invoke(supervisor.clj:263)
~[storm-core-0.9.0.1.jar:na]
at backtype.storm.event$event_manager$fn__3072.invoke(event.clj:24)
~[storm-core-0.9.0.1.jar:na]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
Caused by: java.io.InvalidClassException:
backtype.storm.daemon.common.Assignment; local class incompatible: stream
classdesc serialVersionUID = 1582431921447335237, local class
serialVersionUID = -5102131895282047148
at
java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:617)
~[na:1.7.0_67]
at
java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
~[na:1.7.0_67]
at
java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
~[na:1.7.0_67]
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
~[na:1.7.0_67]
at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
~[na:1.7.0_67]
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
~[na:1.7.0_67]
at backtype.storm.utils.Utils.deserialize(Utils.java:65)
~[storm-core-0.9.0.1.jar:na]
... 20 common frames omitted
2014-11-26 13:22:35 b.s.util [INFO] Halting process: ("Error when
processing an event")
Really can't diagnose what kind of problems are they.
Thanks
Alec
On Wed, Nov 26, 2014 at 1:27 PM, Harsha <st...@harsha.io> wrote:
>
> do you see any errors in logs/supervisor.log
> -Harsha
>
> On Wed, Nov 26, 2014, at 01:23 PM, Sa Li wrote:
>
> Seems I never be able to make supervisors started properly.
>
> On Wed, Nov 26, 2014 at 12:52 PM, Sa Li <sa.in.v...@gmail.com> wrote:
>
> I am using storm-0.9.0.1.
>
> thanks
>
>
>
> On Wed, Nov 26, 2014 at 12:48 PM, Sa Li <sa.in.v...@gmail.com> wrote:
>
>
> Hi, all
>
> I have configured a storm cluster, 1 nimbus 2 supervisors, but seems I
> have trouble to start supervisors. Here is the storm.yaml in nimbus:
>
> storm.zookeeper.servers:
> - "10.100.70.128"
> - "10.100.70.28"
> - "10.100.70.29"
> storm.zookeeper.port: 2181
> nimbus.host: "10.100.70.128"
> storm.local.dir: "/app/storm"
> java.library.path: "/usr/lib/jvm/java-7-openjdk-amd64"
> supervisor.slots.ports:
> - 6700
> - 6701
> - 6702
> - 6703
> nimbus.childopts: "-Xms2048m -Djava.net.preferIPv4Stack=true"
> ui.childopts: "-Xms1024m -Djava.net.preferIPv4Stack=true"
> logviewer.childopts: "-Xmx512m"
> supervisor.childopts: "-Xms1024m -Djava.net.preferIPv4Stack=true"
> worker.childopts: "-Xmx768m -Djava.net.preferIPv4Stack=true"
> topology.trident.batch.emit.interval.millis: 100
> topology.message.timeout.secs: 300
>
> This the yaml in supervisor nodes:
>
> storm.zookeeper.servers:
> - "10.100.70.128"
> - "10.100.70.28"
> - "10.100.70.29"
> storm.zookeeper.port: 2181
> nimbus.host: "10.100.70.128"
> storm.local.dir: "/app/storm"
> java.library.path: "/usr/lib/jvm/java-7-openjdk-amd64
> nimbus.childopts: "-Xms2048m -Djava.net.preferIPv4Stack=true"
> ui.childopts: "-Xms1024m -Djava.net.preferIPv4Stack=true"
> logviewer.childopts: "-Xmx512m"
> supervisor.childopts: "-Xms1024m -Djava.net.preferIPv4Stack=true"
> worker.childopts: "-Xmx768m -Djava.net.preferIPv4Stack=true"
> topology.trident.batch.emit.interval.millis: 100
> topology.message.timeout.secs: 300
>
> I start nimbus, ui, supervisors by supervisord, and I found
> supervisorctl status
> storm-supervisor BACKOFF Exited too quickly (process
> log may have details)
>
> And I start it in manual, the same thing, I see 0 supervisors in UI, but
> when I check in zookeeper, seems to detect only one supervisor which may be
> the one in the same node with nimbus.:
> [zk: localhost:2181(CONNECTED) 2] ls /storm/supervisors
> [c557e7e8-4549-4965-81b3-004334f0e831]
>
> Anyone how to start the supervisors properly?
>
> thanks
>
>
> Alec
>
>
>
>
>
>
>
>
>
