Hi Sa Li,
As per log detail it's cross version error. Because you had said in first
mail you are using storm-0.9.1 version for coding or build topology and
running on storm-core-0.9.2-incubating.
Try to install, build and run your topology on same version of storm it
will solve your problem.
On Thu, Nov 27, 2014 at 3:51 AM, Sa Li <sa.in.v...@gmail.com> wrote:
> Yeah, I found lot of errors, the error in supervisor which is same as node
> of nimbus:
> at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.5.1.jar:na]
> at clojure.core$apply.invoke(core.clj:619) ~[clojure-1.5.1.jar:na]
> at clojure.core$partial$fn__4190.doInvoke(core.clj:2396)
> ~[clojure-1.5.1.jar:na]
> at clojure.lang.RestFn.invoke(RestFn.java:397)
> ~[clojure-1.5.1.jar:na]
> at
> backtype.storm.event$event_manager$fn__2378.invoke(event.clj:39)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_55]
> Caused by: java.io.InvalidClassException: clojure.lang.APersistentMap;
> local class incompatible: stream classdesc serialVersionUID =
> 270281984708184947, local class serialVersionUID = 8648225932767613808
> at
> java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:617)
> ~[na:1.7.0_55]
> at
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
> ~[na:1.7.0_55]
> at
> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
> ~[na:1.7.0_55]
> at
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
> ~[na:1.7.0_55]
> at
> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
> ~[na:1.7.0_55]
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
> ~[na:1.7.0_55]
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> ~[na:1.7.0_55]
> at
> java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
> ~[na:1.7.0_55]
> at java.util.HashMap.readObject(HashMap.java:1184) ~[na:1.7.0_55]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[na:1.7.0_55]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> ~[na:1.7.0_55]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.7.0_55]
> at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_55]
> at
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
> ~[na:1.7.0_55]
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
> ~[na:1.7.0_55]
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> ~[na:1.7.0_55]
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> ~[na:1.7.0_55]
> at
> java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
> ~[na:1.7.0_55]
> at backtype.storm.utils.Utils.deserialize(Utils.java:89)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> ... 11 common frames omitted
> 2014-10-28 15:45:14 b.s.event [ERROR] Error when processing event
> java.lang.RuntimeException: java.io.InvalidClassException:
> clojure.lang.APersistentMap; local class incompatible: stream classdesc
> serialVersionUID = 270281984708184947, local class serialVersionUID =
> 8648225932767613808
> at backtype.storm.utils.Utils.deserialize(Utils.java:93)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at backtype.storm.utils.LocalState.snapshot(LocalState.java:45)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at backtype.storm.utils.LocalState.get(LocalState.java:56)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.daemon.supervisor$mk_synchronize_supervisor$this__6330.invoke(supervisor.clj:307)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at
> backtype.storm.event$event_manager$fn__2378.invoke(event.clj:39)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_55]
> Caused by: java.io.InvalidClassException: clojure.lang.APersistentMap;
> local class incompatible: stream classdesc serialVersionUID =
> 270281984708184947, local class serialVersionUID = 8648225932767613808
> at
> java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:617)
> ~[na:1.7.0_55]
> at
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
> ~[na:1.7.0_55]
> at
> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
> ~[na:1.7.0_55]
> at
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
> ~[na:1.7.0_55]
> at
> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
> ~[na:1.7.0_55]
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
> ~[na:1.7.0_55]
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> ~[na:1.7.0_55]
> at
> java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
> ~[na:1.7.0_55]
> at java.util.HashMap.readObject(HashMap.java:1184) ~[na:1.7.0_55]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[na:1.7.0_55]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> ~[na:1.7.0_55]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.7.0_55]
> at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_55]
> at
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
> ~[na:1.7.0_55]
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
> ~[na:1.7.0_55]
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> ~[na:1.7.0_55]
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> ~[na:1.7.0_55]
> at
> java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
> ~[na:1.7.0_55]
> at backtype.storm.utils.Utils.deserialize(Utils.java:89)
> ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating]
> ... 6 common frames omitted
> 2014-10-28 15:45:14 b.s.util [INFO] Halting process: ("Error when
> processing an event")
> 2014-10-28 15:45:14 b.s.util [INFO] Halting process: ("Error when
> processing an event")
>
> This is the errors from supervisor node:
>
> 2014-11-26 13:22:34 b.s.d.supervisor [INFO] Starting Supervisor with conf
> {"dev.zookeeper.path" "/tmp/dev-storm-zookeeper",
> "topology.tick.tuple.freq.secs" nil,
> "topology.builtin.metrics.bucket.size.secs" 60,
> "topology.fall.back.on.java.serialization" true,
> "topology.max.error.report.per.interval" 5, "zmq.linger.millis" 5000,
> "topology.skip.missing.kryo.registrations" false,
> "storm.messaging.netty.client_worker_threads" 1, "ui.childopts" "-Xms1024m
> -Djava.net.preferIPv4Stack=true", "storm.zookeeper.session.timeout" 20000,
> "nimbus.reassign" true, "topology.trident.batch.emit.interval.millis" 100,
> "nimbus.monitor.freq.secs" 10, "logviewer.childopts" "-Xmx512m
> -Djava.net.preferIPv4Stack=true", "java.library.path"
> "/usr/lib/jvm/java-7-openjdk-amd64", "topology.executor.send.buffer.size"
> 1024, "storm.local.dir" "/app/storm", "storm.messaging.netty.buffer_size"
> 5242880, "supervisor.worker.start.timeout.secs" 120,
> "topology.enable.message.timeouts" true, "nimbus.cleanup.inbox.freq.secs"
> 600, "nimbus.inbox.jar.expiration.secs" 3600, "drpc.worker.threads" 64,
> "topology.worker.shared.thread.pool.size" 4, "nimbus.host" "10.100.70.128",
> "storm.messaging.netty.min_wait_ms" 100, "storm.zookeeper.port" 2181,
> "transactional.zookeeper.port" nil, "topology.executor.receive.buffer.size"
> 1024, "transactional.zookeeper.servers" nil, "storm.zookeeper.root"
> "/storm", "storm.zookeeper.retry.intervalceiling.millis" 30000,
> "supervisor.enable" true, "storm.messaging.netty.server_worker_threads" 1,
> "storm.zookeeper.servers" ["10.100.70.128" "10.100.70.28" "10.100.70.29"],
> "transactional.zookeeper.root" "/transactional", "topology.acker.executors"
> nil, "topology.transfer.buffer.size" 1024, "topology.worker.childopts" nil,
> "drpc.queue.size" 128, "worker.childopts" "-Xmx768m
> -Djava.net.preferIPv4Stack=true", "supervisor.heartbeat.frequency.secs" 5,
> "topology.error.throttle.interval.secs" 10, "zmq.hwm" 0, "drpc.port" 3772,
> "supervisor.monitor.frequency.secs" 3, "drpc.childopts" "-Xmx768m",
> "topology.receiver.buffer.size" 8, "task.heartbeat.frequency.secs" 3,
> "topology.tasks" nil, "storm.messaging.netty.max_retries" 30,
> "topology.spout.wait.strategy"
> "backtype.storm.spout.SleepSpoutWaitStrategy", "topology.max.spout.pending"
> nil, "storm.zookeeper.retry.interval" 1000, "
> topology.sleep.spout.wait.strategy.time.ms" 1,
> "nimbus.topology.validator"
> "backtype.storm.nimbus.DefaultTopologyValidator", "supervisor.slots.ports"
> [6700 6701 6702 6703], "topology.debug" false, "nimbus.task.launch.secs"
> 120, "nimbus.supervisor.timeout.secs" 60, "topology.message.timeout.secs"
> 300, "task.refresh.poll.secs" 10, "topology.workers" 1,
> "supervisor.childopts" "-Xms1024m -Djava.net.preferIPv4Stack=true",
> "nimbus.thrift.port" 6627, "topology.stats.sample.rate" 0.05,
> "worker.heartbeat.frequency.secs" 1, "topology.tuple.serializer"
> "backtype.storm.serialization.types.ListDelegateSerializer",
> "topology.disruptor.wait.strategy"
> "com.lmax.disruptor.BlockingWaitStrategy", "nimbus.task.timeout.secs" 30,
> "storm.zookeeper.connection.timeout" 15000, "topology.kryo.factory"
> "backtype.storm.serialization.DefaultKryoFactory", "drpc.invocations.port"
> 3773, "logviewer.port" 8000, "zmq.threads" 1, "storm.zookeeper.retry.times"
> 5, "storm.thrift.transport"
> "backtype.storm.security.auth.SimpleTransportPlugin",
> "topology.state.synchronization.timeout.secs" 60,
> "supervisor.worker.timeout.secs" 30, "nimbus.file.copy.expiration.secs"
> 600, "storm.messaging.transport" "backtype.storm.messaging.zmq", "
> logviewer.appender.name" "A1", "storm.messaging.netty.max_wait_ms" 1000,
> "drpc.request.timeout.secs" 600, "storm.local.mode.zmq" false, "ui.port"
> 8080, "nimbus.childopts" "-Xms2048m -Djava.net.preferIPv4Stack=true",
> "storm.cluster.mode" "distributed", "topology.optimize" true,
> "topology.max.task.parallelism" nil}
> 2014-11-26 13:22:34 c.n.c.f.i.CuratorFrameworkImpl [INFO] Starting
> 2014-11-26 13:22:34 o.a.z.ZooKeeper [INFO] Initiating client connection,
> connectString=10.100.70.128:2181,10.100.70.28:2181,10.100.70.29:2181
> sessionTimeout=20000 watcher=com.netflix.curator.ConnectionState@782831bc
> 2014-11-26 13:22:34 o.a.z.ClientCnxn [INFO] Opening socket connection to
> server /10.100.70.28:2181
> 2014-11-26 13:22:34 o.a.z.ClientCnxn [INFO] Socket connection established
> to pof-kstorm-dev1.pof.local/10.100.70.28:2181, initiating session
> 2014-11-26 13:22:34 o.a.z.ClientCnxn [INFO] Session establishment complete
> on server pof-kstorm-dev1.pof.local/10.100.70.28:2181, sessionid =
> 0x249edccb5f00047, negotiated timeout = 20000
> 2014-11-26 13:22:34 b.s.zookeeper [INFO] Zookeeper state update:
> :connected:none
> 2014-11-26 13:22:35 o.a.z.ZooKeeper [INFO] Session: 0x249edccb5f00047
> closed
> 2014-11-26 13:22:35 o.a.z.ClientCnxn [INFO] EventThread shut down
> 2014-11-26 13:22:35 c.n.c.f.i.CuratorFrameworkImpl [INFO] Starting
> 2014-11-26 13:22:35 o.a.z.ZooKeeper [INFO] Initiating client connection,
> connectString=10.100.70.128:2181,10.100.70.28:2181,10.100.70.29:2181/storm
> sessionTimeout=20000 watcher=com.netflix.curator.ConnectionState@478b7093
> 2014-11-26 13:22:35 o.a.z.ClientCnxn [INFO] Opening socket connection to
> server /10.100.70.29:2181
> 2014-11-26 13:22:35 o.a.z.ClientCnxn [INFO] Socket connection established
> to pof-kstorm-dev2/10.100.70.29:2181, initiating session
> 2014-11-26 13:22:35 o.a.z.ClientCnxn [INFO] Session establishment complete
> on server pof-kstorm-dev2/10.100.70.29:2181, sessionid =
> 0x349edccb6370045, negotiated timeout = 20000
> 2014-11-26 13:22:35 b.s.d.supervisor [INFO] Starting supervisor with id
> 094ce243-f422-434e-81c0-7361d9fbb606 at host pof-kstorm-dev1.pof.local
> 2014-11-26 13:22:35 b.s.event [ERROR] Error when processing event
> java.lang.RuntimeException: java.io.InvalidClassException:
> backtype.storm.daemon.common.Assignment; local class incompatible: stream
> classdesc serialVersionUID = 1582431921447335237, local class
> serialVersionUID = -5102131895282047148
> at backtype.storm.utils.Utils.deserialize(Utils.java:69)
> ~[storm-core-0.9.0.1.jar:na]
> at
> backtype.storm.cluster$maybe_deserialize.invoke(cluster.clj:178)
> ~[storm-core-0.9.0.1.jar:na]
> at
> backtype.storm.cluster$mk_storm_cluster_state$reify__2115.assignment_info(cluster.clj:233)
> ~[storm-core-0.9.0.1.jar:na]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[na:1.7.0_67]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> ~[na:1.7.0_67]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.7.0_67]
> at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_67]
> at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
> ~[clojure-1.4.0.jar:na]
> at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
> ~[clojure-1.4.0.jar:na]
> at
> backtype.storm.daemon.supervisor$assignments_snapshot$iter__6028__6032$fn__6033.invoke(supervisor.clj:25)
> ~[storm-core-0.9.0.1.jar:na]
> at clojure.lang.LazySeq.sval(LazySeq.java:42)
> ~[clojure-1.4.0.jar:na]
> at clojure.lang.LazySeq.seq(LazySeq.java:60)
> ~[clojure-1.4.0.jar:na]
> at clojure.lang.RT.seq(RT.java:473) ~[clojure-1.4.0.jar:na]
> at clojure.core$seq.invoke(core.clj:133) ~[clojure-1.4.0.jar:na]
> at clojure.core$dorun.invoke(core.clj:2725) ~[clojure-1.4.0.jar:na]
> at clojure.core$doall.invoke(core.clj:2741) ~[clojure-1.4.0.jar:na]
> at
> backtype.storm.daemon.supervisor$assignments_snapshot.invoke(supervisor.clj:25)
> ~[storm-core-0.9.0.1.jar:na]
> at
> backtype.storm.daemon.supervisor$mk_synchronize_supervisor$this__6251.invoke(supervisor.clj:263)
> ~[storm-core-0.9.0.1.jar:na]
> at
> backtype.storm.event$event_manager$fn__3072.invoke(event.clj:24)
> ~[storm-core-0.9.0.1.jar:na]
> at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
> Caused by: java.io.InvalidClassException:
> backtype.storm.daemon.common.Assignment; local class incompatible: stream
> classdesc serialVersionUID = 1582431921447335237, local class
> serialVersionUID = -5102131895282047148
> at
> java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:617)
> ~[na:1.7.0_67]
> at
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
> ~[na:1.7.0_67]
> at
> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
> ~[na:1.7.0_67]
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
> ~[na:1.7.0_67]
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> ~[na:1.7.0_67]
> at
> java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
> ~[na:1.7.0_67]
> at backtype.storm.utils.Utils.deserialize(Utils.java:65)
> ~[storm-core-0.9.0.1.jar:na]
> ... 20 common frames omitted
> 2014-11-26 13:22:35 b.s.util [INFO] Halting process: ("Error when
> processing an event")
>
> Really can't diagnose what kind of problems are they.
>
> Thanks
>
> Alec
>
>
> On Wed, Nov 26, 2014 at 1:27 PM, Harsha <st...@harsha.io> wrote:
>
>>
>> do you see any errors in logs/supervisor.log
>> -Harsha
>>
>> On Wed, Nov 26, 2014, at 01:23 PM, Sa Li wrote:
>>
>> Seems I never be able to make supervisors started properly.
>>
>> On Wed, Nov 26, 2014 at 12:52 PM, Sa Li <sa.in.v...@gmail.com> wrote:
>>
>> I am using storm-0.9.0.1.
>>
>> thanks
>>
>>
>>
>> On Wed, Nov 26, 2014 at 12:48 PM, Sa Li <sa.in.v...@gmail.com> wrote:
>>
>>
>> Hi, all
>>
>> I have configured a storm cluster, 1 nimbus 2 supervisors, but seems I
>> have trouble to start supervisors. Here is the storm.yaml in nimbus:
>>
>> storm.zookeeper.servers:
>> - "10.100.70.128"
>> - "10.100.70.28"
>> - "10.100.70.29"
>> storm.zookeeper.port: 2181
>> nimbus.host: "10.100.70.128"
>> storm.local.dir: "/app/storm"
>> java.library.path: "/usr/lib/jvm/java-7-openjdk-amd64"
>> supervisor.slots.ports:
>> - 6700
>> - 6701
>> - 6702
>> - 6703
>> nimbus.childopts: "-Xms2048m -Djava.net.preferIPv4Stack=true"
>> ui.childopts: "-Xms1024m -Djava.net.preferIPv4Stack=true"
>> logviewer.childopts: "-Xmx512m"
>> supervisor.childopts: "-Xms1024m -Djava.net.preferIPv4Stack=true"
>> worker.childopts: "-Xmx768m -Djava.net.preferIPv4Stack=true"
>> topology.trident.batch.emit.interval.millis: 100
>> topology.message.timeout.secs: 300
>>
>> This the yaml in supervisor nodes:
>>
>> storm.zookeeper.servers:
>> - "10.100.70.128"
>> - "10.100.70.28"
>> - "10.100.70.29"
>> storm.zookeeper.port: 2181
>> nimbus.host: "10.100.70.128"
>> storm.local.dir: "/app/storm"
>> java.library.path: "/usr/lib/jvm/java-7-openjdk-amd64
>> nimbus.childopts: "-Xms2048m -Djava.net.preferIPv4Stack=true"
>> ui.childopts: "-Xms1024m -Djava.net.preferIPv4Stack=true"
>> logviewer.childopts: "-Xmx512m"
>> supervisor.childopts: "-Xms1024m -Djava.net.preferIPv4Stack=true"
>> worker.childopts: "-Xmx768m -Djava.net.preferIPv4Stack=true"
>> topology.trident.batch.emit.interval.millis: 100
>> topology.message.timeout.secs: 300
>>
>> I start nimbus, ui, supervisors by supervisord, and I found
>> supervisorctl status
>> storm-supervisor BACKOFF Exited too quickly (process
>> log may have details)
>>
>> And I start it in manual, the same thing, I see 0 supervisors in UI, but
>> when I check in zookeeper, seems to detect only one supervisor which may be
>> the one in the same node with nimbus.:
>> [zk: localhost:2181(CONNECTED) 2] ls /storm/supervisors
>> [c557e7e8-4549-4965-81b3-004334f0e831]
>>
>> Anyone how to start the supervisors properly?
>>
>> thanks
>>
>>
>> Alec
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
>
--
Regards,
Swapnil K. Joshi
