Just as a convoluted example with core storm. If I have two spouts tasks and two bolt tasks and use localOrShuffleGrouping tuples would not be serialized and I would not encounter any potential kryo issues because the tuples are avoiding the network. However, if number of tasks stays at 2 each, and number of workers is 3 or 4 then I would have serialization issues because they would be distributed more widely and data would cross the network. That said, since you're using a fields grouping you would cross the network even with just 2 workers so that's probably not it.
On Thu, Oct 9, 2014 at 8:52 AM, Brunner, Bill <[email protected]> wrote: > Very. I am using standard fields grouping – groupBy() in Trident – in > a few places , and I am sending basic Scala data structures around in my > tuples (Maps, Lists, etc). If it were a true serialization issue, why > would my topology run at all? I feel like it should blow up on any cluster > with more than 1 node… As far as how my topology is configured, I’m not > sure what details would be useful to you. > > > > *From:* Nathan Leung [mailto:[email protected]] > *Sent:* Thursday, October 09, 2014 8:18 AM > *To:* user > *Subject:* RE: Serialization issue? > > > > That is odd then. Kryo serializes classes in a more fully qualified manner > the first time it sees the class for an object, and records and index of > closes out has seen before. It serializes classes by index when it sees > them subsequently. On deserialization it rebuilds the index as it > encounters new class information. Your error means that it sees a class > index value that it hasn't rebuilt yet. The only time I've run into this > kind of error was when I was mucking with custom serialization. > > What type of grouping are you using, and what type of data are you sending > your collection? Also, how is your topology configured? Have you try > writing a standalone program to serialize your data with Kryo outside of > storm? > > On Oct 9, 2014 7:57 AM, "Brunner, Bill" <[email protected]> wrote: > > 0.9.2-incubating, and all my external jar dependencies are on a network > share and symlinked to the /storm/lib folder (not that it would matter, but > this was also happening on a completely different environment when the jars > were not symlinked). > > > > *From:* Nathan Leung [mailto:[email protected]] > *Sent:* Thursday, October 09, 2014 7:42 AM > *To:* user > *Subject:* Re: Serialization issue? > > > > Can you double check that you're running the same version on all three > nodes? > > On Oct 9, 2014 7:35 AM, "Brunner, Bill" <[email protected]> wrote: > > I am seeing serialization issues on a single worker node, while the other > nodes are unaffected. I usually get the error shortly after the topology is > loaded. It happens on a single node any time the total worker nodes in the > cluster is > 2, ie, the topology runs w/o error on 2 nodes (1 worker each), > but when I introduce a 3rd node, that node fails with the serialization > error. Similarly with 4 worker nodes, 3 run and the 4th fails. I have > included the stack trace below. FYI I am using the default serializer. I > have tried cycling nimbus/zk and each node many times, and it does not help > (I have also done a complete reinstall on the failing node(s)) Hopefully > someone can help. Thanks > > > > Bill > > > > 2014-10-06 11:17:30 b.s.util [ERROR] Async loop died! > > java.lang.RuntimeException: com.esotericsoftware.kryo.KryoException: > Encountered unregistered class ID: 38 > > at > backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128) > ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] > > at > backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) > ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] > > at > backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) > ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] > > at > backtype.storm.daemon.executor$fn__5641$fn__5653$fn__5700.invoke(executor.clj:746) > ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] > > at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431) > ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] > > at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na] > > at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45] > > Caused by: com.esotericsoftware.kryo.KryoException: Encountered > unregistered class ID: 38 > > at > com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:119) > ~[kryo-2.21.jar:na] > > at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:610) > ~[kryo-2.21.jar:na] > > at > com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:721) > ~[kryo-2.21.jar:na] > > at > com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:109) > ~[kryo-2.21.jar:na] > > at > com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:18) > ~[kryo-2.21.jar:na] > > at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:629) > ~[kryo-2.21.jar:na] > > at > backtype.storm.serialization.KryoValuesDeserializer.deserializeFrom(KryoValuesDeserializer.java:38) > ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] > > at > backtype.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:53) > ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] > > at > backtype.storm.daemon.executor$mk_task_receiver$fn__5564.invoke(executor.clj:396) > ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] > > at > backtype.storm.disruptor$clojure_handler$reify__745.onEvent(disruptor.clj:58) > ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] > > at > backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125) > ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] > > ... 6 common frames omitted > ------------------------------ > > This message, and any attachments, is for the intended recipient(s) only, > may contain information that is privileged, confidential and/or proprietary > and subject to important terms and conditions available at > http://www.bankofamerica.com/emaildisclaimer. If you are not the intended > recipient, please delete this message. > ------------------------------ > > This message, and any attachments, is for the intended recipient(s) only, > may contain information that is privileged, confidential and/or proprietary > and subject to important terms and conditions available at > http://www.bankofamerica.com/emaildisclaimer. If you are not the intended > recipient, please delete this message. > ------------------------------ > This message, and any attachments, is for the intended recipient(s) only, > may contain information that is privileged, confidential and/or proprietary > and subject to important terms and conditions available at > http://www.bankofamerica.com/emaildisclaimer. If you are not the intended > recipient, please delete this message. >
