[
https://issues.apache.org/jira/browse/STORM-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17079063#comment-17079063
]
Simon Cooper commented on STORM-3582:
-------------------------------------
We have found the issue that was causing our exceptions - in our bolts we have
multiple threads that were accessing OutputCollectors concurrently. In storm 1,
it seems that collectors are thread-safe, but in storm 2 they are not. This
meant the serialization buffer was being written to by multiple threads at the
same time, causing data corruption which was only spotted when it was
deserialized on the target worker, resulting in seemingly random exceptions as
the system tried to interpret the random data.
Adding a mutex to all calls to output collectors has fixed this problem for us.
> Kryo errors when using custom serialization
> -------------------------------------------
>
> Key: STORM-3582
> URL: https://issues.apache.org/jira/browse/STORM-3582
> Project: Apache Storm
> Issue Type: Bug
> Components: storm-core
> Affects Versions: 2.1.0
> Reporter: Julien Nioche
> Priority: Blocker
>
> (this has been reported on the user list in January and previously by a
> different user)
>
> ------------------------------------------------------------------------------------
>
> My code works fine with Storm 1.x but the workers crash constantly with Storm
> 2.x
>
> Some exceptions look like
>
> {{_com.esotericsoftware.kryo.KryoException: Buffer underflow._}}
> {{_at com.esotericsoftware.kryo.io.Input.require(Input.java:199)
> ~[kryo-3.0.3.jar:?]_}}
> {{_at com.esotericsoftware.kryo.io.Input.readUtf8_slow(Input.java:575)
> ~[kryo-3.0.3.jar:?]_}}
> {{_at com.esotericsoftware.kryo.io.Input.readUtf8(Input.java:553)
> ~[kryo-3.0.3.jar:?]_}}
> {{_at com.esotericsoftware.kryo.io.Input.readString(Input.java:483)
> ~[kryo-3.0.3.jar:?]_}}
> {{whereas others are}}
>
> {{_2020-01-21 11:13:39.368 o.a.s.m.n.StormServerHandler
> Netty-server-localhost-6701-worker-1 [ERROR] server errors in handling the
> request_}}
> {{_com.esotericsoftware.kryo.KryoException: Encountered unregistered class
> ID: *95*
> at
> com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:137)
> ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670) ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:781)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:153)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:39)
> ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
> ~[kryo-3.0.3.jar:?]
> at
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
> ~[kryo-3.0.3.jar:?]
> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:689)
> ~[kryo-3.0.3.jar:?]
> at
> org.apache.storm.serialization.KryoValuesDeserializer.deserializeFrom(KryoValuesDeserializer.java:31)
> ~[storm-client-2.1.0.jar:2.1.0]_}}
> {{the class ID values are random integers. }}
> {{}}
> The tuples in my topologies contain mostly standard classes like String; the
> only exception is the following class
>
> [https://github.com/DigitalPebble/storm-crawler/blob/2.x/core/src/main/java/com/digitalpebble/stormcrawler/Metadata.java#L41]
>
> for which we specify a custom serialization for Kryo.
>
> My configurations contain
>
> | topology.kryo.register:|- com.digitalpebble.stormcrawler.Metadata|
> to register the custom class.
>
> The user who had reported the problem first also used custom serialization.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
