Hi mates !
I’ve implemented a job, that stores it’s progress using MapState[K, V], where K
- is java.lang.String, and V - is a collection of some typed objects
java.util.List[SomeClass[_]]
When Flink is trying to serialize this state, it is using kryo serializer for
value object and fails with StackOverflowException
java.lang.StackOverflowError
at java.util.HashMap.hash(HashMap.java:338)
at java.util.HashMap.get(HashMap.java:556)
at com.esotericsoftware.kryo.Generics.getConcreteClass(Generics.java:43)
This problem is related with the known bug in kryo
(https://github.com/EsotericSoftware/kryo/issues/341), and reveals itself only
when type of SomeClass is a java.util.BitSet.
I’ve checked my job locally (from IDE) with latest (4.0.2
<https://mvnrepository.com/artifact/com.esotericsoftware/kryo/4.0.2>) kryo lib,
and it works fine, but I couldn’t change kryo version for distributed mode,
because it’s packaged into fat-jar (flink-dist_2.11-1.6.1.jar), that
contains all runtime dependencies for Flink.
Maybe you can give me any advices, how to solve this issue, or register a
separate serializers for this case ?
Thx for your help.
Sincerely yours,
Rinat Sharipov
Software Engineer at 1DMP CORE Team
email: r.shari...@cleverdata.ru <mailto:a.totma...@cleverdata.ru>
mobile: +7 (925) 416-37-26
CleverDATA
make your data clever
