Hi, I would suggest to avoid Kryo for state serialization, especially if this job is meant for production usage. It might get in the way in the future when you might decide to upgrade your value state schema.
To do that, when declaring the descriptor for your MapState, provide a specific serializer for your value ( *java.util.List[SomeClass[_]]* ). You should be able to use Flink's ListSerializer for this. By providing a specific serializer, this bypasses Flink's type extraction for your state which determines to use the KryoSerializer as a fallback for unrecognizable types. You can find more information about custom state serialization here [1]. Cheers, Gordon [1] https://ci.apache.org/projects/flink/flink-docs-master/dev/stream/state/custom_serialization.html On Wed, Feb 13, 2019 at 2:56 AM Rinat <r.shari...@cleverdata.ru> wrote: > Hi mates ! > > I’ve implemented a job, that stores it’s progress using *MapState[K, V]*, > where *K* - is *java.lang.String*, and *V* - is a collection of some > typed objects *java.util.List[SomeClass[_]]* > When Flink is trying to serialize this state, it is using kryo serializer > for value object and fails with *StackOverflowException* > > *java.lang.StackOverflowError* > * at java.util.HashMap.hash(HashMap.java:338)* > * at java.util.HashMap.get(HashMap.java:556)* > * at com.esotericsoftware.kryo.Generics.getConcreteClass(Generics.java:43)* > > This problem is related with the known bug in *kryo* ( > https://github.com/EsotericSoftware/kryo/issues/341), and reveals itself > only when type of* SomeClass* is a *java.util.BitSet.* > > I’ve checked my job locally (from IDE) with latest (4.0.2 > <https://mvnrepository.com/artifact/com.esotericsoftware/kryo/4.0.2>) > kryo lib, and it works fine, but I couldn’t change kryo version for > distributed mode, because it’s packaged into fat-jar > (flink-dist_2.11-1.6.1.jar), that > contains all runtime dependencies for Flink. > > Maybe you can give me any advices, how to solve this issue, or register a > separate serializers for this case ? > > Thx for your help. > > > Sincerely yours, > *Rinat Sharipov* > Software Engineer at 1DMP CORE Team > > email: r.shari...@cleverdata.ru <a.totma...@cleverdata.ru> > mobile: +7 (925) 416-37-26 > > CleverDATA > make your data clever > >
