Thank you for the clarification. I thought so to, Unfortunately my state are generics based and those are definitely not treated as a POJO , though it has all the constructs ( no arg constructor, getters/setters etc ). I will likely take an at least once hit by
Changing the uid of that specific Operator, and restart with Allow non-restored state ... This will ignore state that cannot be restored ( for the previous uid ) , construct state for the new uid and not affect other operators ( including the kafka consumer operators ). I can live with it, I think. On Fri, Sep 16, 2022 at 2:55 AM Schwalbe Matthias < matthias.schwa...@viseca.ch> wrote: > Hi Vishal, > > > > Good news and bad news đ: > > > > - Bad: Kryo serializer cannot be used for schema evolution, see [1] > - Good: not all is lost here, > - If you happen to have state that you cannot afford to lose, you > can transcode it by means of the savepoint API [2], > - However, this takes quite some effort > - In general, if you ever plan to migrate/extend your schemas, choose > a data type that supports schema migration [1], > - In your case, PoJo types would be the closest to your original > implementation > - You can disable Kryo in configuration to avoid this situation in the > future, by the way, > - Kryo serializer is quite slow compared to the other options and I > believe it is only there as a (emergency) fallback solution: [3] > > > > Feel free to ask for clarification đ > > > > Thias > > > > > > > > [1] > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/schema_evolution/#kryo-cannot-be-used-for-schema-evolution > > [2] > https://nightlies.apache.org/flink/flink-docs-master/docs/libs/state_processor_api/ > > [3] > https://flink.apache.org/news/2020/04/15/flink-serialization-tuning-vol-1.html > > > > > > > > *From:* Vishal Santoshi <vishal.santo...@gmail.com> > *Sent:* Friday, September 16, 2022 1:17 AM > *To:* user <user@flink.apache.org> > *Subject:* Re: A question about restoring state with an additional > variable with kryo > > > > â *EXTERNAL MESSAGE â **CAUTION: Think Before You Click *â > > > > The exception thrown is as follows. I realize that it is trying to read > the long value. How do I signal to kryo that it is OK and that he object > can have a default value > > > > Caused by: java.io.EOFException: No more bytes left. > > at org.apache.flink.api.java.typeutils.runtime.NoFetchingInput > .require(NoFetchingInput.java:80) > > at com.esotericsoftware.kryo.io.Input.readVarLong(Input.java:690) > > at com.esotericsoftware.kryo.io.Input.readLong(Input.java:685) > > at com.esotericsoftware.kryo.serializers. > DefaultSerializers$LongSerializer.read(DefaultSerializers.java:133) > > at com.esotericsoftware.kryo.serializers. > DefaultSerializers$LongSerializer.read(DefaultSerializers.java:123) > > at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730) > > at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField > .java:113) > > at com.esotericsoftware.kryo.serializers.FieldSerializer.read( > FieldSerializer.java:528) > > at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) > > at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer > .deserialize(KryoSerializer.java:354) > > at org.apache.flink.api.common.typeutils.CompositeSerializer > .deserialize(CompositeSerializer.java:156) > > at org.apache.flink.contrib.streaming.state.RocksDBValueState.value( > RocksDBValueState.java:89) > > > > On Thu, Sep 15, 2022 at 7:10 PM Vishal Santoshi <vishal.santo...@gmail.com> > wrote: > > << How do I make sure that when reconstituting the state, kryo does not > complain? It tries to map the previous state to the new definition of Class > A and complains that it cannot read the value for `String b`. > > > > >> How do I make sure that when reconstituting the state, kryo does not > complain? It tries to map the previous state to the new definition of Class > A and complains that it cannot read the value for `long b`. > > > > Sorry a typo > > > > > > On Thu, Sep 15, 2022 at 7:04 PM Vishal Santoshi <vishal.santo...@gmail.com> > wrote: > > I have state in rocksDB that represents say > > > > class A { > > String a > > } > > > > I now change my class and add another variable > > > > > Class A { > > String a; > > long b = 0; > > } > > > > How do I make sure that when reconstituting the state, kryo does not > complain? It tries to map the previous state to the new definition of Class > A and complains that it cannot read the value for `String b`. > > > > Unfortunately the state is not using POJO serializer. > > > > Thanks and Regards. > > > > Vishal > > > > > > > > > > Diese Nachricht ist ausschliesslich fĂŒr den Adressaten bestimmt und > beinhaltet unter UmstĂ€nden vertrauliche Mitteilungen. Da die > Vertraulichkeit von e-Mail-Nachrichten nicht gewĂ€hrleistet werden kann, > ĂŒbernehmen wir keine Haftung fĂŒr die GewĂ€hrung der Vertraulichkeit und > Unversehrtheit dieser Mitteilung. Bei irrtĂŒmlicher Zustellung bitten wir > Sie um Benachrichtigung per e-Mail und um Löschung dieser Nachricht sowie > eventueller AnhĂ€nge. Jegliche unberechtigte Verwendung oder Verbreitung > dieser Informationen ist streng verboten. > > This message is intended only for the named recipient and may contain > confidential or privileged information. As the confidentiality of email > communication cannot be guaranteed, we do not accept any responsibility for > the confidentiality and the intactness of this message. If you have > received it in error, please advise the sender by return e-mail and delete > this message and any attachments. Any unauthorised use or dissemination of > this information is strictly prohibited. >
