Thanks, I'll check it out. On Sun, Sep 18, 2022 at 7:33 PM David Anderson <dander...@apache.org> wrote:
> Vishal, > > If you decide you can't live with dropping that state, [1] is a complete > example showing how to migrate from Kryo by using the state processor API. > > David > > [1] > https://www.docs.immerok.cloud/docs/cookbook/migrating-state-away-from-kryo/ > > > On Fri, Sep 16, 2022 at 8:32 AM Vishal Santoshi <vishal.santo...@gmail.com> > wrote: > >> Thank you for the clarification. I thought so to, >> >> Unfortunately my state are generics based and those are definitely not >> treated as a POJO , though it has all the constructs ( no arg constructor, >> getters/setters etc ). I will likely take an at least once hit by >> >> Changing the uid of that specific Operator, and restart with Allow >> non-restored state ... This will ignore state that cannot be restored ( >> for the previous uid ) , construct state for the new uid and not affect >> other operators ( including the kafka consumer operators ). I can live with >> it, I think. >> >> On Fri, Sep 16, 2022 at 2:55 AM Schwalbe Matthias < >> matthias.schwa...@viseca.ch> wrote: >> >>> Hi Vishal, >>> >>> >>> >>> Good news and bad news đ: >>> >>> >>> >>> - Bad: Kryo serializer cannot be used for schema evolution, see [1] >>> - Good: not all is lost here, >>> - If you happen to have state that you cannot afford to lose, you >>> can transcode it by means of the savepoint API [2], >>> - However, this takes quite some effort >>> - In general, if you ever plan to migrate/extend your schemas, >>> choose a data type that supports schema migration [1], >>> - In your case, PoJo types would be the closest to your original >>> implementation >>> - You can disable Kryo in configuration to avoid this situation in >>> the future, by the way, >>> - Kryo serializer is quite slow compared to the other options and I >>> believe it is only there as a (emergency) fallback solution: [3] >>> >>> >>> >>> Feel free to ask for clarification đ >>> >>> >>> >>> Thias >>> >>> >>> >>> >>> >>> >>> >>> [1] >>> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/schema_evolution/#kryo-cannot-be-used-for-schema-evolution >>> >>> [2] >>> https://nightlies.apache.org/flink/flink-docs-master/docs/libs/state_processor_api/ >>> >>> [3] >>> https://flink.apache.org/news/2020/04/15/flink-serialization-tuning-vol-1.html >>> >>> >>> >>> >>> >>> >>> >>> *From:* Vishal Santoshi <vishal.santo...@gmail.com> >>> *Sent:* Friday, September 16, 2022 1:17 AM >>> *To:* user <user@flink.apache.org> >>> *Subject:* Re: A question about restoring state with an additional >>> variable with kryo >>> >>> >>> >>> â *EXTERNAL MESSAGE â **CAUTION: Think Before You Click *â >>> >>> >>> >>> The exception thrown is as follows. I realize that it is trying to read >>> the long value. How do I signal to kryo that it is OK and that he object >>> can have a default value >>> >>> >>> >>> Caused by: java.io.EOFException: No more bytes left. >>> >>> at org.apache.flink.api.java.typeutils.runtime.NoFetchingInput >>> .require(NoFetchingInput.java:80) >>> >>> at com.esotericsoftware.kryo.io.Input.readVarLong(Input.java:690) >>> >>> at com.esotericsoftware.kryo.io.Input.readLong(Input.java:685) >>> >>> at com.esotericsoftware.kryo.serializers. >>> DefaultSerializers$LongSerializer.read(DefaultSerializers.java:133) >>> >>> at com.esotericsoftware.kryo.serializers. >>> DefaultSerializers$LongSerializer.read(DefaultSerializers.java:123) >>> >>> at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730) >>> >>> at com.esotericsoftware.kryo.serializers.ObjectField.read( >>> ObjectField.java:113) >>> >>> at com.esotericsoftware.kryo.serializers.FieldSerializer.read( >>> FieldSerializer.java:528) >>> >>> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) >>> >>> at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer >>> .deserialize(KryoSerializer.java:354) >>> >>> at org.apache.flink.api.common.typeutils.CompositeSerializer >>> .deserialize(CompositeSerializer.java:156) >>> >>> at org.apache.flink.contrib.streaming.state.RocksDBValueState.value( >>> RocksDBValueState.java:89) >>> >>> >>> >>> On Thu, Sep 15, 2022 at 7:10 PM Vishal Santoshi < >>> vishal.santo...@gmail.com> wrote: >>> >>> << How do I make sure that when reconstituting the state, kryo does not >>> complain? It tries to map the previous state to the new definition of Class >>> A and complains that it cannot read the value for `String b`. >>> >>> >>> >>> >> How do I make sure that when reconstituting the state, kryo does not >>> complain? It tries to map the previous state to the new definition of Class >>> A and complains that it cannot read the value for `long b`. >>> >>> >>> >>> Sorry a typo >>> >>> >>> >>> >>> >>> On Thu, Sep 15, 2022 at 7:04 PM Vishal Santoshi < >>> vishal.santo...@gmail.com> wrote: >>> >>> I have state in rocksDB that represents say >>> >>> >>> >>> class A { >>> >>> String a >>> >>> } >>> >>> >>> >>> I now change my class and add another variable >>> >>> >>> >>> >>> Class A { >>> >>> String a; >>> >>> long b = 0; >>> >>> } >>> >>> >>> >>> How do I make sure that when reconstituting the state, kryo does not >>> complain? It tries to map the previous state to the new definition of Class >>> A and complains that it cannot read the value for `String b`. >>> >>> >>> >>> Unfortunately the state is not using POJO serializer. >>> >>> >>> >>> Thanks and Regards. >>> >>> >>> >>> Vishal >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> Diese Nachricht ist ausschliesslich fĂŒr den Adressaten bestimmt und >>> beinhaltet unter UmstĂ€nden vertrauliche Mitteilungen. Da die >>> Vertraulichkeit von e-Mail-Nachrichten nicht gewĂ€hrleistet werden kann, >>> ĂŒbernehmen wir keine Haftung fĂŒr die GewĂ€hrung der Vertraulichkeit und >>> Unversehrtheit dieser Mitteilung. Bei irrtĂŒmlicher Zustellung bitten wir >>> Sie um Benachrichtigung per e-Mail und um Löschung dieser Nachricht sowie >>> eventueller AnhĂ€nge. Jegliche unberechtigte Verwendung oder Verbreitung >>> dieser Informationen ist streng verboten. >>> >>> This message is intended only for the named recipient and may contain >>> confidential or privileged information. As the confidentiality of email >>> communication cannot be guaranteed, we do not accept any responsibility for >>> the confidentiality and the intactness of this message. If you have >>> received it in error, please advise the sender by return e-mail and delete >>> this message and any attachments. Any unauthorised use or dissemination of >>> this information is strictly prohibited. >>> >>
