Hi Till, Thanks for the quick reply, I'm using 1.3.2 atm.
Cheers, Niels On Feb 19, 2018 19:10, "Till Rohrmann" <trohrm...@apache.org> wrote: > Hi Niels, > > which version of Flink are you using? Currently, Flink does not support to > upgrade the TypeSerializer itself, if I'm not mistaken. As you've > described, it will try to use the old serializer stored in the checkpoint > stream to restore state. > > I've pulled Gordon into the conversation who can tell you a little bit > more about the current capability and limitations of state evolution. > > Cheers, > Till > > On Mon, Feb 19, 2018 at 4:14 PM, Niels <[hidden email] > <http:///user/SendEmail.jtp?type=node&node=18437&i=0>> wrote: > >> Hi all, >> >> I'm currently trying to use Avro in order to evolve our data present in >> Flink's Managed State. I've extended the TypeSerializer class successfully >> for this purpose, but still have issues using Schema Evolution. >> >> *The problem:* >> When we try to read data (deserialize from savepoint) with a new >> serialiser >> and a new schema, Flink seems to use the old schema of the old serializer >> (written to the savepoint). This results in an old GenericRecord that >> doesn't adhere to the new Avro schema. >> >> *What seems to happen to me is the following* (Say we evolve from dataV1 >> to >> dataV2): >> - State containing dataV1 is serialized with avro schema V1 to a >> check/savepoint. Along with the data, the serializer itself is written. >> - Upon restore, the old serializer is retrieved from the data (therefore >> needs to be on the classpath). Data is restored using this old serializer. >> The new serializer provided is only used for writes. >> >> If this is indeed the case it explains our aforementioned problem. If you >> have any pointers as to whether this is true and what a possible solution >> would be that would be very much appreciated! >> >> Thanks! >> Niels >> >> >> >> -- >> Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4. >> nabble.com/ >> > > > > ------------------------------ > If you reply to this email, your message will be added to the discussion > below: > http://apache-flink-user-mailing-list-archive.2336050. > n4.nabble.com/Managed-State-Custom-Serializer-with-Avro-tp18419p18437.html > To unsubscribe from Managed State Custom Serializer with Avro, click here > <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=18419&code=bmllbHNkZW5pc3NlbkBnbWFpbC5jb218MTg0MTl8MTgzMDMxMDYx> > . > NAML > <http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> >
