Could we try using schemas? On Thu, Jun 4, 2020 at 9:40 AM Maximilian Michels <m...@apache.org> wrote:
> I was under the assumption that this should work but ProtoCoder keeps a > reference of the class used to serialize. That causes the snapshot to > break. > > We can fix this by: > > a) writing/using coders which do not keep instances of evolving classes > b) adding an interface to Beam for Coder serialization/deserialization > c) adding a mode to Flink which allows to use newly supplied coders > instead of having to load the old coder > > From all the options (a) is the most feasible for you. It looks like > neither ProtoCoder nor AvroCoder fall into this category. > > -Max > > On 04.06.20 16:22, Ivan San Jose wrote: > > I've changed my Java model in order to use ProtoCoder (with > > @DefaultCoder(ProtoCoder.class)), but I'm getting same error when > > tryingto restore the last taken checkpoint after adding an attribute to > > that model. > > > > What do you think it could happen? It seems that state saved within the > > checkpoint is still using Java serialization mechanism... > > > > On Thu, 2020-06-04 at 13:05 +0000, Ivan San Jose wrote: > >> Thanks Max for your response. I'd try with AvroCoder then. But I > >> still > >> have a question, I guess AvroCoder is generating the AVRO schema > >> using > >> Java reflection, and then that generated schema is saved within the > >> Flink checkpoint, right? > >> > >> On Wed, 2020-06-03 at 18:00 +0200, Maximilian Michels wrote: > >>> Hi Ivan, > >>> > >>> Moving to the new type serializer snapshot interface is not going > >>> to > >>> solve this problem because we cannot version the coder through the > >>> Beam > >>> coder interface. That is only possible through Flink. However, it > >>> is > >>> usually not trivial. > >>> > >>> In Beam, when you evolve your data model, the only way you can > >>> maintain > >>> compatible is to use a serialization format which can evolve, e.g. > >>> KafkaIO or Protobuf. > >>> > >>> Cheers, > >>> Max > >>> > >>> On 03.06.20 16:47, Ivan San Jose wrote: > >>>> Hi, we have a Beam application running with Flink runner and we > >>>> are > >>>> struggling using Flink checkpoints. Everytime we evolve the > >>>> source > >>>> code > >>>> modifying a Java model, an exception is thrown when trying to > >>>> restore > >>>> last checkpoint taken: > >>>> > >>>> Caused by: java.lang.IllegalStateException: Could not Java- > >>>> deserialize > >>>> TypeSerializer while restoring checkpoint metadata for serializer > >>>> snapshot > >>>> 'org.apache.beam.runners.flink.translation.types.CoderTypeSeriali > >>>> ze > >>>> r$Le > >>>> gacySnapshot'. Please update to the TypeSerializerSnapshot > >>>> interface > >>>> that removes Java Serialization to avoid this problem in the > >>>> future. > >>>> at > >>>> org.apache.flink.api.common.typeutils.TypeSerializerConfigSnapsho > >>>> t. > >>>> rest > >>>> oreSerializer(TypeSerializerConfigSnapshot.java:138) > >>>> at > >>>> org.apache.flink.runtime.state.StateSerializerProvider.previousSc > >>>> he > >>>> maSe > >>>> rializer(StateSerializerProvider.java:189) > >>>> at > >>>> org.apache.flink.runtime.state.StateSerializerProvider.currentSch > >>>> em > >>>> aSer > >>>> ializer(StateSerializerProvider.java:164) > >>>> at > >>>> org.apache.flink.runtime.state.RegisteredOperatorStateBackendMeta > >>>> In > >>>> fo.g > >>>> etPartitionStateSerializer(RegisteredOperatorStateBackendMetaInfo > >>>> .j > >>>> ava: > >>>> 113) > >>>> at > >>>> org.apache.flink.runtime.state.OperatorStateRestoreOperation.rest > >>>> or > >>>> e(Op > >>>> eratorStateRestoreOperation.java:94) > >>>> at > >>>> org.apache.flink.runtime.state.DefaultOperatorStateBackendBuilder > >>>> .b > >>>> uild > >>>> (DefaultOperatorStateBackendBuilder.java:83) > >>>> ... 12 more > >>>> Caused by: java.io.InvalidClassException: > >>>> internal.model.dimension.Dimension; local class incompatible: > >>>> stream > >>>> classdesc serialVersionUID = -223148029368332375, local class > >>>> serialVersionUID = 4489864664852536553 > >>>> > >>>> As you can see the exception is complaining about class was > >>>> evolved > >>>> and > >>>> they are not compatible any more. > >>>> > >>>> After checking some documentation and Beam source code... > >>>> > https://flink.apache.org/ecosystem/2020/02/22/apache-beam-how-beam-runs-on-top-of-flink.html > >>>> (Serializers > >>>> vs > >>>> Coders) > >>>> > https://github.com/apache/beam/blob/785609f22d013411b7973bbf9e2d15c3c8171fb2/runners/flink/1.8/src/main/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializer.java > >>>> > >>>> It seems that Beam coders are wrapped into Flink's > >>>> TypeSerializers, > >>>> and, at the end, Beam coders are the ones in charge of > >>>> serialize/deserialize objects. > >>>> > >>>> Also reading > >>>> > https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/stream/state/custom_serialization.html#migrating-from-deprecated-serializer-snapshot-apis-before-flink-17 > >>>> , and seeing CoderTyperSerializer implementation it seems Beam is > >>>> implementing a deprecated interface and should be moved to new > >>>> TypeSerializerSnapshot one. To be honest I don't know if that > >>>> would > >>>> solve my problem, but the exception is clearly saying "Please > >>>> update to > >>>> the TypeSerializerSnapshot interface that removes Java > >>>> Serialization to > >>>> avoid this problem in the future.", so I gues would solve it. > >>>> > >>>> Can someone help me here? I could try to update the > >>>> implementation > >>>> if > >>>> someone give me some hints, because right now I'm a little bit > >>>> lost. > >>>> > >>>> Thanks > >>>> > >>>> > >>>> Este correo electrónico y sus adjuntos son de naturaleza > >>>> confidencial. A no ser que usted sea el destinatario, no puede > >>>> utilizar, copiar o desvelar tanto el mensaje como cualquier > >>>> información contenida en el mensaje. Si no es el destinatario, > >>>> debe > >>>> borrar este correo y notificar al remitente inmediatamente. > >>>> Cualquier punto de vista u opinión expresada en este correo > >>>> electrónico son únicamente del remitente, a no ser que se indique > >>>> lo contrario. Todos los derechos de autor en cualquier material > >>>> de > >>>> este correo son reservados. Todos los correos electrónicos, > >>>> salientes o entrantes, pueden ser grabados y monitorizados para > >>>> uso > >>>> legítimo del negocio. Nos encontramos exentos de toda > >>>> responsabilidad ante cualquier perdida o daño que surja o resulte > >>>> de la recepción, uso o transmisión de este correo electrónico > >>>> hasta > >>>> el máximo permitido por la ley. > >>>> > >>>> This email and any attachment to it are confidential. Unless you > >>>> are the intended recipient, you may not use, copy or disclose > >>>> either the message or any information contained in the message. > >>>> If > >>>> you are not the intended recipient, you should delete this email > >>>> and notify the sender immediately. Any views or opinions > >>>> expressed > >>>> in this email are those of the sender only, unless otherwise > >>>> stated. All copyright in any of the material in this email is > >>>> reserved. All emails, incoming and outgoing, may be recorded and > >>>> monitored for legitimate business purposes. We exclude all > >>>> liability for any loss or damage arising or resulting from the > >>>> receipt, use or transmission of this email to the fullest extent > >>>> permitted by law. > >>>> > >>> > >>> -- > >>> This message has been scanned and is believed to be clean. > >>> > >> > >> Este correo electrónico y sus adjuntos son de naturaleza > >> confidencial. A no ser que usted sea el destinatario, no puede > >> utilizar, copiar o desvelar tanto el mensaje como cualquier > >> información contenida en el mensaje. Si no es el destinatario, debe > >> borrar este correo y notificar al remitente inmediatamente. Cualquier > >> punto de vista u opinión expresada en este correo electrónico son > >> únicamente del remitente, a no ser que se indique lo contrario. Todos > >> los derechos de autor en cualquier material de este correo son > >> reservados. Todos los correos electrónicos, salientes o entrantes, > >> pueden ser grabados y monitorizados para uso legítimo del negocio. > >> Nos encontramos exentos de toda responsabilidad ante cualquier > >> perdida o daño que surja o resulte de la recepción, uso o transmisión > >> de este correo electrónico hasta el máximo permitido por la ley. > >> > >> This email and any attachment to it are confidential. Unless you are > >> the intended recipient, you may not use, copy or disclose either the > >> message or any information contained in the message. If you are not > >> the intended recipient, you should delete this email and notify the > >> sender immediately. Any views or opinions expressed in this email are > >> those of the sender only, unless otherwise stated. All copyright in > >> any of the material in this email is reserved. All emails, incoming > >> and outgoing, may be recorded and monitored for legitimate business > >> purposes. We exclude all liability for any loss or damage arising or > >> resulting from the receipt, use or transmission of this email to the > >> fullest extent permitted by law. > > > > > > Este correo electrónico y sus adjuntos son de naturaleza confidencial. A > no ser que usted sea el destinatario, no puede utilizar, copiar o desvelar > tanto el mensaje como cualquier información contenida en el mensaje. Si no > es el destinatario, debe borrar este correo y notificar al remitente > inmediatamente. Cualquier punto de vista u opinión expresada en este correo > electrónico son únicamente del remitente, a no ser que se indique lo > contrario. Todos los derechos de autor en cualquier material de este correo > son reservados. Todos los correos electrónicos, salientes o entrantes, > pueden ser grabados y monitorizados para uso legítimo del negocio. Nos > encontramos exentos de toda responsabilidad ante cualquier perdida o daño > que surja o resulte de la recepción, uso o transmisión de este correo > electrónico hasta el máximo permitido por la ley. > > > > This email and any attachment to it are confidential. Unless you are the > intended recipient, you may not use, copy or disclose either the message or > any information contained in the message. If you are not the intended > recipient, you should delete this email and notify the sender immediately. > Any views or opinions expressed in this email are those of the sender only, > unless otherwise stated. All copyright in any of the material in this email > is reserved. All emails, incoming and outgoing, may be recorded and > monitored for legitimate business purposes. We exclude all liability for > any loss or damage arising or resulting from the receipt, use or > transmission of this email to the fullest extent permitted by law. > > >
