Re: A question about Kryo and Window State
Actually, yes. I have a job already running with "FieldSerializer" in production. Any insights will be appreciated. On Sat, Jun 23, 2018 at 7:39 AM, Vishal Santoshi wrote: > Thanks. > > On Thu, Jun 21, 2018 at 4:34 AM, Tzu-Li (Gordon) Tai > wrote: > >> Hi Vishal, >> >> Kryo has a serializer called `CompatibleFieldSerializer` that allows for >> simple backward compatibility changes, such as adding non-optional fields / >> removing fields. >> >> If using the KryoSerializer is a must, then a good thing to do is to >> register Kryo's `CompatibleFieldSerializer` as the serializer to be used >> for that specific type. >> By default, Kryo doesn’t use the `CompatibleFieldSerializer`, so you >> would have to explicitly register this. >> >> The issue is, I think it wouldn’t be possible to use the >> `CompatibleFieldSerializer` _after_ the bytes were already written with the >> default, non-compatible version (the `FieldSerializer`). >> So, AFAIK, this should only work if your Kryo state type was registered >> with the `CompatibleFieldSerializer` from the very beginning. >> There could be a workaround, but unfortunately that would require >> changing some code in the `KryoSerializer`. >> If you require this because your job is already running in production and >> data was already written by the `FieldSerializer`, please let me know and >> I’ll go into more details about this. >> >> Cheers, >> Gordon >> >> On 21 June 2018 at 10:14:15 AM, Fabian Hueske (fhue...@gmail.com) wrote: >> >> Hi Vishal, >> >> In general, Kryo serializers are not very upgrade friendly. >> Serializer compatibility [1] might be right approach here, but Gordon (in >> CC) might know more about this. >> >> Best, Fabian >> >> [1] https://ci.apache.org/projects/flink/flink-docs-release-1.5/ >> dev/stream/state/custom_serialization.html#handling- >> serializer-upgrades-and-compatibility >> >> 2018-06-18 12:06 GMT+02:00 Vishal Santoshi : >> >>> Any more insight? >>> >>> On Wed, Jun 13, 2018, 3:34 PM Vishal Santoshi >>> wrote: >>> Any ideas on the standard way ( or any roundabout way ) of doing a version upgrade that looks back ward compatible. The @FieldSerializer.Optional("0") actually does ignore the field ( even if reset ) giving it the default value if kyro is used. It has to do with the FieldSerializer behaves . There is another Serializer ( Composite I believe ) that allows for such back ward compatible changes. I know some work is being done in 1.6 to allow for above use case and I think Google Data Flow does provide some avenues. Thanks much Vishal On Tue, Jun 12, 2018 at 11:30 PM, Vishal Santoshi < vishal.santo...@gmail.com> wrote: > I have a running pipe with Window State in a class say > > Class A{ > long a; > } > > It uses the default KryoSerializer > > I want to add a field to > > Class A { > long a; > long b; > } > > I need to suspend with SP and resume with the new version of Class A > > > Is there a definite way to do this. I tried > > Class A { > long a; >@FieldSerializer.Optional("0") > long b; > } > > but that seems to default to 0 , even when the Aggregation is putting > in values. > > Could somebody give pointers as to how to solve this > > Thanks a ton. > > > > >> >
Re: A question about Kryo and Window State
Thanks. On Thu, Jun 21, 2018 at 4:34 AM, Tzu-Li (Gordon) Tai wrote: > Hi Vishal, > > Kryo has a serializer called `CompatibleFieldSerializer` that allows for > simple backward compatibility changes, such as adding non-optional fields / > removing fields. > > If using the KryoSerializer is a must, then a good thing to do is to > register Kryo's `CompatibleFieldSerializer` as the serializer to be used > for that specific type. > By default, Kryo doesn’t use the `CompatibleFieldSerializer`, so you would > have to explicitly register this. > > The issue is, I think it wouldn’t be possible to use the > `CompatibleFieldSerializer` _after_ the bytes were already written with the > default, non-compatible version (the `FieldSerializer`). > So, AFAIK, this should only work if your Kryo state type was registered > with the `CompatibleFieldSerializer` from the very beginning. > There could be a workaround, but unfortunately that would require changing > some code in the `KryoSerializer`. > If you require this because your job is already running in production and > data was already written by the `FieldSerializer`, please let me know and > I’ll go into more details about this. > > Cheers, > Gordon > > On 21 June 2018 at 10:14:15 AM, Fabian Hueske (fhue...@gmail.com) wrote: > > Hi Vishal, > > In general, Kryo serializers are not very upgrade friendly. > Serializer compatibility [1] might be right approach here, but Gordon (in > CC) might know more about this. > > Best, Fabian > > [1] https://ci.apache.org/projects/flink/flink-docs- > release-1.5/dev/stream/state/custom_serialization.html# > handling-serializer-upgrades-and-compatibility > > 2018-06-18 12:06 GMT+02:00 Vishal Santoshi : > >> Any more insight? >> >> On Wed, Jun 13, 2018, 3:34 PM Vishal Santoshi >> wrote: >> >>> Any ideas on the standard way ( or any roundabout way ) of doing a >>> version upgrade that looks back ward compatible. >>> The @FieldSerializer.Optional("0") actually does ignore the field ( >>> even if reset ) giving it the default value if kyro is used. It has to do >>> with the FieldSerializer behaves . There is another Serializer ( >>> Composite I believe ) that allows for such back ward compatible changes. >>> >>> >>> I know some work is being done in 1.6 to allow for above use case and I >>> think Google Data Flow does provide some avenues. >>> >>> Thanks much >>> >>> Vishal >>> >>> >>> >>> On Tue, Jun 12, 2018 at 11:30 PM, Vishal Santoshi < >>> vishal.santo...@gmail.com> wrote: >>> I have a running pipe with Window State in a class say Class A{ long a; } It uses the default KryoSerializer I want to add a field to Class A { long a; long b; } I need to suspend with SP and resume with the new version of Class A Is there a definite way to do this. I tried Class A { long a; @FieldSerializer.Optional("0") long b; } but that seems to default to 0 , even when the Aggregation is putting in values. Could somebody give pointers as to how to solve this Thanks a ton. >>> >
Re: A question about Kryo and Window State
Hi Vishal, Kryo has a serializer called `CompatibleFieldSerializer` that allows for simple backward compatibility changes, such as adding non-optional fields / removing fields. If using the KryoSerializer is a must, then a good thing to do is to register Kryo's `CompatibleFieldSerializer` as the serializer to be used for that specific type. By default, Kryo doesn’t use the `CompatibleFieldSerializer`, so you would have to explicitly register this. The issue is, I think it wouldn’t be possible to use the `CompatibleFieldSerializer` _after_ the bytes were already written with the default, non-compatible version (the `FieldSerializer`). So, AFAIK, this should only work if your Kryo state type was registered with the `CompatibleFieldSerializer` from the very beginning. There could be a workaround, but unfortunately that would require changing some code in the `KryoSerializer`. If you require this because your job is already running in production and data was already written by the `FieldSerializer`, please let me know and I’ll go into more details about this. Cheers, Gordon On 21 June 2018 at 10:14:15 AM, Fabian Hueske (fhue...@gmail.com) wrote: Hi Vishal, In general, Kryo serializers are not very upgrade friendly. Serializer compatibility [1] might be right approach here, but Gordon (in CC) might know more about this. Best, Fabian [1] https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/stream/state/custom_serialization.html#handling-serializer-upgrades-and-compatibility 2018-06-18 12:06 GMT+02:00 Vishal Santoshi : Any more insight? On Wed, Jun 13, 2018, 3:34 PM Vishal Santoshi wrote: Any ideas on the standard way ( or any roundabout way ) of doing a version upgrade that looks back ward compatible. The @FieldSerializer.Optional("0") actually does ignore the field ( even if reset ) giving it the default value if kyro is used. It has to do with the FieldSerializer behaves . There is another Serializer ( Composite I believe ) that allows for such back ward compatible changes. I know some work is being done in 1.6 to allow for above use case and I think Google Data Flow does provide some avenues. Thanks much Vishal On Tue, Jun 12, 2018 at 11:30 PM, Vishal Santoshi wrote: I have a running pipe with Window State in a class say Class A{ long a; } It uses the default KryoSerializer I want to add a field to Class A { long a; long b; } I need to suspend with SP and resume with the new version of Class A Is there a definite way to do this. I tried Class A { long a; @FieldSerializer.Optional("0") long b; } but that seems to default to 0 , even when the Aggregation is putting in values. Could somebody give pointers as to how to solve this Thanks a ton.
Re: A question about Kryo and Window State
Hi Vishal, In general, Kryo serializers are not very upgrade friendly. Serializer compatibility [1] might be right approach here, but Gordon (in CC) might know more about this. Best, Fabian [1] https://ci.apache.org/projects/flink/flink-docs-release-1.5/dev/stream/state/custom_serialization.html#handling-serializer-upgrades-and-compatibility 2018-06-18 12:06 GMT+02:00 Vishal Santoshi : > Any more insight? > > On Wed, Jun 13, 2018, 3:34 PM Vishal Santoshi > wrote: > >> Any ideas on the standard way ( or any roundabout way ) of doing a >> version upgrade that looks back ward compatible. >> The @FieldSerializer.Optional("0") actually does ignore the field ( >> even if reset ) giving it the default value if kyro is used. It has to do >> with the FieldSerializer behaves . There is another Serializer ( >> Composite I believe ) that allows for such back ward compatible changes. >> >> >> I know some work is being done in 1.6 to allow for above use case and I >> think Google Data Flow does provide some avenues. >> >> Thanks much >> >> Vishal >> >> >> >> On Tue, Jun 12, 2018 at 11:30 PM, Vishal Santoshi < >> vishal.santo...@gmail.com> wrote: >> >>> I have a running pipe with Window State in a class say >>> >>> Class A{ >>> long a; >>> } >>> >>> It uses the default KryoSerializer >>> >>> I want to add a field to >>> >>> Class A { >>> long a; >>> long b; >>> } >>> >>> I need to suspend with SP and resume with the new version of Class A >>> >>> >>> Is there a definite way to do this. I tried >>> >>> Class A { >>> long a; >>>@FieldSerializer.Optional("0") >>> long b; >>> } >>> >>> but that seems to default to 0 , even when the Aggregation is putting in >>> values. >>> >>> Could somebody give pointers as to how to solve this >>> >>> Thanks a ton. >>> >>> >>> >>> >>
Re: A question about Kryo and Window State
Any more insight? On Wed, Jun 13, 2018, 3:34 PM Vishal Santoshi wrote: > Any ideas on the standard way ( or any roundabout way ) of doing a version > upgrade that looks back ward compatible. > The @FieldSerializer.Optional("0") actually does ignore the field ( > even if reset ) giving it the default value if kyro is used. It has to do > with the FieldSerializer behaves . There is another Serializer ( > Composite I believe ) that allows for such back ward compatible changes. > > > I know some work is being done in 1.6 to allow for above use case and I > think Google Data Flow does provide some avenues. > > Thanks much > > Vishal > > > > On Tue, Jun 12, 2018 at 11:30 PM, Vishal Santoshi < > vishal.santo...@gmail.com> wrote: > >> I have a running pipe with Window State in a class say >> >> Class A{ >> long a; >> } >> >> It uses the default KryoSerializer >> >> I want to add a field to >> >> Class A { >> long a; >> long b; >> } >> >> I need to suspend with SP and resume with the new version of Class A >> >> >> Is there a definite way to do this. I tried >> >> Class A { >> long a; >>@FieldSerializer.Optional("0") >> long b; >> } >> >> but that seems to default to 0 , even when the Aggregation is putting in >> values. >> >> Could somebody give pointers as to how to solve this >> >> Thanks a ton. >> >> >> >> >
Re: A question about Kryo and Window State
Any ideas on the standard way ( or any roundabout way ) of doing a version upgrade that looks back ward compatible. The @FieldSerializer.Optional("0") actually does ignore the field ( even if reset ) giving it the default value if kyro is used. It has to do with the FieldSerializer behaves . There is another Serializer ( Composite I believe ) that allows for such back ward compatible changes. I know some work is being done in 1.6 to allow for above use case and I think Google Data Flow does provide some avenues. Thanks much Vishal On Tue, Jun 12, 2018 at 11:30 PM, Vishal Santoshi wrote: > I have a running pipe with Window State in a class say > > Class A{ > long a; > } > > It uses the default KryoSerializer > > I want to add a field to > > Class A { > long a; > long b; > } > > I need to suspend with SP and resume with the new version of Class A > > > Is there a definite way to do this. I tried > > Class A { > long a; >@FieldSerializer.Optional("0") > long b; > } > > but that seems to default to 0 , even when the Aggregation is putting in > values. > > Could somebody give pointers as to how to solve this > > Thanks a ton. > > > >
A question about Kryo and Window State
I have a running pipe with Window State in a class say Class A{ long a; } It uses the default KryoSerializer I want to add a field to Class A { long a; long b; } I need to suspend with SP and resume with the new version of Class A Is there a definite way to do this. I tried Class A { long a; @FieldSerializer.Optional("0") long b; } but that seems to default to 0 , even when the Aggregation is putting in values. Could somebody give pointers as to how to solve this Thanks a ton.