[RESULT][VOTE] Drop TypeSerializerConfigSnapshot and savepoint support from Flink versions < 1.8.0
Hi everyone, The proposal [1] has been accepted. I've created a ticket to follow the actual implementation: https://issues.apache.org/jira/browse/FLINK-29807 Binding votes in favour: Chesnay Yuan Dawid Gordon Konstantin Piotr Non-binding: Hangxiang Yu there were no votes against. Best, Piotrek [1] https://lists.apache.org/thread/x5d0p08pf2wx47njogsgqct0k5rpfrl4
Re: [VOTE] Drop TypeSerializerConfigSnapshot and savepoint support from Flink versions < 1.8.0
Thanks for the vote. +1 (binding) from my side as well. The proposal has been accepted. I've created a ticket to follow the actual implementation: https://issues.apache.org/jira/browse/FLINK-29807 Binding votes in favour: Chesnay Yuan Dawid Gordon Konstantin Piotr Non-binding: Hangxiang Yu there were no votes against. Thank you for your votes! Best, Piotrek pon., 31 paź 2022 o 11:39 Chesnay Schepler napisał(a): > +1 > > On 28/10/2022 16:57, Piotr Nowojski wrote: > > Hi, > > > > As discussed on the dev mailing list [0] I would like to start a vote to > > drop support of older savepoint formats (for Flink versions older than > > 1.8). You can find the original explanation from the aforementioned dev > > mailing list thread at the bottom of this message. > > > > Draft PR containing the proposed change you can find here: > > https://github.com/apache/flink/pull/21056 > > > > Vote will be open at least until Wednesday, November 2nd 18:00 CET. > > > > Best, > > Piotrek > > > > [0] https://lists.apache.org/thread/v1q28zg5jhxcqrpq67pyv291nznd3n0w > > > > I would like to open a discussion to remove the long deprecated > > (@PublicEvolving) TypeSerializerConfigSnapshot class [1] and the related > > code. > > > > The motivation behind this move is two fold. One reason is that it > > complicates our code base unnecessarily and creates confusion on how to > > actually implement custom serializers. The immediate reason is that I > > wanted to clean up Flink's configuration stack a bit and refactor the > > ExecutionConfig class [2]. This refactor would keep the API compatibility > > of the ExecutionConfig, but it would break savepoint compatibility with > > snapshots written with some of the old serializers, which had > > ExecutionConfig as a field and were serialized in the snapshot. This > issue > > has been resolved by the introduction of TypeSerializerSnapshot in Flink > > 1.7 [3], where serializers are no longer part of the snapshot. > > > > TypeSerializerConfigSnapshot has been deprecated and no longer used by > > built-in serializers since Flink 1.8 [4] and [5]. Users were encouraged > to > > migrate to TypeSerializerSnapshot since then with their own custom > > serializers. That has been plenty of time for the migration. > > > > This proposal would have the following impact for the users: > > 1. we would drop support for recovery from savepoints taken with Flink < > > 1.7.0 for all built in types serializers > > 2. we would drop support for recovery from savepoints taken with Flink < > > 1.8.0 for built in kryo serializers > > 3. we would drop support for recovery from savepoints taken with Flink < > > 1.17 for custom serializers using deprecated TypeSerializerConfigSnapshot > > > > 1. and 2. would have a simple migration path. Users migrating from those > > old savepoints would have to first start his job using a Flink version > from > > the [1.8, 1.16] range, and take a new savepoint that would be compatible > > with Flink 1.17. > > 3. This is a bit more problematic, because users would have to first > > migrate their own custom serializers to use TypeSerializerSnapshot > (using a > > Flink version from the [1.8, 1.16]), take a savepoint, and only then > > migrate to Flink 1.17. However users had already 4 years to migrate, > which > > in my opinion has been plenty of time to do so. > > > > As a side effect, we could also drop support for some of the legacy > > metadata serializers from LegacyStateMetaInfoReaders and potentially > other > > places that we are keeping for the sake of compatibility with old > > savepoints. > > > > [1] > > > https://nightlies.apache.org/flink/flink-docs-master/api/java/org/apache/flink/api/common/typeutils/TypeSerializerConfigSnapshot.html > > [2] https://issues.apache.org/jira/browse/FLINK-29379 > > [3] https://issues.apache.org/jira/browse/FLINK-9377 > > [4] https://issues.apache.org/jira/browse/FLINK-9376 > > [5] https://issues.apache.org/jira/browse/FLINK-11323 > > > >
Re: [VOTE] Drop TypeSerializerConfigSnapshot and savepoint support from Flink versions < 1.8.0
+1 On 28/10/2022 16:57, Piotr Nowojski wrote: Hi, As discussed on the dev mailing list [0] I would like to start a vote to drop support of older savepoint formats (for Flink versions older than 1.8). You can find the original explanation from the aforementioned dev mailing list thread at the bottom of this message. Draft PR containing the proposed change you can find here: https://github.com/apache/flink/pull/21056 Vote will be open at least until Wednesday, November 2nd 18:00 CET. Best, Piotrek [0] https://lists.apache.org/thread/v1q28zg5jhxcqrpq67pyv291nznd3n0w I would like to open a discussion to remove the long deprecated (@PublicEvolving) TypeSerializerConfigSnapshot class [1] and the related code. The motivation behind this move is two fold. One reason is that it complicates our code base unnecessarily and creates confusion on how to actually implement custom serializers. The immediate reason is that I wanted to clean up Flink's configuration stack a bit and refactor the ExecutionConfig class [2]. This refactor would keep the API compatibility of the ExecutionConfig, but it would break savepoint compatibility with snapshots written with some of the old serializers, which had ExecutionConfig as a field and were serialized in the snapshot. This issue has been resolved by the introduction of TypeSerializerSnapshot in Flink 1.7 [3], where serializers are no longer part of the snapshot. TypeSerializerConfigSnapshot has been deprecated and no longer used by built-in serializers since Flink 1.8 [4] and [5]. Users were encouraged to migrate to TypeSerializerSnapshot since then with their own custom serializers. That has been plenty of time for the migration. This proposal would have the following impact for the users: 1. we would drop support for recovery from savepoints taken with Flink < 1.7.0 for all built in types serializers 2. we would drop support for recovery from savepoints taken with Flink < 1.8.0 for built in kryo serializers 3. we would drop support for recovery from savepoints taken with Flink < 1.17 for custom serializers using deprecated TypeSerializerConfigSnapshot 1. and 2. would have a simple migration path. Users migrating from those old savepoints would have to first start his job using a Flink version from the [1.8, 1.16] range, and take a new savepoint that would be compatible with Flink 1.17. 3. This is a bit more problematic, because users would have to first migrate their own custom serializers to use TypeSerializerSnapshot (using a Flink version from the [1.8, 1.16]), take a savepoint, and only then migrate to Flink 1.17. However users had already 4 years to migrate, which in my opinion has been plenty of time to do so. As a side effect, we could also drop support for some of the legacy metadata serializers from LegacyStateMetaInfoReaders and potentially other places that we are keeping for the sake of compatibility with old savepoints. [1] https://nightlies.apache.org/flink/flink-docs-master/api/java/org/apache/flink/api/common/typeutils/TypeSerializerConfigSnapshot.html [2] https://issues.apache.org/jira/browse/FLINK-29379 [3] https://issues.apache.org/jira/browse/FLINK-9377 [4] https://issues.apache.org/jira/browse/FLINK-9376 [5] https://issues.apache.org/jira/browse/FLINK-11323
Re: [VOTE] Drop TypeSerializerConfigSnapshot and savepoint support from Flink versions < 1.8.0
+1 Best Yuan On Mon, Oct 31, 2022 at 5:01 PM Dawid Wysakowicz wrote: > +1 > > On 28/10/2022 16:57, Piotr Nowojski wrote: > > Hi, > > > > As discussed on the dev mailing list [0] I would like to start a vote to > > drop support of older savepoint formats (for Flink versions older than > > 1.8). You can find the original explanation from the aforementioned dev > > mailing list thread at the bottom of this message. > > > > Draft PR containing the proposed change you can find here: > > https://github.com/apache/flink/pull/21056 > > > > Vote will be open at least until Wednesday, November 2nd 18:00 CET. > > > > Best, > > Piotrek > > > > [0] https://lists.apache.org/thread/v1q28zg5jhxcqrpq67pyv291nznd3n0w > > > > I would like to open a discussion to remove the long deprecated > > (@PublicEvolving) TypeSerializerConfigSnapshot class [1] and the related > > code. > > > > The motivation behind this move is two fold. One reason is that it > > complicates our code base unnecessarily and creates confusion on how to > > actually implement custom serializers. The immediate reason is that I > > wanted to clean up Flink's configuration stack a bit and refactor the > > ExecutionConfig class [2]. This refactor would keep the API compatibility > > of the ExecutionConfig, but it would break savepoint compatibility with > > snapshots written with some of the old serializers, which had > > ExecutionConfig as a field and were serialized in the snapshot. This > issue > > has been resolved by the introduction of TypeSerializerSnapshot in Flink > > 1.7 [3], where serializers are no longer part of the snapshot. > > > > TypeSerializerConfigSnapshot has been deprecated and no longer used by > > built-in serializers since Flink 1.8 [4] and [5]. Users were encouraged > to > > migrate to TypeSerializerSnapshot since then with their own custom > > serializers. That has been plenty of time for the migration. > > > > This proposal would have the following impact for the users: > > 1. we would drop support for recovery from savepoints taken with Flink < > > 1.7.0 for all built in types serializers > > 2. we would drop support for recovery from savepoints taken with Flink < > > 1.8.0 for built in kryo serializers > > 3. we would drop support for recovery from savepoints taken with Flink < > > 1.17 for custom serializers using deprecated TypeSerializerConfigSnapshot > > > > 1. and 2. would have a simple migration path. Users migrating from those > > old savepoints would have to first start his job using a Flink version > from > > the [1.8, 1.16] range, and take a new savepoint that would be compatible > > with Flink 1.17. > > 3. This is a bit more problematic, because users would have to first > > migrate their own custom serializers to use TypeSerializerSnapshot > (using a > > Flink version from the [1.8, 1.16]), take a savepoint, and only then > > migrate to Flink 1.17. However users had already 4 years to migrate, > which > > in my opinion has been plenty of time to do so. > > > > As a side effect, we could also drop support for some of the legacy > > metadata serializers from LegacyStateMetaInfoReaders and potentially > other > > places that we are keeping for the sake of compatibility with old > > savepoints. > > > > [1] > > > https://nightlies.apache.org/flink/flink-docs-master/api/java/org/apache/flink/api/common/typeutils/TypeSerializerConfigSnapshot.html > > [2] https://issues.apache.org/jira/browse/FLINK-29379 > > [3] https://issues.apache.org/jira/browse/FLINK-9377 > > [4] https://issues.apache.org/jira/browse/FLINK-9376 > > [5] https://issues.apache.org/jira/browse/FLINK-11323 > > >
Re: [VOTE] Drop TypeSerializerConfigSnapshot and savepoint support from Flink versions < 1.8.0
+1 On 28/10/2022 16:57, Piotr Nowojski wrote: Hi, As discussed on the dev mailing list [0] I would like to start a vote to drop support of older savepoint formats (for Flink versions older than 1.8). You can find the original explanation from the aforementioned dev mailing list thread at the bottom of this message. Draft PR containing the proposed change you can find here: https://github.com/apache/flink/pull/21056 Vote will be open at least until Wednesday, November 2nd 18:00 CET. Best, Piotrek [0] https://lists.apache.org/thread/v1q28zg5jhxcqrpq67pyv291nznd3n0w I would like to open a discussion to remove the long deprecated (@PublicEvolving) TypeSerializerConfigSnapshot class [1] and the related code. The motivation behind this move is two fold. One reason is that it complicates our code base unnecessarily and creates confusion on how to actually implement custom serializers. The immediate reason is that I wanted to clean up Flink's configuration stack a bit and refactor the ExecutionConfig class [2]. This refactor would keep the API compatibility of the ExecutionConfig, but it would break savepoint compatibility with snapshots written with some of the old serializers, which had ExecutionConfig as a field and were serialized in the snapshot. This issue has been resolved by the introduction of TypeSerializerSnapshot in Flink 1.7 [3], where serializers are no longer part of the snapshot. TypeSerializerConfigSnapshot has been deprecated and no longer used by built-in serializers since Flink 1.8 [4] and [5]. Users were encouraged to migrate to TypeSerializerSnapshot since then with their own custom serializers. That has been plenty of time for the migration. This proposal would have the following impact for the users: 1. we would drop support for recovery from savepoints taken with Flink < 1.7.0 for all built in types serializers 2. we would drop support for recovery from savepoints taken with Flink < 1.8.0 for built in kryo serializers 3. we would drop support for recovery from savepoints taken with Flink < 1.17 for custom serializers using deprecated TypeSerializerConfigSnapshot 1. and 2. would have a simple migration path. Users migrating from those old savepoints would have to first start his job using a Flink version from the [1.8, 1.16] range, and take a new savepoint that would be compatible with Flink 1.17. 3. This is a bit more problematic, because users would have to first migrate their own custom serializers to use TypeSerializerSnapshot (using a Flink version from the [1.8, 1.16]), take a savepoint, and only then migrate to Flink 1.17. However users had already 4 years to migrate, which in my opinion has been plenty of time to do so. As a side effect, we could also drop support for some of the legacy metadata serializers from LegacyStateMetaInfoReaders and potentially other places that we are keeping for the sake of compatibility with old savepoints. [1] https://nightlies.apache.org/flink/flink-docs-master/api/java/org/apache/flink/api/common/typeutils/TypeSerializerConfigSnapshot.html [2] https://issues.apache.org/jira/browse/FLINK-29379 [3] https://issues.apache.org/jira/browse/FLINK-9377 [4] https://issues.apache.org/jira/browse/FLINK-9376 [5] https://issues.apache.org/jira/browse/FLINK-11323 OpenPGP_0x31D2DD10BFC15A2D.asc Description: OpenPGP public key OpenPGP_signature Description: OpenPGP digital signature
Re: [VOTE] Drop TypeSerializerConfigSnapshot and savepoint support from Flink versions < 1.8.0
+1 (non-binding) On Sat, Oct 29, 2022 at 2:21 AM Tzu-Li (Gordon) Tai wrote: > +1 > > On Fri, Oct 28, 2022 at 10:21 AM Konstantin Knauf > wrote: > > > +1 (binding) > > > > Am Fr., 28. Okt. 2022 um 16:58 Uhr schrieb Piotr Nowojski < > > pnowoj...@apache.org>: > > > > > Hi, > > > > > > As discussed on the dev mailing list [0] I would like to start a vote > to > > > drop support of older savepoint formats (for Flink versions older than > > > 1.8). You can find the original explanation from the aforementioned dev > > > mailing list thread at the bottom of this message. > > > > > > Draft PR containing the proposed change you can find here: > > > https://github.com/apache/flink/pull/21056 > > > > > > Vote will be open at least until Wednesday, November 2nd 18:00 CET. > > > > > > Best, > > > Piotrek > > > > > > [0] https://lists.apache.org/thread/v1q28zg5jhxcqrpq67pyv291nznd3n0w > > > > > > I would like to open a discussion to remove the long deprecated > > > (@PublicEvolving) TypeSerializerConfigSnapshot class [1] and the > related > > > code. > > > > > > The motivation behind this move is two fold. One reason is that it > > > complicates our code base unnecessarily and creates confusion on how to > > > actually implement custom serializers. The immediate reason is that I > > > wanted to clean up Flink's configuration stack a bit and refactor the > > > ExecutionConfig class [2]. This refactor would keep the API > compatibility > > > of the ExecutionConfig, but it would break savepoint compatibility with > > > snapshots written with some of the old serializers, which had > > > ExecutionConfig as a field and were serialized in the snapshot. This > > issue > > > has been resolved by the introduction of TypeSerializerSnapshot in > Flink > > > 1.7 [3], where serializers are no longer part of the snapshot. > > > > > > TypeSerializerConfigSnapshot has been deprecated and no longer used by > > > built-in serializers since Flink 1.8 [4] and [5]. Users were encouraged > > to > > > migrate to TypeSerializerSnapshot since then with their own custom > > > serializers. That has been plenty of time for the migration. > > > > > > This proposal would have the following impact for the users: > > > 1. we would drop support for recovery from savepoints taken with Flink > < > > > 1.7.0 for all built in types serializers > > > 2. we would drop support for recovery from savepoints taken with Flink > < > > > 1.8.0 for built in kryo serializers > > > 3. we would drop support for recovery from savepoints taken with Flink > < > > > 1.17 for custom serializers using deprecated > TypeSerializerConfigSnapshot > > > > > > 1. and 2. would have a simple migration path. Users migrating from > those > > > old savepoints would have to first start his job using a Flink version > > from > > > the [1.8, 1.16] range, and take a new savepoint that would be > compatible > > > with Flink 1.17. > > > 3. This is a bit more problematic, because users would have to first > > > migrate their own custom serializers to use TypeSerializerSnapshot > > (using a > > > Flink version from the [1.8, 1.16]), take a savepoint, and only then > > > migrate to Flink 1.17. However users had already 4 years to migrate, > > which > > > in my opinion has been plenty of time to do so. > > > > > > As a side effect, we could also drop support for some of the legacy > > > metadata serializers from LegacyStateMetaInfoReaders and potentially > > other > > > places that we are keeping for the sake of compatibility with old > > > savepoints. > > > > > > [1] > > > > > > > > > https://nightlies.apache.org/flink/flink-docs-master/api/java/org/apache/flink/api/common/typeutils/TypeSerializerConfigSnapshot.html > > > [2] https://issues.apache.org/jira/browse/FLINK-29379 > > > [3] https://issues.apache.org/jira/browse/FLINK-9377 > > > [4] https://issues.apache.org/jira/browse/FLINK-9376 > > > [5] https://issues.apache.org/jira/browse/FLINK-11323 > > > > > > > > > -- > > https://twitter.com/snntrable > > https://github.com/knaufk > > > -- Best, Hangxiang.
Re: [VOTE] Drop TypeSerializerConfigSnapshot and savepoint support from Flink versions < 1.8.0
+1 On Fri, Oct 28, 2022 at 10:21 AM Konstantin Knauf wrote: > +1 (binding) > > Am Fr., 28. Okt. 2022 um 16:58 Uhr schrieb Piotr Nowojski < > pnowoj...@apache.org>: > > > Hi, > > > > As discussed on the dev mailing list [0] I would like to start a vote to > > drop support of older savepoint formats (for Flink versions older than > > 1.8). You can find the original explanation from the aforementioned dev > > mailing list thread at the bottom of this message. > > > > Draft PR containing the proposed change you can find here: > > https://github.com/apache/flink/pull/21056 > > > > Vote will be open at least until Wednesday, November 2nd 18:00 CET. > > > > Best, > > Piotrek > > > > [0] https://lists.apache.org/thread/v1q28zg5jhxcqrpq67pyv291nznd3n0w > > > > I would like to open a discussion to remove the long deprecated > > (@PublicEvolving) TypeSerializerConfigSnapshot class [1] and the related > > code. > > > > The motivation behind this move is two fold. One reason is that it > > complicates our code base unnecessarily and creates confusion on how to > > actually implement custom serializers. The immediate reason is that I > > wanted to clean up Flink's configuration stack a bit and refactor the > > ExecutionConfig class [2]. This refactor would keep the API compatibility > > of the ExecutionConfig, but it would break savepoint compatibility with > > snapshots written with some of the old serializers, which had > > ExecutionConfig as a field and were serialized in the snapshot. This > issue > > has been resolved by the introduction of TypeSerializerSnapshot in Flink > > 1.7 [3], where serializers are no longer part of the snapshot. > > > > TypeSerializerConfigSnapshot has been deprecated and no longer used by > > built-in serializers since Flink 1.8 [4] and [5]. Users were encouraged > to > > migrate to TypeSerializerSnapshot since then with their own custom > > serializers. That has been plenty of time for the migration. > > > > This proposal would have the following impact for the users: > > 1. we would drop support for recovery from savepoints taken with Flink < > > 1.7.0 for all built in types serializers > > 2. we would drop support for recovery from savepoints taken with Flink < > > 1.8.0 for built in kryo serializers > > 3. we would drop support for recovery from savepoints taken with Flink < > > 1.17 for custom serializers using deprecated TypeSerializerConfigSnapshot > > > > 1. and 2. would have a simple migration path. Users migrating from those > > old savepoints would have to first start his job using a Flink version > from > > the [1.8, 1.16] range, and take a new savepoint that would be compatible > > with Flink 1.17. > > 3. This is a bit more problematic, because users would have to first > > migrate their own custom serializers to use TypeSerializerSnapshot > (using a > > Flink version from the [1.8, 1.16]), take a savepoint, and only then > > migrate to Flink 1.17. However users had already 4 years to migrate, > which > > in my opinion has been plenty of time to do so. > > > > As a side effect, we could also drop support for some of the legacy > > metadata serializers from LegacyStateMetaInfoReaders and potentially > other > > places that we are keeping for the sake of compatibility with old > > savepoints. > > > > [1] > > > > > https://nightlies.apache.org/flink/flink-docs-master/api/java/org/apache/flink/api/common/typeutils/TypeSerializerConfigSnapshot.html > > [2] https://issues.apache.org/jira/browse/FLINK-29379 > > [3] https://issues.apache.org/jira/browse/FLINK-9377 > > [4] https://issues.apache.org/jira/browse/FLINK-9376 > > [5] https://issues.apache.org/jira/browse/FLINK-11323 > > > > > -- > https://twitter.com/snntrable > https://github.com/knaufk >
Re: [VOTE] Drop TypeSerializerConfigSnapshot and savepoint support from Flink versions < 1.8.0
+1 (binding) Am Fr., 28. Okt. 2022 um 16:58 Uhr schrieb Piotr Nowojski < pnowoj...@apache.org>: > Hi, > > As discussed on the dev mailing list [0] I would like to start a vote to > drop support of older savepoint formats (for Flink versions older than > 1.8). You can find the original explanation from the aforementioned dev > mailing list thread at the bottom of this message. > > Draft PR containing the proposed change you can find here: > https://github.com/apache/flink/pull/21056 > > Vote will be open at least until Wednesday, November 2nd 18:00 CET. > > Best, > Piotrek > > [0] https://lists.apache.org/thread/v1q28zg5jhxcqrpq67pyv291nznd3n0w > > I would like to open a discussion to remove the long deprecated > (@PublicEvolving) TypeSerializerConfigSnapshot class [1] and the related > code. > > The motivation behind this move is two fold. One reason is that it > complicates our code base unnecessarily and creates confusion on how to > actually implement custom serializers. The immediate reason is that I > wanted to clean up Flink's configuration stack a bit and refactor the > ExecutionConfig class [2]. This refactor would keep the API compatibility > of the ExecutionConfig, but it would break savepoint compatibility with > snapshots written with some of the old serializers, which had > ExecutionConfig as a field and were serialized in the snapshot. This issue > has been resolved by the introduction of TypeSerializerSnapshot in Flink > 1.7 [3], where serializers are no longer part of the snapshot. > > TypeSerializerConfigSnapshot has been deprecated and no longer used by > built-in serializers since Flink 1.8 [4] and [5]. Users were encouraged to > migrate to TypeSerializerSnapshot since then with their own custom > serializers. That has been plenty of time for the migration. > > This proposal would have the following impact for the users: > 1. we would drop support for recovery from savepoints taken with Flink < > 1.7.0 for all built in types serializers > 2. we would drop support for recovery from savepoints taken with Flink < > 1.8.0 for built in kryo serializers > 3. we would drop support for recovery from savepoints taken with Flink < > 1.17 for custom serializers using deprecated TypeSerializerConfigSnapshot > > 1. and 2. would have a simple migration path. Users migrating from those > old savepoints would have to first start his job using a Flink version from > the [1.8, 1.16] range, and take a new savepoint that would be compatible > with Flink 1.17. > 3. This is a bit more problematic, because users would have to first > migrate their own custom serializers to use TypeSerializerSnapshot (using a > Flink version from the [1.8, 1.16]), take a savepoint, and only then > migrate to Flink 1.17. However users had already 4 years to migrate, which > in my opinion has been plenty of time to do so. > > As a side effect, we could also drop support for some of the legacy > metadata serializers from LegacyStateMetaInfoReaders and potentially other > places that we are keeping for the sake of compatibility with old > savepoints. > > [1] > > https://nightlies.apache.org/flink/flink-docs-master/api/java/org/apache/flink/api/common/typeutils/TypeSerializerConfigSnapshot.html > [2] https://issues.apache.org/jira/browse/FLINK-29379 > [3] https://issues.apache.org/jira/browse/FLINK-9377 > [4] https://issues.apache.org/jira/browse/FLINK-9376 > [5] https://issues.apache.org/jira/browse/FLINK-11323 > -- https://twitter.com/snntrable https://github.com/knaufk
[VOTE] Drop TypeSerializerConfigSnapshot and savepoint support from Flink versions < 1.8.0
Hi, As discussed on the dev mailing list [0] I would like to start a vote to drop support of older savepoint formats (for Flink versions older than 1.8). You can find the original explanation from the aforementioned dev mailing list thread at the bottom of this message. Draft PR containing the proposed change you can find here: https://github.com/apache/flink/pull/21056 Vote will be open at least until Wednesday, November 2nd 18:00 CET. Best, Piotrek [0] https://lists.apache.org/thread/v1q28zg5jhxcqrpq67pyv291nznd3n0w I would like to open a discussion to remove the long deprecated (@PublicEvolving) TypeSerializerConfigSnapshot class [1] and the related code. The motivation behind this move is two fold. One reason is that it complicates our code base unnecessarily and creates confusion on how to actually implement custom serializers. The immediate reason is that I wanted to clean up Flink's configuration stack a bit and refactor the ExecutionConfig class [2]. This refactor would keep the API compatibility of the ExecutionConfig, but it would break savepoint compatibility with snapshots written with some of the old serializers, which had ExecutionConfig as a field and were serialized in the snapshot. This issue has been resolved by the introduction of TypeSerializerSnapshot in Flink 1.7 [3], where serializers are no longer part of the snapshot. TypeSerializerConfigSnapshot has been deprecated and no longer used by built-in serializers since Flink 1.8 [4] and [5]. Users were encouraged to migrate to TypeSerializerSnapshot since then with their own custom serializers. That has been plenty of time for the migration. This proposal would have the following impact for the users: 1. we would drop support for recovery from savepoints taken with Flink < 1.7.0 for all built in types serializers 2. we would drop support for recovery from savepoints taken with Flink < 1.8.0 for built in kryo serializers 3. we would drop support for recovery from savepoints taken with Flink < 1.17 for custom serializers using deprecated TypeSerializerConfigSnapshot 1. and 2. would have a simple migration path. Users migrating from those old savepoints would have to first start his job using a Flink version from the [1.8, 1.16] range, and take a new savepoint that would be compatible with Flink 1.17. 3. This is a bit more problematic, because users would have to first migrate their own custom serializers to use TypeSerializerSnapshot (using a Flink version from the [1.8, 1.16]), take a savepoint, and only then migrate to Flink 1.17. However users had already 4 years to migrate, which in my opinion has been plenty of time to do so. As a side effect, we could also drop support for some of the legacy metadata serializers from LegacyStateMetaInfoReaders and potentially other places that we are keeping for the sake of compatibility with old savepoints. [1] https://nightlies.apache.org/flink/flink-docs-master/api/java/org/apache/flink/api/common/typeutils/TypeSerializerConfigSnapshot.html [2] https://issues.apache.org/jira/browse/FLINK-29379 [3] https://issues.apache.org/jira/browse/FLINK-9377 [4] https://issues.apache.org/jira/browse/FLINK-9376 [5] https://issues.apache.org/jira/browse/FLINK-11323
