> Can't programmers just code up migration tools to the current version
of Kryo or whatever serialization platform you choose?
Well yes, if someone writes a tool to implement a reasonable migration
path than we may be able to upgrade Kryo.
Until that happens we are blocked on upgrading Kryo.
> Versions older than 3.x aren't supposed to compile nor run correctly
under Java 11+.
In fact our Java 17 support is currently blocked by an issue that we
suspect is related to Kryo.
https://issues.apache.org/jira/browse/FLINK-24998
> I'd presume you would make a tool to upgrade files with Kryo
persisted state in savepoints and checkpoints
That doesn't cover everything and may not necessarily be a viable approach.
Kryo is exposed a fair bit in our APIs (mistakes of the past...) so
users that have custom Serializers might also have to change things.
Upgrading Kryo is thus also an API breaking change.
As for viability, such an approach implies taking down the application
on one Flink version, converting the state, and restarting the job on a
newer Flink version.
This implies a certain amount of downtime for the application, which
depending on the state size may just not be acceptable to a user.
Having to migrate the savepoint and upgrading Flink at the same time is
also not ideal since it makes the effort more complicated; being able to
run the job on the same Flink version with a different Kryo version
would make things easier, but that'd mean we have to be able to run 2
Kryo versions in parallel.
Something else to consider is when we already break everything to
upgrade Kryo, then maybe things should be re-written such that upgrading
Kryo isn't such a problem in the future; in essence reworking how Kryo
is integrated into Flink.
That said, the v5 migration guide is quite interesting; specifically
that Kryo offers a versioned jar.
On 09/02/2023 17:32, Clayton Wohl wrote:
What do you mean you are blocked? Can't programmers just code up
migration tools to the current version of Kryo or whatever
serialization platform you choose?
Can't you follow the Kryo migration guide that supports loading data
serialized with Kryo v2 and reserializing with Kryo v5?
https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5
I'd presume you would make a tool to upgrade files with Kryo persisted
state in savepoints and checkpoints, that would allow for users to
register custom serializers. I also presume that new versions of Flink
would politely refuse to start with old format state files and require
the migration process to be completed.
Kryo v2 also pulls in objenesis v2.1 from 2013, before Java 8.
Versions older than 3.x aren't supposed to compile nor run correctly
under Java 11+.
On Thu, Feb 9, 2023 at 2:34 AM Chesnay Schepler <ches...@apache.org>
wrote:
> you can't reasonably stay on the 2015 version forever, refuse to
adopt any of the updates or fixes in the 8 years since then, and
reasonably expect things to continue to work well.
We are well aware that Kryo is a ticking time bomb.
> Is there any possibility a future release of Flink can upgrade
to a
recent version of Kryo serialization?
Of course there is, but someone needs to figure out a way to do this
without breaking everything or providing a reasonable upgrade path,
which has been blocking us so far.
On 09/02/2023 07:34, Clayton Wohl wrote:
> I've noticed the latest Flink is using the Kryo serializer library
> version 2.24.0 which is back from 2015!
>
> The Kryo project is actively maintained, it's on version 5.4.0, so
> 2.24.0 is really quite ancient. I presume the concern is
maintaining
> compatibility with persisted savepoints. That's a valid concern,
but
> you can't reasonably stay on the 2015 version forever, refuse to
adopt
> any of the updates or fixes in the 8 years since then, and
reasonably
> expect things to continue to work well.
>
> Is there any possibility a future release of Flink can upgrade to a
> recent version of Kryo serialization?
>
>
>
