https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5#migration-guide
Kroy themselves state that v5 likely can't read v2 data.
However, both versions can be on the classpath without classpath as v5
offers a versioned artifact that includes the version in the package.
It probably wouldn't be difficult to migrate a savepoint to Kryo v5,
purely from a read/write perspective.
The bigger question is how we expose this new Kryo version in the API.
If we stick to the versioned jar we need to either duplicate all current
Kryo-related APIs or find a better way to integrate other serialization
stacks.
On 30/03/2023 17:50, Piotr Nowojski wrote:
Hey,
> 1. The Flink community agrees that we upgrade Kryo to a later
version, which means breaking all checkpoint/savepoint compatibility
and releasing a Flink 2.0 with Java 17 support added and Java 8 and
Flink Scala API support dropped. This is probably the quickest way,
but would still mean that we expose Kryo in the Flink APIs, which is
the main reason why we haven't been able to upgrade Kryo at all.
This sounds pretty bad to me.
Has anyone looked into what it would take to provide a smooth
migration from Kryo2 -> Kryo5?
Best,
Piotrek
czw., 30 mar 2023 o 16:54 Alexis Sarda-Espinosa
<sarda.espin...@gmail.com> napisał(a):
Hi Martijn,
just to be sure, if all state-related classes use a POJO
serializer, Kryo will never come into play, right? Given
FLINK-16686 [1], I wonder how many users actually have jobs with
Kryo and RocksDB, but even if there aren't many, that still leaves
those who don't use RocksDB for checkpoints/savepoints.
If Kryo were to stay in the Flink APIs in v1.X, is it impossible
to let users choose between v2/v5 jars by separating them like
log4j2 jars?
[1] https://issues.apache.org/jira/browse/FLINK-16686
Regards,
Alexis.
Am Do., 30. März 2023 um 14:26 Uhr schrieb Martijn Visser
<martijnvis...@apache.org>:
Hi all,
I also saw a thread on this topic from Clayton Wohl [1] on
this topic, which I'm including in this discussion thread to
avoid that it gets lost.
From my perspective, there's two main ways to get to Java 17:
1. The Flink community agrees that we upgrade Kryo to a later
version, which means breaking all checkpoint/savepoint
compatibility and releasing a Flink 2.0 with Java 17 support
added and Java 8 and Flink Scala API support dropped. This is
probably the quickest way, but would still mean that we expose
Kryo in the Flink APIs, which is the main reason why we
haven't been able to upgrade Kryo at all.
2. There's a contributor who makes a contribution that bumps
Kryo, but either a) automagically reads in all old
checkpoints/savepoints in using Kryo v2 and writes them to new
snapshots using Kryo v5 (like is mentioned in the Kryo
migration guide [2][3] or b) provides an offline tool that
allows users that are interested in migrating their snapshots
manually before starting from a newer version. That
potentially could prevent the need to introduce a new Flink
major version. In both scenarios, ideally the contributor
would also help with avoiding the exposure of Kryo so that we
will be in a better shape in the future.
It would be good to get the opinion of the community for
either of these two options, or potentially for another one
that I haven't mentioned. If it appears that there's an
overall agreement on the direction, I would propose that a
FLIP gets created which describes the entire process.
Looking forward to the thoughts of others, including the Users
(therefore including the User ML).
Best regards,
Martijn
[1]
https://lists.apache.org/thread/qcw8wy9dv8szxx9bh49nz7jnth22p1v2
[2]
https://lists.apache.org/thread/gv49jfkhmbshxdvzzozh017ntkst3sgq
[3] https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5
On Sun, Mar 19, 2023 at 8:16 AM Tamir Sagi
<tamir.s...@niceactimize.com> wrote:
I agree, there are several options to mitigate the
migration from v2 to v5.
yet, Oracle roadmap is to end JDK 11 support in September
this year.
________________________________
From: ConradJam <jam.gz...@gmail.com>
Sent: Thursday, March 16, 2023 4:36 AM
To: d...@flink.apache.org <d...@flink.apache.org>
Subject: Re: [Discussion] - Release major Flink version to
support JDK 17 (LTS)
EXTERNAL EMAIL
Thanks for your start this discuss
I have been tracking this problem for a long time, until I
saw a
conversation in ISSUSE a few days ago and learned that the
Kryo version
problem will affect the JDK17 compilation of snapshots [1]
FLINK-24998 ,
As @cherry said it ruined our whole effort towards JDK17
I am in favor of providing an external tool to migrate
from Kryo old
version checkpoint to the new Kryo new checkpoint at one
time (Maybe this
tool start in flink 2.0 ?), does this tool currently have
any plans or
ideas worth discuss
I think it should not be difficult to be compatible with
JDK11 and JDK17.
We should indeed abandon JDK8 in 2.0.0. It is also
mentioned in the doc
that it is marked as Deprecated [2]
Here I add that we need to pay attention to the version of
Scala and the
version of JDK17
[1] FLINK-24998 IGSEGV in Kryo / C2 CompilerThread on Java 17
https://issues.apache.org/jira/browse/FLINK-24998
[2] FLINK-30501 Update Flink build instruction to
deprecate Java 8 instead
of requiring Java 11
https://issues.apache.org/jira/browse/FLINK-30501
Tamir Sagi <tamir.s...@niceactimize.com> 于2023年3月16日周四
00:54写道：
> Hey dev community,
>
> I'm writing this email to kick off a discussion
following this epic:
>
FLINK-15736<https://issues.apache.org/jira/browse/FLINK-15736>.
>
> We are moving towards JDK 17 (LTS) , the only blocker
now is Flink which
> currently remains on JDK 11 (LTS). Flink does not
support JDK 17 yet, with
> no timeline, the reason, based on the aforementioned
ticket is the
> following tickets
>
> 1. FLINK-24998 - SIGSEGV in Kryo / C2 CompilerThread
on Java 17<
> https://issues.apache.org/jira/browse/FLINK-24998>.
> 2. FLINK-3154 - Update Kryo version from 2.24.0 to
latest Kryo LTS
> version<https://issues.apache.org/jira/browse/FLINK-3154>
>
> My question is whether it is possible to release a major
version (Flink
> 2.0.0) using the latest Kryo version for those who don't
need to restore
> old savepoints/checkpoints in newer format.
>
> 1. Leverage JDK 17 features within JVM
> 2. Moving from the old format to the newer one will
be handled only
> once - a mitigation can be achieved by a conversion tool
or external
> serializers, both can be provided later on.
>
> I'd like to emphasize that the next JDK LTS (21) will be
released this
> September. furthermore, Flink already supports JDK
12-15, which is very
> close to JDK 17 (LTS) - that was released in September
2021. JDK 11 will
> become a legacy soon, as more frameworks moving towards
JDK 17 and are less
> likely to support JDK 11 in the near future. (For
example, Spring Boot 3
> requires JDK 17 already).
>
> Thank you for your consideration of my request.
>
> Tamir.
>
>
>
>
> Confidentiality: This communication and any attachments
are intended for
> the above-named persons only and may be confidential
and/or legally
> privileged. Any opinions expressed in this communication
are not
> necessarily those of NICE Actimize. If this
communication has come to you
> in error you must take no action based on it, nor must
you copy or show it
> to anyone; please delete/destroy and inform the sender
by e-mail
> immediately.
> Monitoring: NICE Actimize may monitor incoming and
outgoing e-mails.
> Viruses: Although we have taken steps toward ensuring
that this e-mail and
> attachments are free from any virus, we advise that in
keeping with good
> computing practice the recipient should ensure they are
actually virus free.
>
--
Best
ConradJam
Confidentiality: This communication and any attachments
are intended for the above-named persons only and may be
confidential and/or legally privileged. Any opinions
expressed in this communication are not necessarily those
of NICE Actimize. If this communication has come to you in
error you must take no action based on it, nor must you
copy or show it to anyone; please delete/destroy and
inform the sender by e-mail immediately.
Monitoring: NICE Actimize may monitor incoming and
outgoing e-mails.
Viruses: Although we have taken steps toward ensuring that
this e-mail and attachments are free from any virus, we
advise that in keeping with good computing practice the
recipient should ensure they are actually virus free.
