Is the intention to bump the Flink major version and only support Java
17+? If so, can Scala not be upgraded at the same time?
Thanks,
Thomas
On Thu, Apr 27, 2023 at 4:53 PM Martijn Visser
<martijnvis...@apache.org> wrote:
Scala 2.12.7 doesn't compile on Java 17, see
https://issues.apache.org/jira/browse/FLINK-25000.
On Thu, Apr 27, 2023 at 3:11 PM Jing Ge <j...@ververica.com> wrote:
> Thanks Tamir for the information. According to the latest
comment of the
> task FLINK-24998, this bug should be gone while using the latest
JDK 17. I
> was wondering whether it means that there are no more issues to
stop us
> releasing a major Flink version to support Java 17? Did I miss
something?
>
> Best regards,
> Jing
>
> On Thu, Apr 27, 2023 at 8:18 AM Tamir Sagi
<tamir.s...@niceactimize.com>
> wrote:
>
>> More details about the JDK bug here
>> https://bugs.openjdk.org/browse/JDK-8277529
>>
>> Related Jira ticket
>> https://issues.apache.org/jira/browse/FLINK-24998
>>
>> ------------------------------
>> *From:* Jing Ge via user <u...@flink.apache.org>
>> *Sent:* Monday, April 24, 2023 11:15 PM
>> *To:* Chesnay Schepler <ches...@apache.org>
>> *Cc:* Piotr Nowojski <pnowoj...@apache.org>; Alexis
Sarda-Espinosa <
>> sarda.espin...@gmail.com>; Martijn Visser
<martijnvis...@apache.org>;
>> dev@flink.apache.org <dev@flink.apache.org>; user
<u...@flink.apache.org>
>> *Subject:* Re: [Discussion] - Release major Flink version to
support JDK
>> 17 (LTS)
>>
>>
>> *EXTERNAL EMAIL*
>>
>>
>> Thanks Chesnay for working on this. Would you like to share
more info
>> about the JDK bug?
>>
>> Best regards,
>> Jing
>>
>> On Mon, Apr 24, 2023 at 11:39 AM Chesnay Schepler
<ches...@apache.org>
>> wrote:
>>
>> As it turns out Kryo isn't a blocker; we ran into a JDK bug.
>>
>> On 31/03/2023 08:57, Chesnay Schepler wrote:
>>
>>
>>
https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5#migration-guide
>>
>> Kroy themselves state that v5 likely can't read v2 data.
>>
>> However, both versions can be on the classpath without
classpath as v5
>> offers a versioned artifact that includes the version in the
package.
>>
>> It probably wouldn't be difficult to migrate a savepoint to
Kryo v5,
>> purely from a read/write perspective.
>>
>> The bigger question is how we expose this new Kryo version in
the API. If
>> we stick to the versioned jar we need to either duplicate all
current
>> Kryo-related APIs or find a better way to integrate other
serialization
>> stacks.
>> On 30/03/2023 17:50, Piotr Nowojski wrote:
>>
>> Hey,
>>
>> > 1. The Flink community agrees that we upgrade Kryo to a later
version,
>> which means breaking all checkpoint/savepoint compatibility and
releasing a
>> Flink 2.0 with Java 17 support added and Java 8 and Flink Scala
API support
>> dropped. This is probably the quickest way, but would still
mean that we
>> expose Kryo in the Flink APIs, which is the main reason why we
haven't been
>> able to upgrade Kryo at all.
>>
>> This sounds pretty bad to me.
>>
>> Has anyone looked into what it would take to provide a smooth
migration
>> from Kryo2 -> Kryo5?
>>
>> Best,
>> Piotrek
>>
>> czw., 30 mar 2023 o 16:54 Alexis Sarda-Espinosa
<sarda.espin...@gmail.com>
>> napisał(a):
>>
>> Hi Martijn,
>>
>> just to be sure, if all state-related classes use a POJO
serializer, Kryo
>> will never come into play, right? Given FLINK-16686 [1], I
wonder how many
>> users actually have jobs with Kryo and RocksDB, but even if
there aren't
>> many, that still leaves those who don't use RocksDB for
>> checkpoints/savepoints.
>>
>> If Kryo were to stay in the Flink APIs in v1.X, is it
impossible to let
>> users choose between v2/v5 jars by separating them like log4j2
jars?
>>
>> [1] https://issues.apache.org/jira/browse/FLINK-16686
>>
>> Regards,
>> Alexis.
>>
>> Am Do., 30. März 2023 um 14:26 Uhr schrieb Martijn Visser <
>> martijnvis...@apache.org>:
>>
>> Hi all,
>>
>> I also saw a thread on this topic from Clayton Wohl [1] on this
topic,
>> which I'm including in this discussion thread to avoid that it
gets lost.
>>
>> From my perspective, there's two main ways to get to Java 17:
>>
>> 1. The Flink community agrees that we upgrade Kryo to a later
version,
>> which means breaking all checkpoint/savepoint compatibility and
releasing a
>> Flink 2.0 with Java 17 support added and Java 8 and Flink Scala
API support
>> dropped. This is probably the quickest way, but would still
mean that we
>> expose Kryo in the Flink APIs, which is the main reason why we
haven't been
>> able to upgrade Kryo at all.
>> 2. There's a contributor who makes a contribution that bumps
Kryo, but
>> either a) automagically reads in all old checkpoints/savepoints
in using
>> Kryo v2 and writes them to new snapshots using Kryo v5 (like is
mentioned
>> in the Kryo migration guide [2][3] or b) provides an offline
tool that
>> allows users that are interested in migrating their snapshots
manually
>> before starting from a newer version. That potentially could
prevent the
>> need to introduce a new Flink major version. In both scenarios,
ideally the
>> contributor would also help with avoiding the exposure of Kryo
so that we
>> will be in a better shape in the future.
>>
>> It would be good to get the opinion of the community for either
of these
>> two options, or potentially for another one that I haven't
mentioned. If it
>> appears that there's an overall agreement on the direction, I
would propose
>> that a FLIP gets created which describes the entire process.
>>
>> Looking forward to the thoughts of others, including the Users
(therefore
>> including the User ML).
>>
>> Best regards,
>>
>> Martijn
>>
>> [1]
https://lists.apache.org/thread/qcw8wy9dv8szxx9bh49nz7jnth22p1v2
>> [2]
https://lists.apache.org/thread/gv49jfkhmbshxdvzzozh017ntkst3sgq
>> [3] https://github.com/EsotericSoftware/kryo/wiki/Migration-to-v5
>>
>> On Sun, Mar 19, 2023 at 8:16 AM Tamir Sagi
<tamir.s...@niceactimize.com>
>> wrote:
>>
>> I agree, there are several options to mitigate the migration
from v2 to
>> v5.
>> yet, Oracle roadmap is to end JDK 11 support in September this
year.
>>
>>
>>
>> ________________________________
>> From: ConradJam <jam.gz...@gmail.com>
>> Sent: Thursday, March 16, 2023 4:36 AM
>> To: dev@flink.apache.org <dev@flink.apache.org>
>> Subject: Re: [Discussion] - Release major Flink version to
support JDK 17
>> (LTS)
>>
>> EXTERNAL EMAIL
>>
>>
>>
>> Thanks for your start this discuss
>>
>>
>> I have been tracking this problem for a long time, until I saw a
>> conversation in ISSUSE a few days ago and learned that the Kryo
version
>> problem will affect the JDK17 compilation of snapshots [1]
FLINK-24998 ,
>>
>> As @cherry said it ruined our whole effort towards JDK17
>>
>> I am in favor of providing an external tool to migrate from
Kryo old
>> version checkpoint to the new Kryo new checkpoint at one time
(Maybe this
>> tool start in flink 2.0 ?), does this tool currently have any
plans or
>> ideas worth discuss
>>
>>
>> I think it should not be difficult to be compatible with JDK11
and JDK17.
>> We should indeed abandon JDK8 in 2.0.0. It is also mentioned in
the doc
>> that it is marked as Deprecated [2]
>>
>>
>> Here I add that we need to pay attention to the version of
Scala and the
>> version of JDK17
>>
>>
>> [1] FLINK-24998 IGSEGV in Kryo / C2 CompilerThread on Java 17
>> https://issues.apache.org/jira/browse/FLINK-24998
>>
>> [2] FLINK-30501 Update Flink build instruction to deprecate
Java 8 instead
>> of requiring Java 11
https://issues.apache.org/jira/browse/FLINK-30501
>>
>> Tamir Sagi <tamir.s...@niceactimize.com> 于2023年3月16日周四
00:54写道：
>>
>> > Hey dev community,
>> >
>> > I'm writing this email to kick off a discussion following
this epic:
>> > FLINK-15736<https://issues.apache.org/jira/browse/FLINK-15736>.
>> >
>> > We are moving towards JDK 17 (LTS) , the only blocker now is
Flink which
>> > currently remains on JDK 11 (LTS). Flink does not support JDK
17 yet,
>> with
>> > no timeline, the reason, based on the aforementioned ticket
is the
>> > following tickets
>> >
>> > 1. FLINK-24998 - SIGSEGV in Kryo / C2 CompilerThread on
Java 17<
>> > https://issues.apache.org/jira/browse/FLINK-24998>.
>> > 2. FLINK-3154 - Update Kryo version from 2.24.0 to latest
Kryo LTS
>> > version<https://issues.apache.org/jira/browse/FLINK-3154>
>> >
>> > My question is whether it is possible to release a major
version (Flink
>> > 2.0.0) using the latest Kryo version for those who don't need
to restore
>> > old savepoints/checkpoints in newer format.
>> >
>> > 1. Leverage JDK 17 features within JVM
>> > 2. Moving from the old format to the newer one will be
handled only
>> > once - a mitigation can be achieved by a conversion tool or
external
>> > serializers, both can be provided later on.
>> >
>> > I'd like to emphasize that the next JDK LTS (21) will be
released this
>> > September. furthermore, Flink already supports JDK 12-15,
which is very
>> > close to JDK 17 (LTS) - that was released in September 2021.
JDK 11
>> will
>> > become a legacy soon, as more frameworks moving towards JDK
17 and are
>> less
>> > likely to support JDK 11 in the near future. (For example,
Spring Boot 3
>> > requires JDK 17 already).
>> >
>> > Thank you for your consideration of my request.
>> >
>> > Tamir.
>> >
>> >
>> >
>> >
>> > Confidentiality: This communication and any attachments are
intended for
>> > the above-named persons only and may be confidential and/or
legally
>> > privileged. Any opinions expressed in this communication are not
>> > necessarily those of NICE Actimize. If this communication has
come to
>> you
>> > in error you must take no action based on it, nor must you
copy or show
>> it
>> > to anyone; please delete/destroy and inform the sender by e-mail
>> > immediately.
>> > Monitoring: NICE Actimize may monitor incoming and outgoing
e-mails.
>> > Viruses: Although we have taken steps toward ensuring that
this e-mail
>> and
>> > attachments are free from any virus, we advise that in
keeping with good
>> > computing practice the recipient should ensure they are
actually virus
>> free.
>> >
>>
>>
>> --
>> Best
>>
>> ConradJam
>>
>> Confidentiality: This communication and any attachments are
intended for
>> the above-named persons only and may be confidential and/or legally
>> privileged. Any opinions expressed in this communication are not
>> necessarily those of NICE Actimize. If this communication has
come to you
>> in error you must take no action based on it, nor must you copy
or show it
>> to anyone; please delete/destroy and inform the sender by e-mail
>> immediately.
>> Monitoring: NICE Actimize may monitor incoming and outgoing
e-mails.
>> Viruses: Although we have taken steps toward ensuring that this
e-mail
>> and attachments are free from any virus, we advise that in
keeping with
>> good computing practice the recipient should ensure they are
actually virus
>> free.
>>
>>
>>
>>
>> Confidentiality: This communication and any attachments are
intended for
>> the above-named persons only and may be confidential and/or legally
>> privileged. Any opinions expressed in this communication are not
>> necessarily those of NICE Actimize. If this communication has
come to you
>> in error you must take no action based on it, nor must you copy
or show it
>> to anyone; please delete/destroy and inform the sender by e-mail
>> immediately.
>> Monitoring: NICE Actimize may monitor incoming and outgoing
e-mails.
>> Viruses: Although we have taken steps toward ensuring that this
e-mail
>> and attachments are free from any virus, we advise that in
keeping with
>> good computing practice the recipient should ensure they are
actually virus
>> free.
>>
>
