Join Us for Flink Forward Asia 2024 in Shanghai (Nov 29-30) & Jakarta (Dec 5)!

[Xintong Song]

Dear Flink Community,


We are excited to share some important news with you!


Flink Forward Asia 2024 is coming up with two major events: the first in
Shanghai on November 29-30, and the second in Jakarta on December 5. These
gatherings will focus on the latest developments, future plans, and
production practices within the Apache Flink and Apache Paimon communities.


For the first time since its inception in 2018, Flink Forward Asia is
expanding into Southeast Asia, marking a significant milestone for the
global Apache Flink community.


Additionally, we’re pleased to announce the launch of our new website for
Flink Forward Asia. You can find more information about the events at
asia.flink-forward.org.


We look forward to open discussions on the latest advancements and
innovative applications of Flink technology, and we encourage greater
participation and influence within our community.


Don’t miss out! Register now for Flink Forward Asia Jakarta 2024 or submit
your presentations here: asia.flink-forward.org/jakarta-2024.


You can also register for Flink Forward Asia Shanghai 2024 or submit your
presentations here: asia.flink-forward.org/shanghai-2024.


We hope to see you there!


Best,

Xintong

[SUMMARY] Flink 2.0 Sync 09/25/2024

[Xintong Song]

Hi devs,

Here's a TL;DR summary of today's release sync. See the wiki page [1] for
more details.

   - Breaking changes
  - *All breaking changes* are expected to complete by *Sep 30*. For
  anything that may not finish on-time, please reach out to the release
  managers.
  - Current status: Most are in good progress and will be completed
  this week
  - Blocker: State of dropping Java 8 support is unclear.
   - The release managers have started working on the release announcement
   for the preview version, and will soon open a PR for further collecting
   contents.
   - A dry-run for building the preview release has been conducted.
   - We plan to support Kafka connectors in 2.0-preview, a few more
   commonly used connectors in 2.0.0, and 80% of first-party connectors in the
   following year. A more concrete plan and list is on the way.
   - Due to public holidays in China, *there won't be a release sync next
   week (Oct 2).*


Best,

Xintong


[1]
https://cwiki.apache.org/confluence/display/FLINK/2.0+Release#id-2.0Release-09/11/24

[KINDELY REMINDER] Flink 2.0 Release Sync 09/25/2024

[Xintong Song]

The release sync will be held at the following time.
- 14:00, Sep 25, BJT (GMT+8)
- 08:00, Sep 25, CEST (GMT+2)
- 23:00, Sep 24, PDT (GMT-7)

Everyone is welcome to join via Google Meet [1]. The topics to be discussed
can be found at the Status / Follow-ups section of the release wiki page
[2]. If there's other topics that you'd like to discuss, please feel free
to add to the list.

Best,

Xintong


[1] https://meet.google.com/fre-kcvf-mwt
[2] https://cwiki.apache.org/confluence/display/FLINK/2.0+Release

[jira] [Created] (FLINK-36360) Prepare release process and Scripts for the preview release

[Xintong Song (Jira)]

Xintong Song created FLINK-36360:


 Summary: Prepare release process and Scripts for the preview 
release
 Key: FLINK-36360
 URL: https://issues.apache.org/jira/browse/FLINK-36360
 Project: Flink
  Issue Type: New Feature
  Components: Release System
Reporter: Xintong Song
Assignee: Xintong Song
 Fix For: 2.0-preview






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[NOTICE] Flink 2.0 Release Sync Canceled for 09/18/2024

[Xintong Song]

Hi devs,

Given that many folks, including half of the release managers, are at
Current 2024, we are canceling the release sync today.

If there's anything you'd like to raise or discuss, please feel free to
reach out to the release managers.

Best,

Xintong

[SUMMARY] Flink 2.0 Release Sync 09/11/2024

[Xintong Song]

Hi devs,

Here's a TL;DR summary of today's release sync. See the wiki page [1] for
more details.

   - There are some PyFlink tests that depend on external connectors which
   use deprecated APIs. That blocks removal of many deprecated APIs with CI
   failures. Those tests verify connector functionalities, and should be moved
   to external connector repos. Dian will work on it. To quickly unblock other
   developers, *feel free to disable the problematic test cases* for now,
   and they will be re-enabled after migrating to external repos.
   - We will disable japicmp for @Deprecated APIs until the Flink 2.0
   release. To avoid unexpected breaking changes, we will re-enable the checks
   locally and review all the reported breaking changes right before the
   release.
   - A "2.0-preview" version has been created on Jira. For blocker issues
   of the preview release, please set FixVersion to "2.0-preview" and Priority
   to "Blocker".
   - A "3.0.0" version has been created on Jira, as well as an umbrella
   ticket [2] for Flink breaking changes. For breaking changes that we know
   for sure won't happen in Flink 2.0, please add the corresponding tickets to
   the version and umbrella.

Best,

Xintong


[1]
https://cwiki.apache.org/confluence/display/FLINK/2.0+Release#id-2.0Release-09/11/24

[2] https://issues.apache.org/jira/browse/FLINK-36261

[jira] [Created] (FLINK-36261) Revisit all breaking changes before Flink 2.0 Preview

[Xintong Song (Jira)]

Xintong Song created FLINK-36261:


 Summary: Revisit all breaking changes before Flink 2.0 Preview
 Key: FLINK-36261
 URL: https://issues.apache.org/jira/browse/FLINK-36261
 Project: Flink
  Issue Type: Technical Debt
Reporter: Xintong Song
 Fix For: 2.0-preview


Japicmp for @Deprecated APIs is disabled in FLINK-36207.

We need to check whether there's any unexpected breaking changes right before 
the Flink 2.0 Preview release, by re-enable the checking locally.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-36200) Breaking changes for Flink 3.0

[Xintong Song (Jira)]

Xintong Song created FLINK-36200:


 Summary: Breaking changes for Flink 3.0
 Key: FLINK-36200
 URL: https://issues.apache.org/jira/browse/FLINK-36200
 Project: Flink
  Issue Type: Technical Debt
Reporter: Xintong Song
 Fix For: 3.0.0


This is an umbrella for tracking planed breaking changes that are targeting for 
the next major version bump.

Please be aware that, in order for the breaking changes to be included in the 
next major version bump, the following preparations need to be done in advance:
* The old APIs need to be deprecated, 2 minor releases before the major version 
bump for @Public APIs, and 1 minor release for @PublicEvolving APIs

* All internal usages of the deprecated APIs should be either migrated to 
replacing new APIs (if any) or deprecated / removed. This ensure that we can 
quickly remove the APIs during major version bump without need for extra 
migration efforts.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSSION] Disabling japicmp plugin in master for 2.0

[Xintong Song]

Hi Matthias,

I'm not an expert in japicmp. Is it possible to only disable this for
deprecated APIs? Maybe add an exclusion for @Deprecated? I think that would
reduce the risk of unintended breaking changes to deprecated APIs that are
not yet ready for removal.

Best,

Xintong



On Thu, Aug 29, 2024 at 5:52 PM Gabor Somogyi 
wrote:

> Hi Matthias,
>
> I would turn japicmp for 2.0 off because adding a long list of exceptions
> doesn't give any value.
> 1.x and 2.x is not going to be compatible in any way and when 2.x released,
> then that will be the new
> japicmp baseline (after a heavy migration).
>
> What I see as a potential risk it that we break something which was not
> intended, but fixing those
> hopefully small amount of cases is less effort than maintaining an endless
> list.
>
> BR,
> G
>
>
> On Thu, Aug 29, 2024 at 11:40 AM Matthias Pohl  wrote:
>
> > Hi everyone,
> > for the 2.0 work, we are expecting to run into public API changes quite a
> > bit. This would get picked up by the japicmp plugin. The usual way is to
> > add exclusions to the plugin configuration [1] generating a (presumably
> > long) list of API changes.
> >
> > I'm wondering whether we, instead, would want to disable the plugin [2]
> for
> > 2.0 entirely to lower effort on the contributors side.
> >
> > Best,
> > Matthias
> >
> > [1] https://github.com/apache/flink/blob/master/pom.xml#L2367
> > [2] https://github.com/apache/flink/blob/master/pom.xml#L170
> >
>

[SUMMARY] Flink 2.0 Release Sync 08/28/2024

[Xintong Song]

Hi devs,

Here's a summary of today's release sync.

## About work items

   - Java supports
  - Java 21 will be supported in 2.0
  - Java 17 will be made default in 2.0
  - Supports for Java 8 will be dropped in 2.0
  - Martijn will start a FLIP discussion on deprecating support for
  Java 11. It will not be directly removed in 2.0
   - We will not continue with the REST API and Metrics work items in 2.0
   - We are still collecting new features for this release cycle. Please
   reachout to the release managers.

## About the Preview Release

   - *All breaking changes should be completed in the preview release
   (end-of-Sep.)*
  - There are a lot of internal usages of Source/SinkFunction and
  legacy TableSource/SinkFunction, which cannot be all migrated before the
  preview release. We will first move the public APIs to internal
interfaces,
  then migrate the internal usges gradually.
  - In future, we will update the depreciation process to require
  migrating of internal usages in advance.
   - *We have created a new version "2.0-preview" on Jira.*
  - FixVersion of tickets targeting the preview release, or with code
  changes merged into the master branch between the 1.20.0 and 2.0-preview,
  should be set to 2.0-preview.
  - We will soon move existing tickets from "2.0.0" to this version.
 - We'll check for all deprecated APIs before the preview release,
   making sure all deprecated APis are either removed or we know why they are
   not removed.

Best,

Xintong

[KINDLY REMINDER] Flink 2.0 Release Sync 08/28/2024

[Xintong Song]

The release sync will be held at the following time.
- 14:00, Aug 28, BJT (GMT+8)
- 08:00, Aug 28, CEST (GMT+2)
- 23:00, Aug 27, PDT (GMT-7)

Everyone is welcome to join via Google Meet [1]. The topics to be discussed
can be found at the Status / Follow-ups section of the release wiki page
[2]. If there's other topics that you'd like to discuss, please feel free
to add to the list.

Best,

Xintong


[1] https://meet.google.com/fre-kcvf-mwt

[2] https://cwiki.apache.org/confluence/display/FLINK/2.0+Release

[ANNOUNCE] New Apache Flink Committer - Xuannan Su

[Xintong Song]

Hi everyone,

On behalf of the PMC, I'm happy to announce that Xuannan Su has become a
new Flink Committer!

Xuannan has been contributing to the Apache Flink project since March 2020,
and has been active in core and runtime components of the project, as well
as various ecosystem projects. His representative contributions include the
stream-batch mixed execution mode and configuration improvements for Flink
2.0. He's also an evangelist of Flink and has presented many
widely-spreading talks and articles.

Please join me in congratulating Xuannan Su for becoming an Apache Flink
committer.

Best,
Xintong (on behalf of the Flink PMC)

[SUMMARY] Flink 2.0 Release Sync 08/14/2024

[Xintong Song]

Hi devs,

We just had our first release sync, and here's a summary of it.

## TL; DR

   - We will release a preview version around the end of September. Please
   make sure all API breaking changes are finished by that time.
   - We are in progress of updating the work item lists on wiki page [1].
   Developers please add new features that you'd like to work on in this
   release cycle, and update the information and status of existing items that
   you're responsible for. I'll also reach out to people for status
   clarification.
   - The next release sync will be on 14:00, Aug 21, BJT (GMT+8). Please
   check the Google Calendar [2] for your local time.

## Preview Release

We decided to have a Preview Release before the formal 2.0 release. As
mentioned in [3], the preview release would allow people, especially
partener projects, to adapt to the API breaking changes early, and provide
feedback for us before we commit to the compatibility of these APIs in the
formal 2.0 release.

The preview release would take place at the *end of September*, and *should
include all API breaking changes* (mostly removal of deprecated APIs).
There won't be comprehensive release testing for this preview release, but
will still be a PMC vote (as required by ASF).

If you have problems finishing the API changes by the target time, please
reach out to the release managers.

## Time Plan

The target time plan for the 2.0 release is as follows:

   - *Preview Release: End of Sep, 2024*
   - *Feature Freeze: End of Nov, 2024*
   - *Release: End of Dec, 2024 or Early Jan 2025*

## Work Items

We will have 2 work items lists:

   - Breaking changes: Items are expected to be finished by the preview
   release.
   - Features: Items are expected to be finished by the feature freeze. We
   apply the same time-based rules as previous releases. Features not finished
   by the feature freeze would be postponed to the next release.

We are working on separating and updating the existing work item list.
Please help add new features that you'd like to work on, and also update
the status of existing items that you're responsible for. The release
managers will also reach out to relevant contributors for status updates.

We will also try to identify work items by scanning for deprecated APIs
(@Becket) and JIRA tickets labeled for release 2.0 (@Xintong).

We will go through the work item lists in the release sync next week.
Hopefully, we'll have most of the information updated by that time.



For any other questions, please feel free to reach out to the release
managers.
Looking forward to your feedback.

Best,

Xintong (on behalf of the release managers)


[1] https://cwiki.apache.org/confluence/display/FLINK/2.0+Release

[2] https://calendar.app.google/x6fafuKXoJAPdDby9

[3] https://lists.apache.org/thread/hdrhrlgkrbxr1clxsvfxyj2vl64738qt

Re: [DISCUSS] Flink 2.0 Release Plan

[Xintong Song]

Kindly reminder: The first release sync comes in 2 days.

I've added a "Status / Follow-ups" section in the 2.0 Release wiki page
[1]. If there's any topic you'd like to discuss, please append it to the
"Discussion" list.

Best,

Xintong


[1] https://cwiki.apache.org/confluence/display/FLINK/2.0+Release



On Tue, Aug 6, 2024 at 3:53 PM Xintong Song  wrote:

> Hi devs,
>
> It has been more than 15 months since we kicked off the 2.0 release
> plan[1]. With Flink 1.20 released last week, we finally entered the 2.0
> release cycle.
>
> As previously decided, Becket, Jark, Martijn and I will serve as the
> release managers. We had a brief offline discussion (except for Martijn
> who's on vacation), and here's the outcome.
>
> We are planning to have the *first release sync* at:
> - 14:00-15:00, Aug 14 (Wed), BJT (GMT+8)
> - 08:00-09:00, Aug 14 (Wed), CEST (GMT+2)
> - 23:00-24:00, Aug 13 (Tues), PDT (GMT-7)
> Everyone is welcome to join the release sync. You can subscribe to the
> Google Calendar[2] or join the Google Meet[3] directly. You can also find
> these links on the 2.0 Release wiki page [4].
>
> There are a few things that we'd like to bring up for discussion in this
> thread. *We'd like to hear your opinions before the release sync*, so
> that hopefully we can reach consensus on first release sync. Also, please
> feel free to *raise anything else that you'd like to discuss* in this
> thread*.*
>
> * ## Preview Release*
>
> In a previous thread [5], Jark proposed to have a preview release before
> the formal 2.0 release. The 2.0 release is expected to contain many API
> breaking changes. Having a preview release would allow people, especially
> partener projects, to adapt to the new APIs early, and provide feedback for
> us before we commit to compatibility of these APIs in the formal 2.0
> release.
>
> The preview release can be lightweight. We don't need to complete all
> planned features in it, and can focus only on API breaking changes.
> Ideally, all API breaking changes should be done in the preview release.
> However, we still allow breaking changes between the preview and formal
> releases, in order to fix problems found in feedback on the preview
> release. Most API breaking changes should already have been well planned
> and partially done in previous release cycles, the remaining tasks are
> mainly removing deprecated APIs, making them possible to complete
> relatively soon. Moreover, we don't need to spend a lot of effort on
> testing and stabilizing the release, because we don't expect people to use
> it for production.
>
> *## Time Plan*
>
> If the community agrees to have a preview release, we'd propose to have
> the following time plan.
> - Having a preview release at around the end of September. All foreseeable
> API breaking changes for release 2.0 should be included. There will be no
> feature freeze, nor comprehensive release testing, for the preview release.
> - Feature freeze for the formal 2.0 release at around the end of November.
> This gives us roughly 2 months for collecting feedback from the preview
> release.
> - Formal release at around the end of December, or likely January next
> year.
>
> *## Work Items*
>
> We have previously collected a list of must-have and nice-to-have work
> items [4]. Unfortunately, despite being marked as must-have, some work
> items won't make release 2.0, due to e.g. prior deprecation efforts not
> completed in previous releases. I think this is because we marked all API
> breaking changes as must-have, while some of them don't really have high
> priorities. Meantime, as discussed in [5], we won't block new features not
> listed in the previous list [4]. We will still collect feature plans for
> the 2.0 release cycle, as we did for other recent releases.
>
> So here's my proposal.
> - We will have two lists, for breaking changes and non-breaking changes
> respectively.
> - For breaking changes, they are expected to be completed in the preview
> release, at least the breaking part. Exceptions can be made but would
> require case-by-case discussions with the release managers, similar to
> making exceptions for feature freeze in previous releases.
> - For non-breaking changes, we'll apply the same time-based rules just
> like other minor releases, that only features completed by the feature
> freeze date will be included.
>
> If the community agrees, I can update the wiki page, update the existing
> lists and start collecting new features for this release cycle. In
> addition, I'll go through existing work items, check with the contributors
> and remove those that are no l

[DISCUSS] Flink 2.0 Release Plan

[Xintong Song]

Hi devs,

It has been more than 15 months since we kicked off the 2.0 release
plan[1]. With Flink 1.20 released last week, we finally entered the 2.0
release cycle.

As previously decided, Becket, Jark, Martijn and I will serve as the
release managers. We had a brief offline discussion (except for Martijn
who's on vacation), and here's the outcome.

We are planning to have the *first release sync* at:
- 14:00-15:00, Aug 14 (Wed), BJT (GMT+8)
- 08:00-09:00, Aug 14 (Wed), CEST (GMT+2)
- 23:00-24:00, Aug 13 (Tues), PDT (GMT-7)
Everyone is welcome to join the release sync. You can subscribe to the
Google Calendar[2] or join the Google Meet[3] directly. You can also find
these links on the 2.0 Release wiki page [4].

There are a few things that we'd like to bring up for discussion in this
thread. *We'd like to hear your opinions before the release sync*, so that
hopefully we can reach consensus on first release sync. Also, please feel
free to *raise anything else that you'd like to discuss* in this thread*.*

* ## Preview Release*

In a previous thread [5], Jark proposed to have a preview release before
the formal 2.0 release. The 2.0 release is expected to contain many API
breaking changes. Having a preview release would allow people, especially
partener projects, to adapt to the new APIs early, and provide feedback for
us before we commit to compatibility of these APIs in the formal 2.0
release.

The preview release can be lightweight. We don't need to complete all
planned features in it, and can focus only on API breaking changes.
Ideally, all API breaking changes should be done in the preview release.
However, we still allow breaking changes between the preview and formal
releases, in order to fix problems found in feedback on the preview
release. Most API breaking changes should already have been well planned
and partially done in previous release cycles, the remaining tasks are
mainly removing deprecated APIs, making them possible to complete
relatively soon. Moreover, we don't need to spend a lot of effort on
testing and stabilizing the release, because we don't expect people to use
it for production.

*## Time Plan*

If the community agrees to have a preview release, we'd propose to have the
following time plan.
- Having a preview release at around the end of September. All foreseeable
API breaking changes for release 2.0 should be included. There will be no
feature freeze, nor comprehensive release testing, for the preview release.
- Feature freeze for the formal 2.0 release at around the end of November.
This gives us roughly 2 months for collecting feedback from the preview
release.
- Formal release at around the end of December, or likely January next year.

*## Work Items*

We have previously collected a list of must-have and nice-to-have work
items [4]. Unfortunately, despite being marked as must-have, some work
items won't make release 2.0, due to e.g. prior deprecation efforts not
completed in previous releases. I think this is because we marked all API
breaking changes as must-have, while some of them don't really have high
priorities. Meantime, as discussed in [5], we won't block new features not
listed in the previous list [4]. We will still collect feature plans for
the 2.0 release cycle, as we did for other recent releases.

So here's my proposal.
- We will have two lists, for breaking changes and non-breaking changes
respectively.
- For breaking changes, they are expected to be completed in the preview
release, at least the breaking part. Exceptions can be made but would
require case-by-case discussions with the release managers, similar to
making exceptions for feature freeze in previous releases.
- For non-breaking changes, we'll apply the same time-based rules just like
other minor releases, that only features completed by the feature freeze
date will be included.

If the community agrees, I can update the wiki page, update the existing
lists and start collecting new features for this release cycle. In
addition, I'll go through existing work items, check with the contributors
and remove those that are no longer available.


Looking forward to your feedback.

Best,

Xintong


[1] https://lists.apache.org/thread/b8w5cx0qqbwzzklyn5xxf54vw9ymys1c

[2] https://calendar.app.google/YaCmuXHGmUbC7w659

[3] https://meet.google.com/fre-kcvf-mwt

[4] https://cwiki.apache.org/confluence/display/FLINK/2.0+Release

[5] https://lists.apache.org/thread/rz9n4r8cng8j80hx6zwb19xfjft5t75h

Re: [VOTE] Release 1.20.0, release candidate #2

[Xintong Song]

+1 (binding)

- reviewed flink-web PR
- verified checksum and signature
- verified source archives don't contain binaries
- built from source
- tried example jobs on a standalone cluster, and everything looks fine

Best,

Xintong



On Tue, Jul 30, 2024 at 12:13 AM Jing Ge  wrote:

> Thanks Weijie!
>
> +1 (binding)
>
> - verified signatures
> - verified checksums
> - checked Github release tag
> - reviewed the PRs
> - checked the repo
> - started a local cluster, tried with WordCount, everything was fine.
>
> Best regards,
> Jing
>
>
> On Mon, Jul 29, 2024 at 1:47 PM Samrat Deb  wrote:
>
> > Thank you Weijie for driving 1.20 release
> >
> > +1 (non-binding)
> >
> > - Verified checksums and sha512
> > - Verified signatures
> > - Verified Github release tags
> > - Build from source
> > - Start the flink cluster locally run few jobs (Statemachine and word
> > Count)
> >
> > Bests,
> > Samrat
> >
> >
> > On Mon, Jul 29, 2024 at 3:15 PM Ahmed Hamdy 
> wrote:
> >
> > > Thanks Weijie for driving
> > >
> > > +1 (non-binding)
> > >
> > > - Verified checksums
> > > - Verified signature matches Rui Fan's
> > > - Verified tag exists on Github
> > > - Build from source
> > > - Verified no binaries in source archive
> > > - Reviewed release notes PR (some nits)
> > >
> > > Best Regards
> > > Ahmed Hamdy
> > >
> > >
> > > On Thu, 25 Jul 2024 at 12:21, weijie guo 
> > > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > >
> > > > Please review and vote on the release candidate #2 for the version
> > > 1.20.0,
> > > >
> > > > as follows:
> > > >
> > > >
> > > > [ ] +1, Approve the release
> > > >
> > > > [ ] -1, Do not approve the release (please provide specific comments)
> > > >
> > > >
> > > > The complete staging area is available for your review, which
> includes:
> > > >
> > > > * JIRA release notes [1], and the pull request adding release note
> for
> > > > users [2]
> > > >
> > > > * the official Apache source release and binary convenience releases
> to
> > > be
> > > >
> > > > deployed to dist.apache.org [3], which are signed with the key with
> > > >
> > > > fingerprint B2D64016B940A7E0B9B72E0D7D0528B28037D8BC  [4],
> > > >
> > > > * all artifacts to be deployed to the Maven Central Repository [5],
> > > >
> > > > * source code tag "release-1.20.0-rc2" [6],
> > > >
> > > > * website pull request listing the new release and adding
> announcement
> > > blog
> > > >
> > > > post [7].
> > > >
> > > >
> > > > The vote will be open for at least 72 hours. It is adopted by
> majority
> > > >
> > > > approval, with at least 3 PMC affirmative votes.
> > > >
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12354210
> > > >
> > > > [2] https://github.com/apache/flink/pull/25091
> > > >
> > > > [3] https://dist.apache.org/repos/dist/dev/flink/flink-1.20.0-rc2/
> > > >
> > > > [4] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > >
> > > > [5]
> > > >
> > https://repository.apache.org/content/repositories/orgapacheflink-1752/
> > > >
> > > > [6] https://github.com/apache/flink/releases/tag/release-1.20.0-rc2
> > > >
> > > > [7] https://github.com/apache/flink-web/pull/751
> > > >
> > > >
> > > > Best,
> > > >
> > > > Robert, Rui, Ufuk, Weijie
> > > >
> > >
> >
>

Re: [DISCUSS] CLI action deprecation process

[Xintong Song]

Sounds good to me. Thank you, Ferenc.


Best,

Xintong



On Tue, Jul 23, 2024 at 9:42 PM Ferenc Csaky 
wrote:

> > IIUC, you mean applying the strict mode for the programming APIs
> > as well?
>
> Correct! The original thought that came to my mind that this could
> be achieved with something similar to the "jdeprscan" Java tool,
> that checks for deprecated usage. In our case, the presence of
> @Deprecated and @Public on a used entity would trigger the block
> in strict mode.
>
> Although I definitely agree with that this is more pressing and
> straightforward for the CLI interfaces, so I am good to only
> consider those for now.
>
> I think I will prepare an initial FLIP with the agreed parts and
> will post it to discuss further.
>
> Thanks,
> Ferenc
>
>
> On Monday, 22 July 2024 at 03:44, Xintong Song 
> wrote:
>
> >
> >
> > >
> >
> > > As I think about it, most likely orthogonal to the current
> > > discussion, but this idea seems to be applicable for the
> > > submitted job JARs as well. The same check could be done for all
> > > the submitted files and their deps. IDK if that was your original
> > > thought Xintong or not?
> >
> >
> > IIUC, you mean applying the strict mode for the programming APIs as well?
> >
> > I'm not entirely sure about this. The programming APIs are much less
> stable
> > compared to CLI interfaces. There are APIs being deprecated in almost
> every
> > release. My concern is that this may end up with users never turning the
> > strict mode on due to too many breaks.
> >
> > I'm a bit leaning towards starting with only applying the strict mode to
> > CLI interfaces, and see how it goes. But I'm also open to other opinions.
> >
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Wed, Jul 17, 2024 at 7:59 PM Ferenc Csaky ferenc.cs...@pm.me.invalid
> >
> > wrote:
> >
> > > Hi Xintong, Muhammet,
> > >
> > > Thank you for your thoughts!
> > >
> > > I agree with Xintong regarding 1), and 2).
> > >
> > > About making users aware of these compatibility guarantees,
> > > documentation and CLI print deprecation is what I already aimed
> > > for when I was merging the "run" and "run-application" actions
> > > functionality [1], but I missed the CLI page, which is a very good
> > > point to include. I will address that in a follow-up change.
> > >
> > > I also like the idea of a strict, and compatibility mode. I guess
> > > that should kick in when a "@Public" AND "@Deprecated" annotation
> > > combination is present on the called CLI action.
> > >
> > > With something like this at hand, I think applying the same
> > > deprecation process and annotations for CLI actions that we already
> > > have would make sense. Regarding the strict/compatibility strategy
> > > itself, the straightforward and easy way IMO:
> > >
> > > - Strict mode: In case a @Public entity gets @Deprecated, execution
> > > breaks right away.
> > > - Compatibility mode: This should allow execution until the code
> > > exists.
> > >
> > > I am not sure if bringing in more complexity would be beneficial,
> > > as it would also make it harder for the users to understand.
> > >
> > > As I think about it, most likely orthogonal to the current
> > > discussion, but this idea seems to be applicable for the
> > > submitted job JARs as well. The same check could be done for all
> > > the submitted files and their deps. IDK if that was your original
> > > thought Xintong or not?
> > >
> > > Best,
> > > Ferenc
> > >
> > > [1]
> > >
> https://github.com/apache/flink/commit/e56b54db40a2afac420d8d8952707c2644ba633a
> > >
> > > On Tuesday, 9 July 2024 at 06:02, Xintong Song tonysong...@gmail.com
> > > wrote:
> > >
> > > > I think there are three questions to be anwsered, and here are my two
> > > > cents.
> > > >
> > > > 1) How do we define the compatibility guarantees for cli interfaces?
> > > >
> > > > I'd be +1 for reusing the existing @Public and @PublicEvolving
> > > > annotations,
> > > > as suggested by Ferenc. Having multiple sets of rules in the same
> project
> > > > may easily confuse users.
> > > >
> > > > 2) What deprecation process is required 

Re: [DISCUSS] CLI action deprecation process

[Xintong Song]

>
> As I think about it, most likely orthogonal to the current
> discussion, but this idea seems to be applicable for the
> submitted job JARs as well. The same check could be done for all
> the submitted files and their deps. IDK if that was your original
> thought Xintong or not?
>

IIUC, you mean applying the strict mode for the programming APIs as well?

I'm not entirely sure about this. The programming APIs are much less stable
compared to CLI interfaces. There are APIs being deprecated in almost every
release. My concern is that this may end up with users never turning the
strict mode on due to too many breaks.

I'm a bit leaning towards starting with only applying the strict mode to
CLI interfaces, and see how it goes. But I'm also open to other opinions.


Best,

Xintong



On Wed, Jul 17, 2024 at 7:59 PM Ferenc Csaky 
wrote:

> Hi Xintong, Muhammet,
>
> Thank you for your thoughts!
>
> I agree with Xintong regarding 1), and 2).
>
> About making users aware of these compatibility guarantees,
> documentation and CLI print deprecation is what I already aimed
> for when I was merging the "run" and "run-application" actions
> functionality [1], but I missed the CLI page, which is a very good
> point to include. I will address that in a follow-up change.
>
> I also like the idea of a strict, and compatibility mode. I guess
> that should kick in when a "@Public" AND "@Deprecated" annotation
> combination is present on the called CLI action.
>
> With something like this at hand, I think applying the same
> deprecation process and annotations for CLI actions that we already
> have would make sense. Regarding the strict/compatibility strategy
> itself, the straightforward and easy way IMO:
>
> - Strict mode: In case a @Public entity gets @Deprecated, execution
> breaks right away.
> - Compatibility mode: This should allow execution until the code
> exists.
>
> I am not sure if bringing in more complexity would be beneficial,
> as it would also make it harder for the users to understand.
>
> As I think about it, most likely orthogonal to the current
> discussion, but this idea seems to be applicable for the
> submitted job JARs as well. The same check could be done for all
> the submitted files and their deps. IDK if that was your original
> thought Xintong or not?
>
> Best,
> Ferenc
>
> [1]
> https://github.com/apache/flink/commit/e56b54db40a2afac420d8d8952707c2644ba633a
>
>
>
> On Tuesday, 9 July 2024 at 06:02, Xintong Song 
> wrote:
>
> >
> >
> > I think there are three questions to be anwsered, and here are my two
> cents.
> >
> > 1) How do we define the compatibility guarantees for cli interfaces?
> >
> > I'd be +1 for reusing the existing @Public and @PublicEvolving
> annotations,
> > as suggested by Ferenc. Having multiple sets of rules in the same project
> > may easily confuse users.
> >
> > 2) What deprecation process is required for cli interfaces?
> >
> > If we are reusing the existing annotations, I think we'd better to have
> the
> > same deprecation process as well, for the same reason not to confuse
> users.
> >
> > 3) How do we make user aware of the compatibility guarantees and
> > deprecations of cli interfaces?
> >
> > I think there are several things that we can do.
> > - In addition to codes and JavaDocs, we should also display the
> annotations
> > (Public / PublicEvolving / Experimental Dprecated) in documentation [1]
> and
> > cli helps (things you see when execute `/bin/flink -h`).
> >
> > - Print a warning message if a deprecated command / argument is used, as
> > suggested by Muhammet.
> > - We can also provide a strict / compatible mode. E.g., we use strict
> mode
> > by default, which fails if any deprecated interface is used. In the error
> > message, we hint user that he/she can manually enable the compatible
> mode,
> > which allows deprecated interfaces. This should draw users' attention to
> > the deprecation of the interfaces, while not block the adoption of a new
> > Flink version with breaking changes. There are probably more details to
> be
> > considered, e.g., should we fail immediately once an interface is
> > deprecated in strict mode, how long do we support a deprecated interface
> in
> > compatible mode, how to properly suggest users about the compatible mode
> > while not encourage them to always stay in that mode, etc.
> >
> > Best,
> >
> > Xintong
> >
> >
> > [1]
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/cli/
> 

Re: [VOTE] Release 1.20.0, release candidate #1

[Xintong Song]

+1 (binding)

- reviewed flink-web PR
- verified checksum and signature
- verified source archives don't contain binaries
- built from source
- tried example jobs on a standalone cluster, and everything looks fine

Best,

Xintong



On Thu, Jul 18, 2024 at 4:25 PM Rui Fan <1996fan...@gmail.com> wrote:

> +1(binding)
>
> - Reviewed the flink-web PR (Left some comments)
> - Checked Github release tag
> - Verified signatures
> - Verified sha512 (hashsums)
> - The source archives don't contain any binaries
> - Build the source with Maven 3 and java8 (Checked the license as well)
> - Start the cluster locally with jdk8, and run the StateMachineExample job,
> it works fine.
>
> Best,
> Rui
>
> On Mon, Jul 15, 2024 at 10:59 PM weijie guo 
> wrote:
>
> > Hi everyone,
> >
> >
> > Please review and vote on the release candidate #1 for the version
> 1.20.0,
> >
> > as follows:
> >
> >
> > [ ] +1, Approve the release
> >
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> >
> > The complete staging area is available for your review, which includes:
> >
> > * JIRA release notes [1], and the pull request adding release note for
> >
> > users [2]
> >
> > * the official Apache source release and binary convenience releases to
> be
> >
> > deployed to dist.apache.org [3], which are signed with the key with
> >
> > fingerprint 8D56AE6E7082699A4870750EA4E8C4C05EE6861F  [4],
> >
> > * all artifacts to be deployed to the Maven Central Repository [5],
> >
> > * source code tag "release-1.20.0-rc1" [6],
> >
> > * website pull request listing the new release and adding announcement
> blog
> >
> > post [7].
> >
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> >
> > approval, with at least 3 PMC affirmative votes.
> >
> >
> > [1]
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12354210
> >
> > [2] https://github.com/apache/flink/pull/25091
> >
> > [3] https://dist.apache.org/repos/dist/dev/flink/flink-1.20.0-rc1/
> >
> > [4] https://dist.apache.org/repos/dist/release/flink/KEYS
> >
> > [5]
> > https://repository.apache.org/content/repositories/orgapacheflink-1744/
> >
> > [6] https://github.com/apache/flink/releases/tag/release-1.20.0-rc1
> >
> > [7] https://github.com/apache/flink-web/pull/751
> >
> >
> > Best,
> >
> > Robert, Rui, Ufuk, Weijie
> >
>

Re: [VOTE] FLIP-466: Introduce ProcessFunction Attribute in DataStream API V2

[Xintong Song]

+1(binding)

Best,

Xintong



On Thu, Jul 18, 2024 at 9:40 AM weijie guo 
wrote:

> +1(binding)
>
> Best regards,
>
> Weijie
>
>
> Wencong Liu  于2024年7月17日周三 21:31写道：
>
> > Hi dev,
> >
> > I'd like to start a vote on FLIP-466.
> >
> > Discussion thread:
> > https://lists.apache.org/thread/sw2or62299w0hw9jm5kdqjqj3j8rnrdt
> > FLIP:
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-466%3A+Introduce+ProcessFunction+Attribute+in+DataStream+API+V2
> >
> > Best regards,
> > Wencong Liu
>

Re: [VOTE] FLIP-460: Display source/sink I/O metrics on Flink Web UI

[Xintong Song]

+1 (binding)

Best,

Xintong



On Wed, Jul 17, 2024 at 5:58 AM Aleksandr Pilipenko 
wrote:

> +1 (non-binding)
>
> Thanks,
> Aleksandr
>
> On Tue, 16 Jul 2024 at 22:56, Ahmed Hamdy  wrote:
>
> > +1 (non-binding)
> >
> > Best Regards
> > Ahmed Hamdy
> >
> >
> > On Tue, 16 Jul 2024 at 21:58, Becket Qin  wrote:
> >
> > > +1 (binding)
> > >
> > > Thanks,
> > >
> > > Jiangjie (Becket) Qin
> > >
> > > On Tue, Jul 16, 2024 at 12:19 AM Leonard Xu  wrote:
> > >
> > > > +1(binding)
> > > >
> > > > Best,
> > > > Leonard
> > > >
> > > > > 2024年7月16日 下午3:15，Robert Metzger  写道：
> > > > >
> > > > > +1 (binding)
> > > > >
> > > > > Nice to see this fixed ;)
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Jul 16, 2024 at 8:46 AM Yong Fang 
> wrote:
> > > > >
> > > > >> +1 (binding)
> > > > >>
> > > > >> Best,
> > > > >> FangYong
> > > > >>
> > > > >>
> > > > >> On Tue, Jul 16, 2024 at 1:14 PM Zhanghao Chen <
> > > > zhanghao.c...@outlook.com>
> > > > >> wrote:
> > > > >>
> > > > >>> Hi everyone,
> > > > >>>
> > > > >>>
> > > > >>> Thanks for all the feedback about the FLIP-460: Display
> source/sink
> > > I/O
> > > > >>> metrics on Flink Web UI [1]. The discussion
> > > > >>> thread is here [2]. I'd like to start a vote on it.
> > > > >>>
> > > > >>> The vote will be open for at least 72 hours unless there is an
> > > > objection
> > > > >>> or insufficient votes.
> > > > >>>
> > > > >>> [1]
> > > > >>>
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=309496355
> > > > >>> [2]
> > https://lists.apache.org/thread/sy271nhd2jr1r942f29xbvbgq7fsd841
> > > > >>>
> > > > >>> Best,
> > > > >>> Zhanghao Chen
> > > > >>>
> > > > >>
> > > >
> > > >
> > >
> >
>

Re: [DISCUSS] FLIP-460: Display source/sink I/O metrics on Flink Web UI

[Xintong Song]

+1

Best,

Xintong



On Sat, Jul 13, 2024 at 12:54 AM Őrhidi Mátyás 
wrote:

> +1 on this. Thank you, Zhanghao!
>
> Cheers,
> Matyas
>
> On Fri, Jul 12, 2024 at 5:58 PM Becket Qin  wrote:
>
> > Thanks for the proposal, Zhanghao.
> >
> > +1 on the proposal. It makes a lot of sense.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> >
> > On Tue, May 28, 2024 at 7:43 PM Zhanghao Chen  >
> > wrote:
> >
> > > Hi all,
> > >
> > > I'd like start a discussion on FLIP-460: Display source/sink I/O
> metrics
> > > on Flink Web UI [1].
> > >
> > > Currently, the numRecordsIn & numBytesIn metrics for sources and the
> > > numRecordsOut & numBytesOut metrics for sinks are always 0 on the Flink
> > web
> > > dashboard. It is especially confusing for simple ETL jobs where
> there's a
> > > single chained operator with 0 input rate and 0 output rate. For years,
> > > Flink newbies have been asking "Why my job has zero consumption rate
> and
> > > zero production rate, is it actually working?"
> > >
> > > Connectors implementing FLIP-33 [2] have already exposed these metrics
> on
> > > the operator level, this FLIP takes a further step to expose them on
> the
> > > job overview page on Flink Web UI.
> > >
> > > Looking forward to everyone's feedback and suggestions. Thanks!
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=309496355
> > > [2]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-33%3A+Standardize+Connector+Metrics
> > >
> > > Best,
> > > Zhanghao Chen
> > >
> >
>

Re: [DISCUSS] CLI action deprecation process

[Xintong Song]

I think there are three questions to be anwsered, and here are my two cents.

1) How do we define the compatibility guarantees for cli interfaces?

I'd be +1 for reusing the existing @Public and @PublicEvolving annotations,
as suggested by Ferenc. Having multiple sets of rules in the same project
may easily confuse users.

2) What deprecation process is required for cli interfaces?

If we are reusing the existing annotations, I think we'd better to have the
same deprecation process as well, for the same reason not to confuse users.

3) How do we make user aware of the compatibility guarantees and
deprecations of cli interfaces?

I think there are several things that we can do.
- In addition to codes and JavaDocs, we should also display the annotations
(Public / PublicEvolving / Experimental Dprecated) in documentation [1] and
cli helps (things you see when execute `/bin/flink -h`).
- Print a warning message if a deprecated command / argument is used, as
suggested by Muhammet.
- We can also provide a strict / compatible mode. E.g., we use strict mode
by default, which fails if any deprecated interface is used. In the error
message, we hint user that he/she can manually enable the compatible mode,
which allows deprecated interfaces. This should draw users' attention to
the deprecation of the interfaces, while not block the adoption of a new
Flink version with breaking changes. There are probably more details to be
considered, e.g., should we fail immediately once an interface is
deprecated in strict mode, how long do we support a deprecated interface in
compatible mode, how to properly suggest users about the compatible mode
while not encourage them to always stay in that mode, etc.

Best,

Xintong


[1]
https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/cli/



On Mon, Jul 8, 2024 at 5:54 PM Muhammet Orazov
 wrote:

> Hey Ferenc,
>
> Yes correct. My thoughts were based on finding tradeoff between
> following fully deprecation process and leaner one for CLIs.
>
> Since cli are not like APIs, I think users would be aware of
> deprecation only when were remove the commands. That is they
> try to run their jobs with upgrade and it fails with action
> not available.
>
> So maybe we don't have to follow fully API `@PublicEvolving`
> process for this.
>
> Another maybe user friendly approach would be to inform with
> warning that the `run-application` cli action will be dropped,
> and suggest new action and migration on the log message.
>
> Best,
> Muhammet
>
> On 2024-07-04 20:17, Ferenc Csaky wrote:
> > Hi Muhammet,
> >
> > Thank you for your thoughts!
> >
> >> After two minor releases, and on next major version bump,
> >> we could drop the `run-application` method as suggested
> >> on discussion by Xintong.
> >
> > Here, you describe the deprecation/removal process of a public
> > API by the definition we have in the project now. So if the same
> > applies to a CLI action, why should we not enforce such behavior
> > for those as well?
> >
> > If following the same process for CLIs make sense, we should also
> > enforce the same compatibility guarantees IMO.
> >
> > Best,
> > Ferenc
> >
> >
> >
> > On Friday, 28 June 2024 at 09:30, Muhammet Orazov
> >  wrote:
> >
> >>
> >>
> >> Hey Ferenc,
> >>
> >> Thanks for starting the discussion!
> >>
> >> I agree that the CLI is user facing, but I think we don't
> >> have to treat it as other public APIs.
> >>
> >> I'd propose to throw user friendly exception for
> >> `run-application` with suggestion to use `run` case instead.
> >> This would make users aware of the change and require them
> >> to migrate their scripts.
> >>
> >> After two minor releases, and on next major version bump,
> >> we could drop the `run-application` method as suggested
> >> on discussion by Xintong.
> >>
> >> Best,
> >> Muhammet
> >>
> >>
> >> On 2024-06-26 15:33, Ferenc Csaky wrote:
> >>
> >> > Hello devs,
> >> > I would like to open a discussion about considerations regarding how
> to
> >> > deprecate CLI
> >> > actions, and what compatibility guarantees should apply to such cases.
> >> > The topic came up in
> >> > a previous discussion [1] about a current FLIP to merge the run and
> >> > run-application
> >> > behavior [2].
> >> >
> >> > According to Xintong's previous inputs, currently the Flink CLI, or
> its
> >> > actions are not handled
> >> > as public APIs by the existing definition (@Public or @PublicEvolving
> >> > annotated). So
> >> > legally it would be possible to change CLIs anytime. I agree with
> >> > Xintong that CLI actions
> >> > should be considered as public APIs, and as such, compatibility
> >> > guarantees should be
> >> > provided.
> >> >
> >> > CLI actions are defined as private constants in CliFrontend [3], so
> >> > IMO the existing rules
> >> > are not perfectly applicable as is. Both @Public and @PublicEvolving
> >> > can be applied to
> >> > fields (although for @Public that is only true as per the javadoc,
> >> > technically it can only be
> >

Re: [DISCUSS] FLIP-467: Introduce Generalized Watermarks

[Xintong Song]

Hi Jeyhun,

Thanks for working on this FLIP.

In general, I think it's a good idea to generalize the concept of Watermark
to not only representing the advancing of event time, but general
indicators / events / signals that need to be passed along the data
streams. So +1 for working towards this direction.

However, my major concern after reading this FLIP is that, the current
design might be too complicated. It tries take all possible kinds of events
(timestamp watermark, end-of-data, end-of-partition, internal watermark,
and arbitrary user defined watermarks) into consideration, which
complicates the design when it comes to serialization and propagation.

IMHO, this feature, the ability to send custom events across operators
along the data streams, has the potential to become a major differentiator
of DataStream API V2 comparing to V1. For such a feature, I don't think
it's feasible to design everything properly at the very beginning without
enough user feedbacks. I'd suggest to start with a smaller scope, and build
the feature incrementally as new demands and feedbacks arise.

To be specific, I'd suggest to:
1. Only focus on user facing events, thus Watermarks that are either
generated or handled by user codes (process functions and connectors).
Refactor of existing internal events  does not bring any benefit to users,
and may even unstablize existing mechanisms. We could do that incrementally
after the generalized watermark mechsnism becomes stable.
2. Start with a limited set of supported data types and propagation
strategies. We can add suport for arbitrary types and strategies later, if
proved necessary. By that time, we should be able to better understand the
use cases based on real feedbacks.
3. Try to minimize the set of concepts and apis that users need to
understand and work with, and make them simple and easy to understand. I'm
not saying we should not discuss designs of internal implementations in
this FLIP. Just it would be easier to understand the FLIP if it presents
first how users should understand and use the feature, then the key
internal designs in order to achieve that.

# Some detailed suggestions

## Use cases

Concrete use cases are usually helpful for designing such general
mechanism. You may examine the design by trying to use it to fulfill the
demands from the use cases. In cases you are looking for such use cases in
addition to the event-time watermaks, here are some inputs.
- In FLIP-309/327/329 [1-3], we proposed to detect the data freshness from
source, and use that information for various improvements. In DataStream
API V1, such information is carried by RecordAttributes, which is quite
similar to the genralized watermark except that we do not allow defining
arbitrary custom attributes.
- In Flink CDC, there are two phases, scaning the table at certain time
point, and cosuming the binlog from that time point. In the first phase,
there's only +I but no +U/-U/-D in the changelog, and downstream operators
can do many optimizations based on that information. We haven't bring those
optimizations to the community, because that requires the runtime layer to
understand the concept of table / sql changelogs. If we can send custom
events accross operators, without requiring runtime to understand those
events, the problem would be solved.
- In processing-time temporal join, the join operator does not wait for the
build side to complete before consuming the probe side data. This is
because the build side is contineously updated and the join operator does
not know when the initial build is finished. The result is that, initial
data from the probe side that should be joined with initial data from the
build side are missed. If we can send a signal from the build side source
to the join operator, notifying about the completion of initial build, the
problem would be solved. Similar to the previous case, such information
should not be understood by the runtime layer.

## Watermark Definition

The FLIP defines the new generalized Watermak as "indicators in data
streams", which is a bit too general.

I think we should at least include the following information:
- non-data events / records / indicators
- flow along the data streams
- can be generated / handled by process functions, connectors, and the
framework
- may need to be aligned across parallel data streams

## Types of Watermarks

Requiring users to always implement serializations for custom watermarks
might be a bit too heavy. Alternatively, we may consider only support
primitive types for Watermarks, i.e., BOOLEAN, LONG, etc. If complex types
are proved necessary in future, we can introduce STRING or BYTES so that
users can do custom serde by themselves.

Another benefit of using primitive types is that, it simplifies the
alignment semantics. Currently in this FLIP, users are required to
implement a WatermarkCombiner, which is not trivil. If we only support
limited types, we can (maybe only) provide built-in combiners for users,
e.g., 

Re: [DISCUSS] FLIP-466: Introduce ProcessFunction Attribute in DataStream API V2

[Xintong Song]

+1 for this FLIP.

I think this is similar to SQL hints, where users can provide optional
information to help the engine execute the workload more efficiently.
Having a unified mechanism for such kind of hints should improve usability
compared to introducing tons of configuration knobs.

Best,

Xintong



On Wed, Jun 26, 2024 at 8:09 PM Wencong Liu  wrote:

> Hi devs,
>
>
> I'm proposing a new FLIP[1] to introduce the ProcessFunction Attribute in
> the
> DataStream API V2. The goal is to optimize job execution by leveraging
> additional information about users' ProcessFunction logic. The proposal
> includes
> the following scenarios where the ProcessFunction Attribute can
> significantly
> enhance optimization:
>
>
> Scenario 1: If the framework recognizes that the ProcessFunction outputs
> data
> only after all input is received, the downstream operators can be
> scheduled until
> the ProcessFunction is finished, which effectively reduces resource
> consumption.
> Ignoring this information could lead to premature scheduling of downstream
> operators with no data to process. This scenario is addressed and
> optimized by FLIP-331[2].
>
>
> Scenario 2: For stream processing, where users are only interested in the
> latest
> result per key at the current time, the framework can optimize by
> adjusting the
> frequency of ProcessFunction outputs. This reduces shuffle data volume and
> downstream operator workload. If this optimization is ignored, each new
> input
> would trigger a new output. This scenario is addressed and
> optimized by FLIP-365[3].
>
>
> Scenario 3: If a user's ProcessFunction neither caches inputs nor outputs,
> recognizing this can enable object reuse for this data within the
> OperatorChain,
> enhancing performance. Without this optimization, data would be copied
> before
> being passed to the next operator. This scenario is addressed and
> optimized by FLIP-329[4].
>
>
> To unify the mechanism for utilizing additional information and optimizing
> jobs,
> we propose introducing the ProcessFunction Attribute represented by
> Java annotations, which allow users to provide relevant information about
> their
> ProcessFunctions. The framework can then use this to optimize job
> execution.
> Importantly, regular job execution remains unaffected whether users use
> this
> attribute or not.
>
>
> Looking forward to discussing this in the upcoming FLIP.
>
>
> Best regards,
> Wencong Liu
>
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-466%3A+Introduce+ProcessFunction+Attribute+in+DataStream+API+V2
> [2]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-331%3A+Support+EndOfStreamTrigger+and+isOutputOnlyAfterEndOfStream+operator+attribute+to+optimize+task+deployment
> [3]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-365%3A+Introduce+flush+interval+to+adjust+the+interval+of+emitting+results+with+idempotent+semantics
> [4]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-329%3A+Add+operator+attribute+to+specify+support+for+object-reuse

Re: [2.0] How to handle on-going feature development in Flink 2.0?

[Xintong Song]

I also don't think we should block new feature development until 2.0. From
my understanding, the new major release is no different from the regular
minor releases for new features.

I think tracking new features, either as nice-to-have items or in a
separate list, is necessary. It helps us understand what's going on in the
release cycle, and what to announce and promote. Maybe we should start a
discussion on updating the 2.0 item list, to 1) collect new items that are
proposed / initiated after the original list being created and 2) to remove
some items that are no longer suitable. I'll discuss this with the other
release managers first.

For the connectors and operators, I think it depends on whether they depend
on any deprecated APIs or internal implementations of Flink. Ideally,
all @Public APIs and @PublicEvolving APIs that we plan to change / remove
should have been deprecated in 1.19 and 1.20 respectively. That means if
the connectors and operators only use non-deprecated @Puclib
and @PublicEvolving APIs in 1.20, hopefully there should not be any
problems upgrading to 2.0.

Best,

Xintong



On Wed, Jun 26, 2024 at 5:20 AM Becket Qin  wrote:

> Thanks for the question, Matthias.
>
> My two cents, I don't think we are blocking new feature development. My
> understanding is that the community will just prioritize removing
> deprecated APIs in the 2.0 dev cycle. Because of that, it is possible that
> some new feature development may slow down a little bit since some
> contributors may be working on the must-have features for 2.0. But policy
> wise, I don't see a reason to block the new feature development for the 2.0
> release feature plan[1].
>
> Process wise, I like your idea of adding the new features as nice-to-have
> in the 2.0 feature list.
>
> Re: David,
> Given it is a major version bump. It is possible that some of the
> downstream projects (e.g. connectors, Paimon, etc) will have to see if a
> major version bump is also needed there. And it is probably going to be
> decisions made on a per-project basis.
> Regarding the Java version specifically, this probably worth a separate
> discussion. According to a recent report[2] on the state of Java, it might
> be a little early to drop support for Java 11. We can discuss this
> separately.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> [1] https://cwiki.apache.org/confluence/display/FLINK/2.0+Release
> [2]
>
> https://newrelic.com/sites/default/files/2024-04/new-relic-state-of-the-java-ecosystem-report-2024-04-30.pdf
>
> On Tue, Jun 25, 2024 at 4:58 AM David Radley 
> wrote:
>
> > Hi,
> > I think this is a great question. I am not sure if this has been covered
> > elsewhere, but it would be good to be clear how this effects the
> connectors
> > and operator repos, with potentially v1 and v2 oriented new featuresI
> > suspect this will be a connector by connector investigation. I am
> thinking
> > connectors with Hadoop eco-system dependencies (e.g. Paimon) which may
> not
> > work nicely with Java 17,
> >
> >  Kind regards, David.
> >
> >
> > From: Matthias Pohl 
> > Date: Tuesday, 25 June 2024 at 09:57
> > To: dev@flink.apache.org 
> > Cc: Xintong Song , martijnvis...@apache.org <
> > martijnvis...@apache.org>, imj...@gmail.com ,
> > becket@gmail.com 
> > Subject: [EXTERNAL] [2.0] How to handle on-going feature development in
> > Flink 2.0?
> > Hi 2.0 release managers,
> > With the 1.20 release branch being cut [1], master is now referring to
> > 2.0-SNAPSHOT. I remember that, initially, the community had the idea of
> > keeping the 2.0 release as small as possible focusing on API changes [2].
> >
> > What does this mean for new features? I guess blocking them until 2.0 is
> > released is not a good option. Shall we treat new features as
> > "nice-to-have" items as documented in the 2.0 release overview [3] and
> > merge them to master like it was done for minor releases in the past? Do
> > you want to add a separate section in the 2.0 release overview [3] to
> list
> > these new features (e.g. FLIP-461 [4]) separately? That might help to
> > manage planned 2.0 deprecations/API removal and new features separately.
> Or
> > do you have a different process in mind?
> >
> > Apologies if this was already discussed somewhere. I didn't manage to
> find
> > anything related to this topic.
> >
> > Best,
> > Matthias
> >
> > [1] https://lists.apache.org/thread/mwnfd7o10xo6ynx0n640pw9v2opbkm8l
> > [2] https://lists.apache.org/thread/b8w5cx0qqbwzzklyn5xxf54vw9ymys1c
> > [3] https://cwiki.apache.org/confluence/display/FLINK/2.0+Release
> > [4]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-461%3A+Synchronize+rescaling+with+checkpoint+creation+to+minimize+reprocessing+for+the+AdaptiveScheduler
> >
> > Unless otherwise stated above:
> >
> > IBM United Kingdom Limited
> > Registered in England and Wales with number 741598
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
> >
>

Re: [ANNOUNCE] Flink 1.20 feature freeze

[Xintong Song]

Hi Ferenc,

About the deprecation process, removing a @Public API requires the API to
be deprecated for at least 2 minor releases AND the removal should be in a
major version bump. That means you cannot remove a @Public API in 2.x when
x is not 0. The tricky part is, run-application as a command-line interface
does not have
any @Public/@PublicEvolving/@Experimental/@Deprecated annotations (nor
CliFrontend, the class it is defined in). Command-line interfaces are
definitely public APIs IMO, but there's no explicit deprecation process for
them, which also means we never committed that command-line interfaces will
stay compatible.

My suggestion would be to start a separate discussion thread on whether and
how to add explicit compatibility guarantees for command-line interfaces.
Without that, "legally" we can change command-line interfaces anytime,
though I'd be negative doing so.

As for the PR, I'd be in favor of not merging it for 1.20. Because we have
passed the feature freeze date for half a week and the PR is still not yet
reviewed, and the necessity for making it into 1.20 is unclear due to
absence of explicit process.

Best,

Xintong



On Wed, Jun 19, 2024 at 1:33 AM Ferenc Csaky 
wrote:

> Hi Robert, Rui, Ufuk, and Weijie,
>
> I would like to raise the PR about merging `flink run` and
> `flink run-application` functionality [1] to get considered as
> part of the 1.20 release.
>
> The reason IMO is that the `run-application` CLI command should be
> removed in the same release when Per-Job mode gets removed. AFAIK
> when we deprecate a public API, it has to stay for 2 minor
> releases to give time for users to adapt. According to that, if
> `run-application` is deprecated in Flink 2.0, it can get removed
> in Flink 2.3. Currently the drop of per-job mode is blocked [2]
> and probably it will not be resolved for a while, but I could
> imagine it would be possible in 2.1 or 2.2.
>
> The change itself is rather small and concise, and Marton Balassi
> volunteered to review it ASAP.
>
> Pls. correct me if I am wrong about the deprecation process.
>
> Looking forward to your opinion!
>
> Thanks,
> Ferenc
>
> [1] https://issues.apache.org/jira/browse/FLINK-35625
> [2] https://issues.apache.org/jira/browse/FLINK-26000
>
>
> On Tuesday, 18 June 2024 at 11:27, weijie guo 
> wrote:
>
> >
> >
> > Hi Zakelly,
> >
> > Thank you for informing us!
> >
> > After discussion, all RMs agreed that this was an important fix that
> should
> > be merged into 1.20.
> >
> > So feel free to merge it.
> >
> > Best regards,
> >
> > Weijie
> >
> >
> > Zakelly Lan zakelly@gmail.com 于2024年6月15日周六 16:29写道：
> >
> > > Hi Robert, Rui, Ufuk and Weijie,
> > >
> > > Thanks for the update!
> > >
> > > FYI: This PR[1] fixes & cleanup the left-over checkpoint directories
> for
> > > file-merging on TM exit. And the second commit fixes the wrong state
> handle
> > > usage. We encountered several unexpected CI fails, so we missed the
> feature
> > > freeze time. It is better to have this PR in 1.20 so I will merge this
> if
> > > you agree. Thanks.
> > >
> > > [1] https://github.com/apache/flink/pull/24933
> > >
> > > Best,
> > > Zakelly
> > >
> > > On Sat, Jun 15, 2024 at 6:00 AM weijie guo guoweijieres...@gmail.com
> > > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > The feature freeze of 1.20 has started now. That means that no new
> > > > features
> > > >
> > > > or improvements should now be merged into the master branch unless
> you
> > > > ask
> > > >
> > > > the release managers first, which has already been done for PRs, or
> > > > pending
> > > >
> > > > on CI to pass. Bug fixes and documentation PRs can still be merged.
> > > >
> > > > - Cutting release branch
> > > >
> > > > Currently we have no blocker issues(beside tickets that used for
> > > > release-testing).
> > > >
> > > > We are planning to cut the release branch on next Friday (June 21) if
> > > > no new test instabilities, and we'll make another announcement in the
> > > > dev mailing list then.
> > > >
> > > > - Cross-team testing
> > > >
> > > > The release testing will start right after we cut the release branch,
> > > > which
> > > >
> > > > is expected to come in the next week. As a prerequisite of it, we
> have
> > > > created
> > > >
> > > > the corresponding instruction ticket in FLINK-35602 [1], please check
> > > > and complete the
> > > >
> > > > documentation and test instruction of your new feature and mark the
> > > > related JIRA
> > > >
> > > > issue in the 1.20 release wiki page [2] before we start testing,
> which
> > > >
> > > > would be quite helpful for other developers to validate your
> features.
> > > >
> > > > Best regards,
> > > >
> > > > Robert, Rui, Ufuk and Weijie
> > > >
> > > > [1]https://issues.apache.org/jira/browse/FLINK-35602
> > > >
> > > > [2] https://cwiki.apache.org/confluence/display/FLINK/1.20+Release
>

Re: [VOTE] FLIP-462: Support Custom Data Distribution for Input Stream of Lookup Join

[Xintong Song]

+1 (binding)

Best,

Xintong



On Mon, Jun 17, 2024 at 11:41 AM Zhanghao Chen 
wrote:

> +1 (unbinding)
>
> Best,
> Zhanghao Chen
> 
> From: weijie guo 
> Sent: Monday, June 17, 2024 10:13
> To: dev 
> Subject: [VOTE] FLIP-462: Support Custom Data Distribution for Input
> Stream of Lookup Join
>
> Hi everyone,
>
>
> Thanks for all the feedback about the FLIP-462: Support Custom Data
> Distribution for Input Stream of Lookup Join [1]. The discussion
> thread is here [2].
>
>
> The vote will be open for at least 72 hours unless there is an
> objection or insufficient votes.
>
>
> Best,
>
> Weijie
>
>
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-462+Support+Custom+Data+Distribution+for+Input+Stream+of+Lookup+Join
>
>
> [2] https://lists.apache.org/thread/kds2zrcdmykrz5lmn0hf9m4phdl60nfb
>

Re: [VOTE] FLIP-464: Merge "flink run" and "flink run-application"

[Xintong Song]

+1(binding)

Best,

Xintong



On Thu, Jun 13, 2024 at 5:15 AM Márton Balassi 
wrote:

> +1 (binding)
>
> On Wed, Jun 12, 2024 at 7:25 PM Őrhidi Mátyás 
> wrote:
>
> > Sounds reasonable,
> > +1
> >
> > Cheers,
> > Matyas
> >
> >
> > On Wed, Jun 12, 2024 at 8:54 AM Mate Czagany  wrote:
> >
> > > Hi,
> > >
> > > Thank you for driving this Ferenc,
> > > +1 (non-binding)
> > >
> > > Regards,
> > > Mate
> > >
> > > Ferenc Csaky  ezt írta (időpont: 2024.
> jún.
> > > 12., Sze, 17:23):
> > >
> > > > Hello devs,
> > > >
> > > > I would like to start a vote about FLIP-464 [1]. The FLIP is about to
> > > > merge back the
> > > > "flink run-application" functionality to "flink run", so the latter
> > will
> > > > be capable to deploy jobs in
> > > > all deployment modes. More details in the FLIP. Discussion thread
> [2].
> > > >
> > > > The vote will be open for at least 72 hours (until 2024 March 23
> 14:03
> > > > UTC) unless there
> > > > are any objections or insufficient votes.
> > > >
> > > > Thanks,Ferenc
> > > >
> > > > [1]
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=311626179
> > > > [2] https://lists.apache.org/thread/xh58xs0y58kqjmfvd4yor79rv6dlcg5g
> > >
> >
>

Re: [DISCUSS] Connector Externalization Retrospective

[Xintong Song]

Thanks for bringing this up, Danny. This is indeed an important issue that
the community needs to improve on.

Personally, I think a mono-repo might not be a bad idea, if we apply
different rules for the connector releases. To be specific:
- flink-connectors 1.19.x contains all connectors that are compatible with
Flink 1.19.x.
- allow not only bug-fixes, but also new features for a third-digit release
(e.g., flink-connectors 1.19.1)

This would allow us to immediately release flink-connectors 1.19.0 right
after flink 1.19.0 is out, excluding connectors that are no longer
compatible with flink 1.19. Then we can have a couple of flink-connectors
1.19.x releases, gradually adding the missing connectors back. In the worst
case, this would result in as many releases as having separated connector
repose. The benefit comes from 1) there are chances to combine releasing of
multiple connectors into one release of the mono repo (if they are ready
around the same time), and 2) no need to maintain a compatibility matrix
and worrying about it being out-of-sync with the code base.

However, one thing I don't like about this approach is that it requires
combining all the repos we just separated from the main-repo to another
mono-repo. That back-and-forth is annoying. So I'm just speaking out my
ideas, but would not strongly insist on this.

And big +1 for compatibility tools and ci checks.

Best,

Xintong



On Tue, Jun 11, 2024 at 2:38 AM David Radley 
wrote:

> Hi Danny,
> I think your proposal is a good one. This is the approach that we took
> with the Egeria project, firstly taking the connectors out of the main
> repo, then connectors having their own versions that incremented
> organically rather then tied to the core release.
>
> Blue sky thinking - I wonder if we could :
> - have a wizard / utility so the user inputs which Flink level they want
> and which connectors; the utility knows the compatibility matrix and
> downloads the appropriate bundles.
> - have the docs interrogate the core and connector repos to check the poms
> for the Flink levels and the pr builds to have ?live? docs showing the
> supported Flink levels. PyTorch does something like this for it?s docs.
>
> Kind regards, David.
>
>
>
> From: Danny Cranmer 
> Date: Monday, 10 June 2024 at 17:26
> To: dev 
> Subject: [EXTERNAL] [DISCUSS] Connector Externalization Retrospective
> Hello Flink community,
>
> It has been over 2 years [1] since we started externalizing the Flink
> connectors to dedicated repositories from the main Flink code base. The
> past discussions can be found here [2]. The community decided to
> externalize the connectors to primarily 1/ improve stability and speed of
> the CI, and 2/ decouple version and release lifecycle to allow the projects
> to evolve independently. The outcome of this has resulted in each connector
> requiring a dedicated release per Flink minor version, which is a burden on
> the community. Flink 1.19.0 was released on 2024-03-18 [3], the first
> supported connector followed roughly 2.5 months later on 2024-06-06 [4]
> (MongoDB). There are still 5 connectors that do not support Flink 1.19 [5].
>
> Two decisions contribute to the high lag between releases. 1/ creating one
> repository per connector instead of a single flink-connector mono-repo and
> 2/ coupling the Flink version to the connector version [6]. A single
> connector repository would reduce the number of connector releases from N
> to 1, but would couple the connector CI and reduce release flexibility.
> Decoupling the connector versions from Flink would eliminate the need to
> release each connector for each new Flink minor version, but we would need
> a new compatibility mechanism.
>
> I propose that from each next connector release we drop the coupling on the
> Flink version. For example, instead of 3.4.0-1.20 (.) we
> would release 3.4.0 (). We can model a compatibility matrix
> within the Flink docs to help users pick the correct versions. This would
> mean we would usually not need to release a new connector version per Flink
> version, assuming there are no breaking changes. Worst case, if breaking
> changes impact all connectors we would still need to release all
> connectors. However, for Flink 1.17 and 1.18 there were only a handful of
> issues (breaking changes), and mostly impacting tests. We could decide to
> align this with Flink 2.0, however I see no compelling reason to do so.
> This was discussed previously [2] as a long term goal once the connector
> APIs are stable. But I think the current compatibility rules support this
> change now.
>
> I would prefer to not create a connector mono-repo. Separate repos gives
> each connector more flexibility to evolve independently, and removing
> unnecessary releases will significantly reduce the release effort.
>
> I would like to hear opinions and ideas from the community. In particular,
> are there any other issues you have observed that we should consider
> addressing?
>
> Thanks,
> Da

Re: [VOTE] Release 1.19.1, release candidate #1

[Xintong Song]

+1 (binding)

- verified checksums and signatures
- built from source
- executed example jobs on a standalone cluster

Best,

Xintong



On Mon, Jun 10, 2024 at 6:54 PM Hong Liang  wrote:

> Thanks for testing the release candidate, everyone. Nice to see coverage on
> different types of testing being done.
>
> I've addressed the comments on the web PR - thanks Rui Fan for good
> comments, and for the reminder from Ahmed :)
>
> We have <24 hours on the vote wait time, and still waiting on 1 more
> binding vote!
>
> Regards,
> Hong
>
> On Sat, Jun 8, 2024 at 11:33 PM Ahmed Hamdy  wrote:
>
> > Hi Hong,
> > Thanks for driving
> >
> > +1 (non-binding)
> >
> > - Verified signatures and hashes
> > - Checked github release tag
> > - Verified licenses
> > - Checked that the source code does not contain binaries
> > - Reviewed Web PR, nit: Could we address the comment of adding
> FLINK-34633
> > in the release
> >
> >
> > Best Regards
> > Ahmed Hamdy
> >
> >
> > On Sat, 8 Jun 2024 at 22:22, Jeyhun Karimov 
> wrote:
> >
> > > Hi Hong,
> > >
> > > Thanks for driving the release.
> > > +1 (non-binding)
> > >
> > > - Verified gpg signature
> > > - Reviewed the PR
> > > - Verified sha512
> > > - Checked github release tag
> > > - Checked that the source code does not contain binaries
> > >
> > > Regards,
> > > Jeyhun
> > >
> > > On Sat, Jun 8, 2024 at 1:52 PM weijie guo 
> > > wrote:
> > >
> > > > Thanks Hong!
> > > >
> > > > +1(binding)
> > > >
> > > > - Verified gpg signature
> > > > - Verified sha512 hash
> > > > - Checked gh release tag
> > > > - Checked all artifacts deployed to maven repo
> > > > - Ran a simple wordcount job on local standalone cluster
> > > > - Compiled from source code with JDK 1.8.0_291.
> > > >
> > > > Best regards,
> > > >
> > > > Weijie
> > > >
> > > >
> > > > Xiqian YU  于2024年6月7日周五 18:23写道：
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > >
> > > > >   *   Checked download links & release tags
> > > > >   *   Verified that package checksums matched
> > > > >   *   Compiled Flink from source code with JDK 8 / 11
> > > > >   *   Ran E2e data integration test jobs on local cluster
> > > > >
> > > > > Regards,
> > > > > yux
> > > > >
> > > > > De : Rui Fan <1996fan...@gmail.com>
> > > > > Date : vendredi, 7 juin 2024 à 17:14
> > > > > À : dev@flink.apache.org 
> > > > > Objet : Re: [VOTE] Release 1.19.1, release candidate #1
> > > > > +1(binding)
> > > > >
> > > > > - Reviewed the flink-web PR (Left some comments)
> > > > > - Checked Github release tag
> > > > > - Verified signatures
> > > > > - Verified sha512 (hashsums)
> > > > > - The source archives do not contain any binaries
> > > > > - Build the source with Maven 3 and java8 (Checked the license as
> > well)
> > > > > - Start the cluster locally with jdk8, and run the
> > StateMachineExample
> > > > job,
> > > > > it works fine.
> > > > >
> > > > > Best,
> > > > > Rui
> > > > >
> > > > > On Thu, Jun 6, 2024 at 11:39 PM Hong Liang 
> wrote:
> > > > >
> > > > > > Hi everyone,
> > > > > > Please review and vote on the release candidate #1 for the flink
> > > > v1.19.1,
> > > > > > as follows:
> > > > > > [ ] +1, Approve the release
> > > > > > [ ] -1, Do not approve the release (please provide specific
> > comments)
> > > > > >
> > > > > >
> > > > > > The complete staging area is available for your review, which
> > > includes:
> > > > > > * JIRA release notes [1],
> > > > > > * the official Apache source release and binary convenience
> > releases
> > > to
> > > > > be
> > > > > > deployed to dist.apache.org [2], which are signed with the key
> > with
> > > > > > fingerprint B78A5EA1 [3],
> > > > > > * all artifacts to be deployed to the Maven Central Repository
> [4],
> > > > > > * source code tag "release-1.19.1-rc1" [5],
> > > > > > * website pull request listing the new release and adding
> > > announcement
> > > > > blog
> > > > > > post [6].
> > > > > >
> > > > > > The vote will be open for at least 72 hours. It is adopted by
> > > majority
> > > > > > approval, with at least 3 PMC affirmative votes.
> > > > > >
> > > > > > Thanks,
> > > > > > Hong
> > > > > >
> > > > > > [1]
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12354399
> > > > > > [2]
> https://dist.apache.org/repos/dist/dev/flink/flink-1.19.1-rc1/
> > > > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > > > [4]
> > > > > >
> > > >
> > https://repository.apache.org/content/repositories/orgapacheflink-1736/
> > > > > > [5]
> > https://github.com/apache/flink/releases/tag/release-1.19.1-rc1
> > > > > > [6] https://github.com/apache/flink-web/pull/745
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

[Xintong Song]

+1 (binding)

Best,

Xintong



On Fri, Jun 7, 2024 at 4:03 PM Yuxin Tan  wrote:

> Hi everyone,
>
> Thanks for all the feedback about the FLIP-459 Support Flink
> hybrid shuffle integration with Apache Celeborn[1].
> The discussion thread is here [2].
>
> I'd like to start a vote for it. The vote will be open for at least
> 72 hours unless there is an objection or insufficient votes.
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-459%3A+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn
> [2] https://lists.apache.org/thread/gy7sm7qrf7yrv1rl5f4vtk5fo463ts33
>
> Best,
> Yuxin
>

Re: [DISCUSS] FLIP-462: Support Custom Data Distribution for Input Stream of Lookup Join

[Xintong Song]

+1 for this proposal.

This FLIP will make it possible for each lookup join parallel task to only
access and cache a subset of the data. This will significantly improve the
performance and reduce the overhead when using Paimon for the dimension
table. And it's general enough to also be leveraged by other connectors.

Best,

Xintong



On Fri, Jun 7, 2024 at 10:01 AM weijie guo 
wrote:

> Hi devs,
>
>
> I'd like to start a discussion about FLIP-462[1]: Support Custom Data
> Distribution for Input Stream of Lookup Join.
>
>
> Lookup Join is an important feature in Flink, It is typically used to
> enrich a table with data that is queried from an external system.
> If we interact with the external systems for each incoming record, we
> incur significant network IO and RPC overhead.
>
> Therefore, most connectors introduce caching to reduce the per-record
> level query overhead. However, because the data distribution of Lookup
> Join's input stream is arbitrary, the cache hit rate is sometimes
> unsatisfactory.
>
>
> We want to introduce a mechanism for the connector to tell the Flink
> planner its desired input stream data distribution or partitioning
> strategy. This can significantly reduce the amount of cached data and
> improve performance of Lookup Join.
>
>
> You can find more details in this FLIP[1]. Looking forward to hearing
> from you, thanks!
>
>
> Best regards,
>
> Weijie
>
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-462+Support+Custom+Data+Distribution+for+Input+Stream+of+Lookup+Join
>

[ANNOUNCE] New Apache Flink PMC Member - Weijie Guo

[Xintong Song]

Hi everyone,

On behalf of the PMC, I'm very happy to announce that Weijie Guo has joined
the Flink PMC!

Weijie has been an active member of the Apache Flink community for many
years. He has made significant contributions in many components, including
runtime, shuffle, sdk, connectors, etc. He has driven / participated in
many FLIPs, authored and reviewed hundreds of PRs, been consistently active
on mailing lists, and also helped with release management of 1.20 and
several other bugfix releases.

Congratulations and welcome Weijie!

Best,

Xintong (on behalf of the Flink PMC)

Re: Native Kubernetes Task Managers

[Xintong Song]

I may not have understood what you mean by the naming scheme. I think the
limitation "pods in a StatefulSet are always terminated in the reverse
order as they are created" comes from Kubernetes and has nothing to do with
the naming scheme.

Best,

Xintong



On Mon, Jun 3, 2024 at 1:13 PM Alexis Sarda-Espinosa <
sarda.espin...@gmail.com> wrote:

> Hi Xintong,
>
> After experimenting a bit, I came to roughly the same conclusion: cleanup
> is what's more or less incompatible if Kubernetes manages the pods. Then it
> might be better to just allow using a more stable pod naming scheme that
> doesn't depend on the attempt number and thus produces more stable task
> manager metrics. I'll explore that.
>
> Regards,
> Alexis.
>
> On Mon, 3 Jun 2024, 03:35 Xintong Song,  wrote:
>
> > I think the reason we didn't choose StatefulSet when introducing the
> Native
> > K8s Deployment is that, IIRC, we want Flink's ResourceManager to have
> full
> > control of the individual pod lifecycles.
> >
> > E.g.,
> > - Pods in a StatefulSet are always terminated in the reverse order as
> they
> > are created. This prevents us from releasing a specific idle TM that is
> not
> > necessarily created lastly.
> > - If a pod is unexpectedly terminated, Flink's ResourceManager should
> > decide whether to restart it or not according to the job status.
> > (Technically, the same issue as above, that we may want pods to be
> > terminated / deleted in a different order.)
> >
> > There might be some other reasons. I just cannot recall all the details.
> >
> > As for determining whether a pod is OOM killed, I think Flink does print
> > diagnostics for terminated pods in JM logs, i.e. the `exitCode`, `reason`
> > and `message` of the `Terminated` container state. In our production, it
> > shows "(exitCode=137, reason=OOMKilled, message=null)". However, since
> the
> > diagnostics are from K8s, I'm not 100% sure whether this behavior is same
> > for all K8s versions,.
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Sun, Jun 2, 2024 at 7:35 PM Alexis Sarda-Espinosa <
> > sarda.espin...@gmail.com> wrote:
> >
> > > Hi devs,
> > >
> > > Some time ago I asked about the way Task Manager pods are handled by
> the
> > > native Kubernetes driver [1]. I have now looked a bit through the
> source
> > > code and I think it could be possible to deploy TMs with a stateful
> set,
> > > which could allow tracking OOM kills as I mentioned in my original
> email,
> > > and could also make it easier to track metrics and create alerts, since
> > the
> > > labels wouldn't change as much.
> > >
> > > One challenge is probably the new elastic scaling features [2], since
> the
> > > driver would have to differentiate between new pod requests due to a TM
> > > terminating, and a request due to scaling. I'm also not sure where
> > > downscaling requests are currently handled.
> > >
> > > I would be interested in taking a look at this and seeing if I can get
> > > something working. I think it would be possible to make it configurable
> > in
> > > a way that maintains backwards compatibility. Would it be ok if I
> enter a
> > > Jira ticket and try it out?
> > >
> > > Regards,
> > > Alexis.
> > >
> > > [1] https://lists.apache.org/thread/jysgdldv8swgf4fhqwqochgf6hq0qs52
> > > [2]
> > >
> > >
> >
> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/deployment/elastic_scaling/
> > >
> >
>

Re: Native Kubernetes Task Managers

[Xintong Song]

I think the reason we didn't choose StatefulSet when introducing the Native
K8s Deployment is that, IIRC, we want Flink's ResourceManager to have full
control of the individual pod lifecycles.

E.g.,
- Pods in a StatefulSet are always terminated in the reverse order as they
are created. This prevents us from releasing a specific idle TM that is not
necessarily created lastly.
- If a pod is unexpectedly terminated, Flink's ResourceManager should
decide whether to restart it or not according to the job status.
(Technically, the same issue as above, that we may want pods to be
terminated / deleted in a different order.)

There might be some other reasons. I just cannot recall all the details.

As for determining whether a pod is OOM killed, I think Flink does print
diagnostics for terminated pods in JM logs, i.e. the `exitCode`, `reason`
and `message` of the `Terminated` container state. In our production, it
shows "(exitCode=137, reason=OOMKilled, message=null)". However, since the
diagnostics are from K8s, I'm not 100% sure whether this behavior is same
for all K8s versions,.

Best,

Xintong



On Sun, Jun 2, 2024 at 7:35 PM Alexis Sarda-Espinosa <
sarda.espin...@gmail.com> wrote:

> Hi devs,
>
> Some time ago I asked about the way Task Manager pods are handled by the
> native Kubernetes driver [1]. I have now looked a bit through the source
> code and I think it could be possible to deploy TMs with a stateful set,
> which could allow tracking OOM kills as I mentioned in my original email,
> and could also make it easier to track metrics and create alerts, since the
> labels wouldn't change as much.
>
> One challenge is probably the new elastic scaling features [2], since the
> driver would have to differentiate between new pod requests due to a TM
> terminating, and a request due to scaling. I'm also not sure where
> downscaling requests are currently handled.
>
> I would be interested in taking a look at this and seeing if I can get
> something working. I think it would be possible to make it configurable in
> a way that maintains backwards compatibility. Would it be ok if I enter a
> Jira ticket and try it out?
>
> Regards,
> Alexis.
>
> [1] https://lists.apache.org/thread/jysgdldv8swgf4fhqwqochgf6hq0qs52
> [2]
>
> https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/deployment/elastic_scaling/
>

Re: [DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

[Xintong Song]

+1 for this proposal.

We have been prototyping this feature internally at Alibaba for a couple of
months. Yuxin, I think we can also publish the prototype codes so the
community can better understand and help with it.

Best,

Xintong



On Tue, May 28, 2024 at 8:34 PM Yuxin Tan  wrote:

> Hi all,
>
> I would like to start a discussion on FLIP-459 Support Flink hybrid shuffle
> integration with
> Apache Celeborn[1]. Flink hybrid shuffle supports transitions between
> memory, disk, and
> remote storage to improve performance and job stability. Concurrently,
> Apache Celeborn
> provides a stable, performant, scalable remote shuffle service. This
> integration proposal is to
> harness the benefits from both hybrid shuffle and Celeborn simultaneously.
>
> Note that this proposal has two parts.
> 1. The Flink-side modifications are in FLIP-459[1].
> 2. The Celeborn-side changes are in CIP-6[2].
>
> Looking forward to everyone's feedback and suggestions. Thank you!
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-459%3A+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn
> [2]
>
> https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn
>
> Best,
> Yuxin
>

Re: [DISCUSS] Flink 1.19.1 release

[Xintong Song]

+1 for the 1.19.1 release and +1 for Hong as the release manager.

Hong, if you need help on PMC-related steps and Danny is not available,
please feel free to reach out to me on slack.

Best,

Xintong



On Mon, May 27, 2024 at 9:44 AM Leonard Xu  wrote:

> +1 for the 1.19.1 release and +1 for Hong as release manager.
>
> Best,
> Leonard
>
> > 2024年5月25日 上午2:55，Danny Cranmer  写道：
> >
> > +1 for the 1.19.1 release and +1 for Hong as release manager.
>
>

Re: [DISCUSS] Proposing an LTS Release for the 1.x Line

[Xintong Song]

>
> I think our starting point should be "We don't backport features, unless
> discussed and agreed on the Dev mailing list".


This makes perfect sense. In general, I think we want to encourage users to
upgrade in order to use the new features. Alternatively, users can also
choose to maintain their own custom patches based on the LTS version. I
personally don't see why backporting new features to old releases is
necessary. In case of exceptions, a mailing list discussion is always a
good direction to go.


Best,

Xintong



On Fri, May 24, 2024 at 9:35 PM David Radley 
wrote:

> Hi Martjin and Alex,
> I agree with your summaries, it will be interesting to see what requests
> there might be for back ports.
>  Kind regards, David.
>
>
>
>
> From: Alexander Fedulov 
> Date: Friday, 24 May 2024 at 14:07
> To: dev@flink.apache.org 
> Subject: [EXTERNAL] Re: [DISCUSS] Proposing an LTS Release for the 1.x Line
> @David
> > I agree with Martijn that we only put features into version 2. Back
> porting to v1 should not be business as usual for features, only for
> security and stability changes.
> Yep, this choice is explicitly reflected in the FLIP [1]
>
> @Martijn
> >  I think our starting point should be "We don't backport features, unless
> discussed and agreed on the Dev mailing list".
> I agree - the baseline is that we do not do that. Only if a very compelling
> argument is made and the community reaches consensus, exceptions could
> potentially be made, but we should try to avoid them.
>
> [1] https://cwiki.apache.org/confluence/x/BApeEg
>
> Best,
> Alex
>
> On Fri, 24 May 2024 at 14:38, Martijn Visser 
> wrote:
>
> > Hi David,
> >
> > > If there is a maintainer willing to merge backported features to v1, as
> > it is important to some part of the community, this should be allowed, as
> > different parts of the community have different priorities and timelines,
> >
> > I don't think this is a good idea. Backporting a feature can cause issues
> > in other components that might be outside the span of expertise of the
> > maintainer that backported said feature, causing the overall stability to
> > be degraded. I think our starting point should be "We don't backport
> > features, unless discussed and agreed on the Dev mailing list". That
> still
> > opens up the ability to backport features but makes it clear where the
> bar
> > lies.
> >
> > Best regards,
> >
> > Martijn
> >
> > On Fri, May 24, 2024 at 11:21 AM David Radley 
> > wrote:
> >
> > > Hi,
> > > I agree with Martijn that we only put features into version 2. Back
> > > porting to v1 should not be business as usual for features, only for
> > > security and stability changes.
> > >
> > > If there is a maintainer willing to merge backported features to v1, as
> > it
> > > is important to some part of the community, this should be allowed, as
> > > different parts of the community have different priorities and
> timelines,
> > >  Kind regards, David.
> > >
> > >
> > > From: Alexander Fedulov 
> > > Date: Thursday, 23 May 2024 at 18:50
> > > To: dev@flink.apache.org 
> > > Subject: [EXTERNAL] Re: [DISCUSS] Proposing an LTS Release for the 1.x
> > Line
> > > Good point, Xintong, I incorporated this item into the FLIP.
> > >
> > > Best,
> > > Alex
> > >
> > > On Wed, 22 May 2024 at 10:37, Xintong Song 
> > wrote:
> > >
> > > > Thanks, Alex.
> > > >
> > > > I see one task that needs to be done once the FLIP is approved, which
> > I'd
> > > > suggest to also mention in the: To explain the LTS policy to users on
> > > > website / documentation (because FLIP is developer-facing) before /
> > upon
> > > > releasing 1.20.
> > > >
> > > > Other than that, the FLIP LGTM.
> > > >
> > > > Best,
> > > >
> > > > Xintong
> > > >
> > > >
> > > >
> > > > On Tue, May 21, 2024 at 5:21 PM Alexander Fedulov <
> > > > alexander.fedu...@gmail.com> wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > let's finalize this discussion. As Martijn suggested, I summarized
> > this
> > > > > thread into a FLIP [1]. Please take a look and let me know if
> there’s
> > > > > anything important that I might have missed.
> > > &

Re: [DISCUSS] Proposing an LTS Release for the 1.x Line

[Xintong Song]

logies.
> The
> > > > > > > > > > > source-/sink-function was used as a concrete example to
> > > > > discuss the
> > > > > > > > > > pattern
> > > > > > > > > > > before we decided to offer LTS. The intention was not
> to
> > > hijack
> > > > > > > this
> > > > > > > > > > thread
> > > > > > > > > > > to discuss how to deprecate them.
> > > > > > > > > > >
> > > > > > > > > > > We all wish that the only thing users need to migrate
> > from
> > > > > Flink
> > > > > > > 1.x
> > > > > > > > to
> > > > > > > > > > 2.0
> > > > > > > > > > > is some code changes in their repos and we all wish
> users
> > > will
> > > > > > > > migrate,
> > > > > > > > > > if
> > > > > > > > > > > LTS has long enough support time. But the question I
> > tried
> > > to
> > > > > > > discuss
> > > > > > > > > is
> > > > > > > > > > > not the wish but the "How?". We might be able to toss
> the
> > > high
> > > > > > > > > migration
> > > > > > > > > > > effort aside(we shouldn't), since it is theoretically
> > still
> > > > > doable
> > > > > > > if
> > > > > > > > > > users
> > > > > > > > > > > have long enough time, even if the effort is extremely
> > > high.
> > > > > > > Another
> > > > > > > > > > > concern is that if "function regressions" is allowed in
> > > 2.0,
> > > > > i.e.
> > > > > > > if
> > > > > > > > > 2.0
> > > > > > > > > > > has a lack of functionalities or bugs compared to 1.x,
> > > there
> > > > > will
> > > > > > > be
> > > > > > > > no
> > > > > > > > > > way
> > > > > > > > > > > for users to do the migration regardless of whether we
> > > > > encourage
> > > > > > > them
> > > > > > > > > to
> > > > > > > > > > > migrate or they haven been given enough time(how long
> is
> > > > > enough?)
> > > > > > > > > because
> > > > > > > > > > > LTS has been offered. How could we help users and avoid
> > > this
> > > > > > > > happening?
> > > > > > > > > > >
> > > > > > > > > > > Best regards,
> > > > > > > > > > > Jing
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Jul 25, 2023 at 6:57 PM Konstantin Knauf <
> > > > > > > kna...@apache.org>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Jing,
> > > > > > > > > > > >
> > > > > > > > > > > > let's not overindex on the Source-/SinkFunction
> > > discussion in
> > > > > > > this
> > > > > > > > > > > thread.
> > > > > > > > > > > >
> > > > > > > > > > > > We will generally drop/break a lot of APIs in Flink
> > 2.0.
> > > So,
> > > > > > > > > naturally
> > > > > > > > > > > > users will need to make more changes to their code in
> > > order
> > > > > to
> > > > > > > > > migrate
> > > > > > > > > > > from
> > > > > > > > > > > > 1.x to Flink 2.0. In order to give them more time to
> do
> > > > > this, we
> > > > > > > > > > support
> > > > > > > > > > > > the last Flink 1.x release for a longer time with bug
> > 

Re: [DISCUSS] Merge "flink run" and "flink run-application" in Flink 2.0

[Xintong Song]

AFAIK, the main purpose of having `run-application` was to make sure
the user is aware that application mode is used, which executes the main
method of the user program in JM rather than in client. This was important
at the time application mode was first introduced, but maybe not that
important anymore, given that per-job mode is deprecated and likely removed
in 2.0. Therefore, +1 for the proposal.

Best,

Xintong



On Thu, May 16, 2024 at 11:35 PM Ferenc Csaky 
wrote:

> Hello devs,
>
> I saw quite some examples when customers were confused about run​, and run-
> application​ in the Flink CLI and I was wondering about the necessity of
> deploying
> Application Mode (AM) jobs with a different command, than Session and
> Per-Job mode jobs.
>
> I can see a point that YarnDeploymentTarget​ [1] and
> KubernetesDeploymentTarget​
> [2] are part of their own maven modules and not known in flink-clients​,
> so the
> deployment mode validations are happening during cluster deployment in
> their specific
> ClusterDescriptor​ implementation [3]. Although these are implementation
> details that
> IMO should not define user-facing APIs.
>
> The command line setup is the same for both run​ and run-application​, so
> I think there
> is a quite simple way to achieve a unified flink run​ experience, but I
> might missed
> something so I would appreciate any inputs on this topic.
>
> Based on my assumptions I think it would be possible to deprecate the run-
> application​ in Flink 1.20 and remove it completely in Flink 2.0. I
> already put together a
> PoC [4], and I was able to deploy AM jobs like this:
>
> flink run --target kubernetes-application ...​
>
> If others also agree with this, I would be happy to open a FLIP. WDYT?
>
> Thanks,
> Ferenc
>
> [1]
> https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/configuration/YarnDeploymentTarget.java
> [2]
> https://github.com/apache/flink/blob/master/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/configuration/KubernetesDeploymentTarget.java
> [3]
> https://github.com/apache/flink/blob/48e5a39c9558083afa7589d2d8b054b625f61ee9/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/KubernetesClusterDescriptor.java#L206
> [4]
> https://github.com/ferenc-csaky/flink/commit/40b3e1b998c7a4273eaaff71d9162c9f1ee039c0

Re: [VOTE] FLIP-450: Improve Runtime Configuration for Flink 2.0

[Xintong Song]

+1 (binding)

Best,

Xintong



On Wed, May 15, 2024 at 6:53 PM weijie guo 
wrote:

> +1(binding)
>
> Best regards,
>
> Weijie
>
>
> Rui Fan <1996fan...@gmail.com> 于2024年5月15日周三 17:50写道：
>
> > +1(binding)
> >
> > Best,
> > Rui
> >
> > On Wed, May 15, 2024 at 5:01 PM Xuannan Su 
> wrote:
> >
> > > Hi everyone,
> > >
> > > Thanks for all the feedback about the FLIP-450: Improve Runtime
> > > Configuration for Flink 2.0 [1] [2].
> > >
> > > I'd like to start a vote for it. The vote will be open for at least 72
> > > hours(excluding weekends,until MAY 20, 12:00AM GMT) unless there is an
> > > objection or an insufficient number of votes.
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-450%3A+Improve+Runtime+Configuration+for+Flink+2.0
> > > [2] https://lists.apache.org/thread/20mkzd31607posls793hxy7mht40xp2x
> > >
> > >
> > > Best regards,
> > > Xuannan
> > >
> >
>

[jira] [Created] (FLINK-35331) Download links for binary releases are displayed as source releases on website

[Xintong Song (Jira)]

Xintong Song created FLINK-35331:


 Summary: Download links for binary releases are displayed as 
source releases on website
 Key: FLINK-35331
 URL: https://issues.apache.org/jira/browse/FLINK-35331
 Project: Flink
  Issue Type: Bug
  Components: Project Website
Reporter: Xintong Song


Take Pre-bundled Hadoop as examples. The content for downloading are binary 
releases, while the link is displayed as "Pre-bundled Hadoop 2.x.y Source 
Release (asc, sha512)". The problem is caused by misusing 
`source_release_[url|asc_url|sha512_url]` for binary contents in the 
corresponding [yaml 
file.|https://github.com/apache/flink-web/blob/asf-site/docs/data/additional_components.yml]

There are many similar cases in the webpage.

And a relevant issues is that, some source releases are displayed as "XXX 
Source Release Source Release", due to including "Source Release" in the `name` 
field of the corresponding yaml file.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSSION] FLIP-450: Improve Runtime Configuration for Flink 2.0

[Xintong Song]

Thanks for driving this effort, Xuannan.

+1 for the proposed changes.

Just one suggestion: Some of the proposed changes involve not solely
changing the configuration options, but are bound to changing / removal of
certain features. E.g., the removal of hash-blocking shuffle and legacy
hybrid shuffle mode, and the behavior change of overdraft network buffers.
Therefore, it might be nicer to provide an implementation plan with a list
of related tasks in the FLIP. This should not block the FLIP though.

Best,

Xintong



On Thu, Apr 25, 2024 at 4:35 PM Xuannan Su  wrote:

> Hi all,
>
> I'd like to start a discussion on FLIP-450: Improve Runtime
> Configuration for Flink 2.0 [1]. As Flink moves toward 2.0, we have
> revisited all runtime configurations and identified several
> improvements to enhance user-friendliness and maintainability. In this
> FLIP, we aim to refine the runtime configuration.
>
> Looking forward to everyone's feedback and suggestions. Thank you!
>
> Best regards,
> Xuannan
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-450%3A+Improve+Runtime+Configuration+for+Flink+2.0
>

Re: [VOTE] FLIP-442: General Improvement to Configuration for Flink 2.0

[Xintong Song]

+1 (binding)

Best,

Xintong



On Wed, Apr 17, 2024 at 4:47 PM Zakelly Lan  wrote:

> +1 binding
>
>
> Best,
> Zakelly
>
> On Wed, Apr 17, 2024 at 2:05 PM Rui Fan <1996fan...@gmail.com> wrote:
>
> > +1(binding)
> >
> > Best,
> > Rui
> >
> > On Wed, Apr 17, 2024 at 1:02 PM Xuannan Su 
> wrote:
> >
> > > Hi everyone,
> > >
> > > Thanks for all the feedback about the FLIP-442: General Improvement to
> > > Configuration for Flink 2.0 [1] [2].
> > >
> > > I'd like to start a vote for it. The vote will be open for at least 72
> > > hours(excluding weekends,until APR 22, 12:00AM GMT) unless there is an
> > > objection or an insufficient number of votes.
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-442%3A+General+Improvement+to+Configuration+for+Flink+2.0
> > > [2] https://lists.apache.org/thread/15k0stwyoknhxvd643ctwjw3fd17pqwk
> > >
> > >
> > > Best regards,
> > > Xuannan
> > >
> >
>

Re: Re: [VOTE] FLIP-423: Disaggregated State Storage and Management (Umbrella FLIP)

[Xintong Song]

+1 (binding)

Best,

Xintong



On Fri, Mar 29, 2024 at 10:28 AM Rui Fan <1996fan...@gmail.com> wrote:

> +1(binding)
>
> Best,
> Rui
>
> On Thu, Mar 28, 2024 at 10:13 PM Jing Ge 
> wrote:
>
> > +1(binding)
> >
> > Best Regards,
> > Jing
> >
> > On Thu, Mar 28, 2024 at 1:28 PM Feifan Wang  wrote:
> >
> > > +1 (non-binding)
> > >
> > > ——
> > >
> > > Best regards,
> > >
> > > Feifan Wang
> > >
> > >
> > >
> > >
> > > At 2024-03-28 20:20:21, "Piotr Nowojski"  wrote:
> > > >+1 (binding)
> > > >
> > > >Piotrek
> > > >
> > > >czw., 28 mar 2024 o 11:43 Yuan Mei 
> napisał(a):
> > > >
> > > >> Hi devs,
> > > >>
> > > >> I'd like to start a vote on the FLIP-423: Disaggregated State
> Storage
> > > and
> > > >> Management (Umbrella FLIP) [1]. The discussion thread is here [2].
> > > >>
> > > >> The vote will be open for at least 72 hours unless there is an
> > > objection or
> > > >> insufficient votes.
> > > >>
> > > >> [1] https://cwiki.apache.org/confluence/x/R4p3EQ
> > > >> [2]
> https://lists.apache.org/thread/ct8smn6g9y0b8730z7rp9zfpnwmj8vf0
> > > >>
> > > >>
> > > >> Best,
> > > >> Yuan
> > > >>
> > >
> >
>

Re: [VOTE] FLIP-425: Asynchronous Execution Model

[Xintong Song]

+1 (binding)

Best,

Xintong



On Fri, Mar 29, 2024 at 11:04 AM Yunfeng Zhou 
wrote:

> +1 (non-bindling)
>
> Best,
> Yunfeng
>
> On Wed, Mar 27, 2024 at 6:28 PM Yanfei Lei  wrote:
> >
> > Hi everyone,
> >
> > Thanks for all the feedback about the FLIP-425: Asynchronous Execution
> > Model [1]. The discussion thread is here [2].
> >
> > The vote will be open for at least 72 hours unless there is an
> > objection or insufficient votes.
> >
> > [1] https://cwiki.apache.org/confluence/x/S4p3EQ
> > [2] https://lists.apache.org/thread/wxn1j848fnfkqsnrs947wh1wmj8n8z0h
> >
> > Best regards,
> > Yanfei
>

Re: [VOTE] FLIP-424: Asynchronous State APIs

[Xintong Song]

+1 (binding)

Best,

Xintong



On Fri, Mar 29, 2024 at 12:51 PM Yuepeng Pan  wrote:

> +1(non-binding)
>
> Best,
> Yuepeng Pan
>
>
> On 2024/03/29 03:03:53 Yunfeng Zhou wrote:
> > +1 (non-binding)
> >
> > Best,
> > Yunfeng
> >
> > On Wed, Mar 27, 2024 at 6:23 PM Zakelly Lan 
> wrote:
> > >
> > > Hi devs,
> > >
> > > I'd like to start a vote on the FLIP-424: Asynchronous State APIs [1].
> The
> > > discussion thread is here [2].
> > >
> > > The vote will be open for at least 72 hours unless there is an
> objection or
> > > insufficient votes.
> > >
> > > [1] https://cwiki.apache.org/confluence/x/SYp3EQ
> > > [2] https://lists.apache.org/thread/nmd9qd0k8l94ygcfgllxms49wmtz1864
> > >
> > >
> > > Best,
> > > Zakelly
> >
>

Re: [DISCUSS] Planning Flink 1.20

[Xintong Song]

Thanks for joining, Ufuk & Robert. +1

Best,

Xintong



On Mon, Mar 25, 2024 at 7:54 PM Leonard Xu  wrote:

> Wow, happy to see Ufuk and Robert join the release managers group.
>
> +1 for the release manager candidates(Weijie, Rui Fan,Ufuk and Robert)
> from my side.
>
>
> Best,
> Leonard
>
>
>
> > 2024年3月25日 下午6:09，Robert Metzger  写道：
> >
> > Hi, thanks for starting the discussion.
> >
> > +1 for the proposed timeline and the three proposed release managers.
> >
> > I'm happy to join the release managers group as well, as a backup for
> Ufuk
> > (unless there are objections about the number of release managers)
> >
> > On Mon, Mar 25, 2024 at 11:04 AM Ufuk Celebi  wrote:
> >
> >> Hey all,
> >>
> >> I'd like to join the release managers for 1.20 as well. I'm looking
> >> forward to getting more actively involved again.
> >>
> >> Cheers,
> >>
> >> Ufuk
> >>
> >> On Sun, Mar 24, 2024, at 11:27 AM, Ahmed Hamdy wrote:
> >>> +1 for the proposed timeline and release managers.
> >>> Best Regards
> >>> Ahmed Hamdy
> >>>
> >>>
> >>> On Fri, 22 Mar 2024 at 07:41, Xintong Song 
> >> wrote:
> >>>
> >>>> +1 for the proposed timeline and Weijie & Rui as the release managers.
> >>>>
> >>>> I think it would be welcomed if another 1-2 volunteers join as the
> >> release
> >>>> managers, but that's not a must. We used to have only 1-2 release
> >> managers
> >>>> for each release,
> >>>>
> >>>> Best,
> >>>>
> >>>> Xintong
> >>>>
> >>>>
> >>>>
> >>>> On Fri, Mar 22, 2024 at 2:55 PM Jark Wu  wrote:
> >>>>
> >>>>> Thanks for kicking this off.
> >>>>>
> >>>>> +1 for the volunteered release managers (Weijie Guo, Rui Fan) and the
> >>>>> targeting date (feature freeze: June 15).
> >>>>>
> >>>>> Best,
> >>>>> Jark
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Fri, 22 Mar 2024 at 14:00, Rui Fan <1996fan...@gmail.com> wrote:
> >>>>>
> >>>>>> Thanks Leonard for this feedback and help!
> >>>>>>
> >>>>>> Best,
> >>>>>> Rui
> >>>>>>
> >>>>>> On Fri, Mar 22, 2024 at 12:36 PM weijie guo <
> >> guoweijieres...@gmail.com
> >>>>>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Thanks Leonard!
> >>>>>>>
> >>>>>>>> I'd like to help you if you need some help like permissions
> >> from
> >>>> PMC
> >>>>>>> side, please feel free to ping me.
> >>>>>>>
> >>>>>>> Nice to know. It'll help a lot!
> >>>>>>>
> >>>>>>> Best regards,
> >>>>>>>
> >>>>>>> Weijie
> >>>>>>>
> >>>>>>>
> >>>>>>> Leonard Xu  于2024年3月22日周五 12:09写道：
> >>>>>>>
> >>>>>>>> +1 for the proposed release managers (Weijie Guo, Rui Fan),
> >> both the
> >>>>> two
> >>>>>>>> candidates are pretty active committers thus I believe they
> >> know the
> >>>>>>>> community development process well. The recent releases have
> >> four
> >>>>>> release
> >>>>>>>> managers, and I am also looking forward to having other
> >> volunteers
> >>>>>>>> join the management of Flink 1.20.
> >>>>>>>>
> >>>>>>>> +1 for targeting date (feature freeze: June 15, 2024),
> >> referring to
> >>>>> the
> >>>>>>>> release cycle of recent versions, release cycle of 4 months
> >> makes
> >>>>> sense
> >>>>>> to
> >>>>>>>> me.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> I'd like to help you if you need some help like per

Re: [DISCUSS] Planning Flink 1.20

[Xintong Song]

+1 for the proposed timeline and Weijie & Rui as the release managers.

I think it would be welcomed if another 1-2 volunteers join as the release
managers, but that's not a must. We used to have only 1-2 release managers
for each release,

Best,

Xintong



On Fri, Mar 22, 2024 at 2:55 PM Jark Wu  wrote:

> Thanks for kicking this off.
>
> +1 for the volunteered release managers (Weijie Guo, Rui Fan) and the
> targeting date (feature freeze: June 15).
>
> Best,
> Jark
>
>
>
>
>
> On Fri, 22 Mar 2024 at 14:00, Rui Fan <1996fan...@gmail.com> wrote:
>
> > Thanks Leonard for this feedback and help!
> >
> > Best,
> > Rui
> >
> > On Fri, Mar 22, 2024 at 12:36 PM weijie guo 
> > wrote:
> >
> > > Thanks Leonard!
> > >
> > > > I'd like to help you if you need some help like permissions from PMC
> > > side, please feel free to ping me.
> > >
> > > Nice to know. It'll help a lot!
> > >
> > > Best regards,
> > >
> > > Weijie
> > >
> > >
> > > Leonard Xu  于2024年3月22日周五 12:09写道：
> > >
> > >> +1 for the proposed release managers (Weijie Guo, Rui Fan), both the
> two
> > >> candidates are pretty active committers thus I believe they know the
> > >> community development process well. The recent releases have four
> > release
> > >> managers, and I am also looking forward to having other volunteers
> > >>  join the management of Flink 1.20.
> > >>
> > >> +1 for targeting date (feature freeze: June 15, 2024), referring to
> the
> > >> release cycle of recent versions, release cycle of 4 months makes
> sense
> > to
> > >> me.
> > >>
> > >>
> > >> I'd like to help you if you need some help like permissions from PMC
> > >> side, please feel free to ping me.
> > >>
> > >> Best,
> > >> Leonard
> > >>
> > >>
> > >> > 2024年3月19日 下午5:35，Rui Fan <1996fan...@gmail.com> 写道：
> > >> >
> > >> > Hi Weijie,
> > >> >
> > >> > Thanks for kicking off 1.20! I'd like to join you and participate in
> > the
> > >> > 1.20 release.
> > >> >
> > >> > Best,
> > >> > Rui
> > >> >
> > >> > On Tue, Mar 19, 2024 at 5:30 PM weijie guo <
> guoweijieres...@gmail.com
> > >
> > >> > wrote:
> > >> >
> > >> >> Hi everyone,
> > >> >>
> > >> >> With the release announcement of Flink 1.19, it's a good time to
> kick
> > >> off
> > >> >> discussion of the next release 1.20.
> > >> >>
> > >> >>
> > >> >> - Release managers
> > >> >>
> > >> >>
> > >> >> I'd like to volunteer as one of the release managers this time. It
> > has
> > >> been
> > >> >> good practice to have a team of release managers from different
> > >> >> backgrounds, so please raise you hand if you'd like to volunteer
> and
> > >> get
> > >> >> involved.
> > >> >>
> > >> >>
> > >> >>
> > >> >> - Timeline
> > >> >>
> > >> >>
> > >> >> Flink 1.19 has been released. With a target release cycle of 4
> > months,
> > >> >> we propose a feature freeze date of *June 15, 2024*.
> > >> >>
> > >> >>
> > >> >>
> > >> >> - Collecting features
> > >> >>
> > >> >>
> > >> >> As usual, we've created a wiki page[1] for collecting new features
> in
> > >> 1.20.
> > >> >>
> > >> >>
> > >> >> In addition, we already have a number of FLIPs that have been voted
> > or
> > >> are
> > >> >> in the process, including pre-works for version 2.0.
> > >> >>
> > >> >>
> > >> >> In the meantime, the release management team will be finalized in
> the
> > >> next
> > >> >> few days, and we'll continue to create Jira Boards and Sync
> meetings
> > >> >> to make it easy
> > >> >> for everyone to get an overview and track progress.
> > >> >>
> > >> >>
> > >> >>
> > >> >> Best regards,
> > >> >>
> > >> >> Weijie
> > >> >>
> > >> >>
> > >> >>
> > >> >> [1] https://cwiki.apache.org/confluence/display/FLINK/1.20+Release
> > >> >>
> > >>
> > >>
> >
>

Re: [VOTE] FLIP-433: State Access on DataStream API V2

[Xintong Song]

+1 (binding)

Best,

Xintong



On Wed, Mar 20, 2024 at 8:30 PM weijie guo 
wrote:

> Hi everyone,
>
>
> Thanks for all the feedback about the FLIP-433: State Access on
> DataStream API V2 [1]. The discussion thread is here [2].
>
>
> The vote will be open for at least 72 hours unless there is an
> objection or insufficient votes.
>
>
> Best regards,
>
> Weijie
>
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-433%3A+State+Access+on+DataStream+API+V2
>
> [2] https://lists.apache.org/thread/8ghzqkvt99p4k7hjqxzwhqny7zb7xrwo
>

Re: [DISCUSS] FLIP-423 ～FLIP-428: Introduce Disaggregated State Storage and Management in Flink 2.0

[Xintong Song]

Thanks for the quick response. Sounds good to me.

Best,

Xintong



On Tue, Mar 19, 2024 at 1:03 PM Zakelly Lan  wrote:

> Hi Xintong,
>
> Thanks for sharing your thoughts!
>
> 1. Regarding Record-ordered and State-ordered of processElement.
> >
> > I understand that while State-ordered likely provides better performance,
> > Record-ordered is sometimes required for correctness. The question is how
> > should a user choose between these two modes? My concern is that such a
> > decision may require users to have in-depth knowledge about the Flink
> > internals, and may lead to correctness issues if State-ordered is chosen
> > improperly.
> >
> > I'd suggest not to expose such a knob, at least in the first version.
> That
> > means always use Record-ordered for custom operators / UDFs, and keep
> > State-ordered for internal usages (built-in operators) only.
> >
>
> Indeed, users may not be able to choose the mode properly. I agree to keep
> such options for internal use.
>
>
> 2. Regarding Strictly-ordered and Out-of-order of Watermarks.
> >
> > I'm not entirely sure about Strictly-ordered being the default, or even
> > being supported. From my understanding, a Watermark(T) only suggests that
> > all records with event time before T has arrived, and it has nothing to
> do
> > with whether records with event time after T has arrived or not. From
> that
> > perspective, preventing certain records from arriving before a Watermark
> is
> > never supported. I also cannot come up with any use case where
> > Strictly-ordered is necessary. This implies the same issue as 1): how
> does
> > the user choose between the two modes?
> >
> > I'd suggest not expose the knob to users and only support Out-of-order,
> > until we see a concrete use case that Strictly-ordered is needed.
> >
>
> The semantics of watermarks do not define the sequence between a watermark
> and subsequent records. For the most part, this is inconsequential, except
> it may affect some current users who have previously relied on the implicit
> assumption of an ordered execution. I'd be fine with initially supporting
> only out-of-order processing. We may consider exposing the
> 'Strictly-ordered' mode once we encounter a concrete use case that
> necessitates it.
>
>
> My philosophies behind not exposing the two config options are:
> > - There are already too many options in Flink that barely know how to use
> > them. I think Flink should try as much as possible to decide its own
> > behavior, rather than throwing all the decisions to the users.
> > - It's much harder to take back knobs than to introduce them. Therefore,
> > options should be introduced only if concrete use cases are identified.
> >
>
> I agree to keep minimal configurable items especially for the MVP. Given
> that we have the opportunity to refine the functionality before the
> framework transitions from @Experimental to @PublicEvolving, it makes sense
> to refrain from presenting user-facing options until we have ensured
> their necessity.
>
>
> Best,
> Zakelly
>
> On Tue, Mar 19, 2024 at 12:06 PM Xintong Song 
> wrote:
>
> > Sorry for joining the discussion late.
> >
> > I have two questions about FLIP-425.
> >
> > 1. Regarding Record-ordered and State-ordered of processElement.
> >
> > I understand that while State-ordered likely provides better performance,
> > Record-ordered is sometimes required for correctness. The question is how
> > should a user choose between these two modes? My concern is that such a
> > decision may require users to have in-depth knowledge about the Flink
> > internals, and may lead to correctness issues if State-ordered is chosen
> > improperly.
> >
> > I'd suggest not to expose such a knob, at least in the first version.
> That
> > means always use Record-ordered for custom operators / UDFs, and keep
> > State-ordered for internal usages (built-in operators) only.
> >
> > 2. Regarding Strictly-ordered and Out-of-order of Watermarks.
> >
> > I'm not entirely sure about Strictly-ordered being the default, or even
> > being supported. From my understanding, a Watermark(T) only suggests that
> > all records with event time before T has arrived, and it has nothing to
> do
> > with whether records with event time after T has arrived or not. From
> that
> > perspective, preventing certain records from arriving before a Watermark
> is
> > never supported. I also cannot come up with any use case where
> > Strictly-ordered is necessary. This impl

Re: [DISCUSS] FLIP-423 ～FLIP-428: Introduce Disaggregated State Storage and Management in Flink 2.0

[Xintong Song]

Sorry for joining the discussion late.

I have two questions about FLIP-425.

1. Regarding Record-ordered and State-ordered of processElement.

I understand that while State-ordered likely provides better performance,
Record-ordered is sometimes required for correctness. The question is how
should a user choose between these two modes? My concern is that such a
decision may require users to have in-depth knowledge about the Flink
internals, and may lead to correctness issues if State-ordered is chosen
improperly.

I'd suggest not to expose such a knob, at least in the first version. That
means always use Record-ordered for custom operators / UDFs, and keep
State-ordered for internal usages (built-in operators) only.

2. Regarding Strictly-ordered and Out-of-order of Watermarks.

I'm not entirely sure about Strictly-ordered being the default, or even
being supported. From my understanding, a Watermark(T) only suggests that
all records with event time before T has arrived, and it has nothing to do
with whether records with event time after T has arrived or not. From that
perspective, preventing certain records from arriving before a Watermark is
never supported. I also cannot come up with any use case where
Strictly-ordered is necessary. This implies the same issue as 1): how does
the user choose between the two modes?

I'd suggest not expose the knob to users and only support Out-of-order,
until we see a concrete use case that Strictly-ordered is needed.


My philosophies behind not exposing the two config options are:
- There are already too many options in Flink that barely know how to use
them. I think Flink should try as much as possible to decide its own
behavior, rather than throwing all the decisions to the users.
- It's much harder to take back knobs than to introduce them. Therefore,
options should be introduced only if concrete use cases are identified.

WDYT?

Best,

Xintong



On Fri, Mar 8, 2024 at 2:45 AM Jing Ge  wrote:

> +1 for Gyula's suggestion. I just finished FLIP-423 which introduced the
> intention of the big change and high level architecture. Great content btw!
> The only public interface change for this FLIP is one new config to use
> ForSt. It makes sense to have one dedicated discussion thread for each
> concrete system design.
>
> @Zakelly The links in your mail do not work except the last one, because
> the FLIP-xxx has been included into the url like
> https://lists.apache.org/thread/nmd9qd0k8l94ygcfgllxms49wmtz1864FLIP-425.
>
> NIT fix:
>
> FLIP-424: https://lists.apache.org/thread/nmd9qd0k8l94ygcfgllxms49wmtz1864
>
> FLIP-425: https://lists.apache.org/thread/wxn1j848fnfkqsnrs947wh1wmj8n8z0h
>
> FLIP-426: https://lists.apache.org/thread/bt931focfl9971cwq194trmf3pkdsxrf
>
> FLIP-427: https://lists.apache.org/thread/vktfzqvb7t4rltg7fdlsyd9sfdmrc4ft
>
> FLIP-428: https://lists.apache.org/thread/vr8f91p715ct4lop6b3nr0fh4z5p312b
>
> Best regards,
> Jing
>
>
>
>
> On Thu, Mar 7, 2024 at 10:14 AM Zakelly Lan  wrote:
>
> > Hi everyone,
> >
> > Thank you all for a lively discussion here, and it is a good time to move
> > forward to more detailed discussions. Thus we open several threads for
> > sub-FLIPs:
> >
> > FLIP-424:
> https://lists.apache.org/thread/nmd9qd0k8l94ygcfgllxms49wmtz1864
> > FLIP-425
> > <
> https://lists.apache.org/thread/nmd9qd0k8l94ygcfgllxms49wmtz1864FLIP-425>:
> > https://lists.apache.org/thread/wxn1j848fnfkqsnrs947wh1wmj8n8z0h
> > FLIP-426
> > <
> https://lists.apache.org/thread/wxn1j848fnfkqsnrs947wh1wmj8n8z0hFLIP-426>:
> > https://lists.apache.org/thread/bt931focfl9971cwq194trmf3pkdsxrf
> > FLIP-427
> > <
> https://lists.apache.org/thread/bt931focfl9971cwq194trmf3pkdsxrfFLIP-427>:
> > https://lists.apache.org/thread/vktfzqvb7t4rltg7fdlsyd9sfdmrc4ft
> > FLIP-428
> > <
> https://lists.apache.org/thread/vktfzqvb7t4rltg7fdlsyd9sfdmrc4ftFLIP-428>:
> > https://lists.apache.org/thread/vr8f91p715ct4lop6b3nr0fh4z5p312b
> >
> > If you want to talk about the overall architecture, roadmap, milestones
> or
> > something related with multiple FLIPs, please post it here. Otherwise you
> > can discuss some details in separate mails. Let's try to avoid repeated
> > discussion in different threads. I will sync important messages here if
> > there are any in the above threads.
> >
> > And reply to @Jeyhun: We will ensure the content between those FLIPs is
> > consistent.
> >
> >
> > Best,
> > Zakelly
> >
> > On Thu, Mar 7, 2024 at 2:16 PM Yuan Mei  wrote:
> >
> > > I have been a bit busy these few weeks and sorry for responding late.
> > >
> > > The original thinking of keeping discussion within one thread is for
> > easier
> > > tracking and avoid for repeated discussion in different threads.
> > >
> > > For details, It might be good to start in different threads if needed.
> > >
> > > We will think of a way to better organize the discussion.
> > >
> > > Best
> > > Yuan
> > >
> > >
> > > On Thu, Mar 7, 2024 at 4:38 AM Jeyhun Karimov 
> > > wrote:
> > >
> > > > Hi,
> > > >
> > 

Re: [ANNOUNCE] Apache Flink 1.19.0 released

[Xintong Song]

Congratulations~!

Best,

Xintong



On Mon, Mar 18, 2024 at 7:02 PM Feng Jin  wrote:

> Congratulations!
>
> Best,
> Feng
>
> On Mon, Mar 18, 2024 at 6:18 PM Yuepeng Pan  wrote:
>
> > Congratulations!
> >
> >
> > Thanks to release managers and everyone involved.
> >
> >
> >
> >
> > Best,
> > Yuepeng Pan
> >
> >
> >
> >
> >
> >
> >
> >
> > At 2024-03-18 18:09:45, "Yubin Li"  wrote:
> > >Congratulations!
> > >
> > >Thanks to release managers and everyone involved.
> > >
> > >On Mon, Mar 18, 2024 at 5:55 PM Hangxiang Yu 
> wrote:
> > >>
> > >> Congratulations!
> > >> Thanks release managers and all involved!
> > >>
> > >> On Mon, Mar 18, 2024 at 5:23 PM Hang Ruan 
> > wrote:
> > >>
> > >> > Congratulations!
> > >> >
> > >> > Best,
> > >> > Hang
> > >> >
> > >> > Paul Lam  于2024年3月18日周一 17:18写道：
> > >> >
> > >> > > Congrats! Thanks to everyone involved!
> > >> > >
> > >> > > Best,
> > >> > > Paul Lam
> > >> > >
> > >> > > > 2024年3月18日 16:37，Samrat Deb  写道：
> > >> > > >
> > >> > > > Congratulations !
> > >> > > >
> > >> > > > On Mon, 18 Mar 2024 at 2:07 PM, Jingsong Li <
> > jingsongl...@gmail.com>
> > >> > > wrote:
> > >> > > >
> > >> > > >> Congratulations!
> > >> > > >>
> > >> > > >> On Mon, Mar 18, 2024 at 4:30 PM Rui Fan <1996fan...@gmail.com>
> > wrote:
> > >> > > >>>
> > >> > > >>> Congratulations, thanks for the great work!
> > >> > > >>>
> > >> > > >>> Best,
> > >> > > >>> Rui
> > >> > > >>>
> > >> > > >>> On Mon, Mar 18, 2024 at 4:26 PM Lincoln Lee <
> > lincoln.8...@gmail.com>
> > >> > > >> wrote:
> > >> > > 
> > >> > >  The Apache Flink community is very happy to announce the
> > release of
> > >> > > >> Apache Flink 1.19.0, which is the fisrt release for the Apache
> > Flink
> > >> > > 1.19
> > >> > > >> series.
> > >> > > 
> > >> > >  Apache Flink® is an open-source stream processing framework
> for
> > >> > > >> distributed, high-performing, always-available, and accurate
> data
> > >> > > streaming
> > >> > > >> applications.
> > >> > > 
> > >> > >  The release is available for download at:
> > >> > >  https://flink.apache.org/downloads.html
> > >> > > 
> > >> > >  Please check out the release blog post for an overview of the
> > >> > > >> improvements for this bugfix release:
> > >> > > 
> > >> > > >>
> > >> > >
> > >> >
> >
> https://flink.apache.org/2024/03/18/announcing-the-release-of-apache-flink-1.19/
> > >> > > 
> > >> > >  The full release notes are available in Jira:
> > >> > > 
> > >> > > >>
> > >> > >
> > >> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12353282
> > >> > > 
> > >> > >  We would like to thank all contributors of the Apache Flink
> > >> > community
> > >> > > >> who made this release possible!
> > >> > > 
> > >> > > 
> > >> > >  Best,
> > >> > >  Yun, Jing, Martijn and Lincoln
> > >> > > >>
> > >> > >
> > >> > >
> > >> >
> > >>
> > >>
> > >> --
> > >> Best,
> > >> Hangxiang.
> >
>

Re: [VOTE] Release 1.19.0, release candidate #2

[Xintong Song]

+1 (binding)

- verified signature and checksum
- verified that source distribution does not contain binaries
- built from source and played with example jobs, everything looks fine
- reviewed the release announcement PR

Best,

Xintong



On Thu, Mar 7, 2024 at 6:01 PM Lincoln Lee  wrote:

> Hi everyone,
>
> Please review and vote on the release candidate #2 for the version 1.19.0,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> The complete staging area is available for your review, which includes:
>
> * JIRA release notes [1], and the pull request adding release note for
> users [2]
> * the official Apache source release and binary convenience releases to be
> deployed to dist.apache.org [3], which are signed with the key with
> fingerprint E57D30ABEE75CA06  [4],
> * all artifacts to be deployed to the Maven Central Repository [5],
> * source code tag "release-1.19.0-rc2" [6],
> * website pull request listing the new release and adding announcement blog
> post [7].
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> [1]
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12353282
> [2] https://github.com/apache/flink/pull/24394
> [3] https://dist.apache.org/repos/dist/dev/flink/flink-1.19.0-rc2/
> [4] https://dist.apache.org/repos/dist/release/flink/KEYS
> [5] https://repository.apache.org/content/repositories/orgapacheflink-1709
> [6] https://github.com/apache/flink/releases/tag/release-1.19.0-rc2
> [7] https://github.com/apache/flink-web/pull/721
>
>
> Best,
> Yun, Jing, Martijn and Lincoln
>

Re: [DISCUSS] Move CheckpointingMode to flink-core

[Xintong Song]

Personally, I'd be in favor of option 2. And based on the fact that
migrating from the deprecated CheckpointingMode to the new one takes barely
any effort (simply re-import the class), I'd be fine with removing the
deprecated class in 2.0.

But I'd also be fine with the other options.

Either way, agree that we should not block 1.19 on this.

Best,

Xintong



On Wed, Feb 28, 2024 at 6:16 PM Junrui Lee  wrote:

> Hi Zakelly,
>
> +1 for option 1. I prefer to minimize unnecessary additional development
> and discussions due to internal code relocations and to avoid imposing
> migration costs on users.
>
> Best regards,
> Junrui
>
> Zakelly Lan  于2024年2月28日周三 14:46写道：
>
> > Hi Lincoln,
> >
> > Given that we have finished the testing for 1.19, I agree it is better
> not
> > merge this into 1.19. Thanks for RMs' attention!
> >
> > Hi Chesney and Junrui,
> >
> > Thanks for your advice. My original intention is to move the class as
> well
> > as change the package to make it clean. But it involves much more effort.
> > Here are several options we have:
> >
> >1. Move CheckpointingMode to flink-core and keep the same package. No
> >more deprecation and API changes. But it will leave a
> >'org.apache.flink.streaming.api' package in flink-core.
> >2. Introduce new CheckpointingMode in package
> >'org.apache.flink.core.execution' and deprecate the old one. Deprecate
> > the
> >corresponding getter/setter of 'CheckpointConfig' and introduce new
> ones
> >with a similar but different name (e.g. set/getCheckpointMode). We
> will
> >discuss the removal of those deprecation later in 2.x.
> >3. Based on 1, move CheckpointingMode to package
> >'org.apache.flink.core.execution' in 2.0. This is a breaking change
> that
> >needs more discussion.
> >
> > Both ways work. I'm slightly inclined to option 1, or option 3 if we all
> > agree, since the new getter/setter may also bring in confusions thus we
> > cannot make the API purely clean. WDYT?
> >
> >
> > Best,
> > Zakelly
> >
> > On Wed, Feb 28, 2024 at 10:14 AM Junrui Lee  wrote:
> >
> > > Hi Zakelly,
> > >
> > > I agree with Chesnay's response. I would suggest that during the
> process
> > of
> > > moving CheckpointingMode from the flink-streaming-java module to the
> > > flink-core module, we should keep the package name unchanged. This
> > approach
> > > would be completely transparent to users. In fact, this practice should
> > be
> > > applicable to many of our desired moves from flink-streaming-java to
> > > higher-level modules, such as flink-runtime and flink-core.
> > >
> > > Best,
> > > Junrui
> > >
> > > Chesnay Schepler  于2024年2月28日周三 05:18写道：
> > >
> > > > Moving classes (== keep the same package) to a module higher up in
> the
> > > > dependency tree should not be a breaking change and can imo be done
> > > > anytime without any risk to users.
> > > >
> > > > On 27/02/2024 17:01, Lincoln Lee wrote:
> > > > > Hi Zakelly,
> > > > >
> > > > > Thanks for letting us 1.19 RMs know about this!
> > > > >
> > > > > This change has been discussed during today's release sync meeting,
> > we
> > > > > suggest not merge it into 1.19.
> > > > > We can continue discussing the removal in 2.x separately.
> > > > >
> > > > > Best,
> > > > > Lincoln Lee
> > > > >
> > > > >
> > > > > Hangxiang Yu  于2024年2月27日周二 11:28写道：
> > > > >
> > > > >> Hi, Zakelly.
> > > > >> Thanks for driving this.
> > > > >> Moving this class to flink-core makes sense to me which could make
> > the
> > > > code
> > > > >> path and configs clearer.
> > > > >> It's marked as @Public from 1.0 and 1.20 should be the next
> > long-term
> > > > >> version, so 1.19 should have been a suitable version to do it.
> > > > >> And also look forward to thoughts of other developers/RMs since
> 1.19
> > > is
> > > > >> currently under a feature freeze status.
> > > > >>
> > > > >> On Mon, Feb 26, 2024 at 6:42 PM Zakelly Lan <
> zakelly@gmail.com>
> > > > wrote:
> > > > >>
> > > > >>> Hi devs,
> > > > >>>
> > > > >>> When working on the FLIP-406[1], I realized that moving all
> options
> > > of
> > > > >>> ExecutionCheckpointingOptions(flink-streaming-java) to
> > > > >>> CheckpointingOptions(flink-core) depends on relocating the
> > > > >>> enum CheckpointingMode(flink-streaming-java) to flink-core
> module.
> > > > >> However,
> > > > >>> the CheckpointingMode is annotated as @Public and used by
> > datastream
> > > > api
> > > > >>> like 'CheckpointConfig#setCheckpointingMode'. So I'd like to
> start
> > a
> > > > >>> discussion on moving the CheckpointingMode to flink-core. It is
> in
> > a
> > > > >> little
> > > > >>> bit of a hurry if we want the old enum to be entirely removed in
> > > Flink
> > > > >> 2.x
> > > > >>> series, since the deprecation should be shipped in the upcoming
> > Flink
> > > > >> 1.19.
> > > > >>> I suggest not creating a dedicated FLIP and treating this as a
> > > sub-task
> > > > >> of
> > > > >>> FLIP-406.
> > > > >>>
> > > > >>> I prepared a m

Re: [ANNOUNCE] Flink 1.19 Cross-team testing completed & sync summary on 02/27/2024

[Xintong Song]

Matthias has already said it on the release sync, but I still want to say
it again. It's amazing how smooth the release testing goes for this
release. Great thanks to the release managers and all contributors who make
this happen.

Best,

Xintong



On Wed, Feb 28, 2024 at 10:10 AM Lincoln Lee  wrote:

> Hi devs,
>
> I'd like to share some highlights from the release sync on 02/27/2024
>
> - Cross-team testing
>
> We've finished all of the testing work[1]. Huge thanks to all contributors
> and volunteers for the effort on this!
>
> - Blockers
>
> Two api change merge requests[2][3] had been discussed, there was an
> agreement on the second pr, as it is a fix for an
> unintended behavior newly introduced in 1.19, and we need to avoid
> releasing it to users. For the 1st pr, we suggest continue the discussing
> separately.
> So we will wait for [3] done and then create the first release candidate
> 1.19.0-rc1(expecting within this week if no new blockers).
>
> - Release notes
>
> Revision to the draft version of the release note[4] has been closed, and
> the formal pr[5] has been submitted,
> also the release announcement pr will be ready later this week, please
> continue to help review before 1.19 release, thanks!
>
> - Sync meeting (https://meet.google.com/vcx-arzs-trv)
>
> We've already switched to weekly release sync, so the next release sync
> will be on Mar 5th, 2024. Feel free to join!
>
> [1] https://issues.apache.org/jira/browse/FLINK-34285
> [2] https://lists.apache.org/thread/2llhhbkcx5w7chp3d6cthoqc8kwfvw6x
> [3]
> https://github.com/apache/flink/pull/24387#pullrequestreview-1902749309
> [4]
> https://docs.google.com/document/d/1HLF4Nhvkln4zALKJdwRErCnPzufh7Z3BhhkWlk9Zh7w
> [5] https://github.com/apache/flink/pull/24394
>
> Best,
> Yun, Jing, Martijn and Lincoln
>

Re: [VOTE] FLIP-410: Config, Context and Processing Timer Service of DataStream API V2

[Xintong Song]

+1 (binding)

Best,

Xintong



On Mon, Feb 26, 2024 at 6:10 PM weijie guo 
wrote:

> Hi everyone,
>
>
> Thanks for all the feedback about the FLIP-410: Config, Context and
> Processing Timer Service of DataStream API V2 [1]. The discussion
> thread is here [2].
>
>
> The vote will be open for at least 72 hours unless there is an
> objection or insufficient votes.
>
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-410%3A++Config%2C+Context+and+Processing+Timer+Service+of+DataStream+API+V2
>
> [2] https://lists.apache.org/thread/70gf028c5gsdb9qhsgpht0chzyp9nogc
>
>
> Best regards,
>
> Weijie
>

Re: [VOTE] FLIP-409: DataStream V2 Building Blocks: DataStream, Partitioning and ProcessFunction

[Xintong Song]

+1 (binding)

Best,

Xintong



On Mon, Feb 26, 2024 at 6:09 PM weijie guo 
wrote:

> Hi everyone,
>
>
> Thanks for all the feedback about the FLIP-409: DataStream V2 Building
> Blocks: DataStream, Partitioning and ProcessFunction [1]. The
> discussion thread is here [2].
>
>
> The vote will be open for at least 72 hours unless there is an
> objection or insufficient votes.
>
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-409%3A+DataStream+V2+Building+Blocks%3A+DataStream%2C+Partitioning+and+ProcessFunction
>
> [2] https://lists.apache.org/thread/cwds0bwbgy3lfdgnlqbfhm6lfvx2qbrv
>
>
> Best regards,
>
> Weijie
>

Re: [VOTE] FLIP-408: [Umbrella] Introduce DataStream API V2

[Xintong Song]

+1 (binding)

Best,

Xintong



On Mon, Feb 26, 2024 at 6:08 PM weijie guo 
wrote:

> Hi everyone,
>
>
> Thanks for all the feedback about the FLIP-408: [Umbrella] Introduce
> DataStream API V2 [1]. The discussion thread is here [2].
>
>
> The vote will be open for at least 72 hours unless there is an
> objection or insufficient votes.
>
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-408%3A+%5BUmbrella%5D+Introduce+DataStream+API+V2
>
> [2] https://lists.apache.org/thread/w8olky9s7fo5h8fl3nj3qbym307zk2l0
>
> Best regards,
>
> Weijie
>

Re: [DISCUSS] FLIP-410: Config, Context and Processing Timer Service of DataStream API V2

[Xintong Song]

LGTM

Best,

Xintong



On Mon, Feb 19, 2024 at 10:48 AM weijie guo 
wrote:

> Hi All,
>
> Based on the discussion thread of FLIP-409, I did a synchronized update to
> this one. In simple terms, added `TwoInputBroadcastStreamProcessFunction`
> related content.
>
>
> Best regards,
>
> Weijie
>
>
> weijie guo  于2024年1月31日周三 15:00写道：
>
> > Hi Xintong,
> >
> > Thanks for the quick reply.
> >
> > > Why introduce a new `MetricManager` rather than just return
> `MetricGroup`
> > from `RuntimeContext`?
> >
> > This is to facilitate possible future extensions. But I thought it
> > through, MetricGroup itself also plays the role of a manager.
> > So I think you are right, I will add a `getMetricGroup` method directly
> in
> > `RuntimeContext`.
> >
> > Best regards,
> >
> > Weijie
> >
> >
> > Xintong Song  于2024年1月31日周三 14:02写道：
> >
> >> >
> >> > > How can users define custom metrics within the `ProcessFunction`?
> >> > Will there be a method like `getMetricGroup` available in the
> >> > `RuntimeContext`?
> >> >
> >> > I think this is a reasonable request. For extensibility, I have added
> >> the
> >> > getMetricManager instead of getMetricGroup to RuntimeContext, we can
> >> use it
> >> > to get the MetricGroup.
> >> >
> >>
> >> Why introduce a new `MetricManager` rather than just return
> `MetricGroup`
> >> from `RuntimeContext`?
> >>
> >> > Q2. The FLIP describes the interface for handling processing
> >> >  timers (ProcessingTimeManager), but it does not mention
> >> > how to delete or update an existing timer. V1 API provides TimeService
> >> > that could delete a timer. Does this mean that
> >> >  once a timer is registered, it cannot be changed?
> >> >
> >> > I think we do need to introduce a method to delete the timer, but I'm
> >> kind
> >> > of curious why we need to update the timer instead of registering a
> new
> >> > one. Anyway, I have updated the FLIP to support delete the timer.
> >> >
> >>
> >> Registering a new timer does not mean the old timer should be removed.
> >> There could be multiple timers.
> >>
> >> If we don't support deleting timers, developers can still decide to do
> >> nothing upon the timer is triggered. In that case, they will need
> >> additional logic to decide whether the timer should be skipped or not in
> >> `onProcessingTimer`. Besides, there could also be additional performance
> >> overhead in frequent calling and skipping the callback.
> >>
> >> Best,
> >>
> >> Xintong
> >>
> >>
> >>
> >> On Tue, Jan 30, 2024 at 3:26 PM weijie guo 
> >> wrote:
> >>
> >> > Hi Wencong,
> >> >
> >> > > Q1. In the "Configuration" section, it is mentioned that
> >> > configurations can be set continuously using the withXXX methods.
> >> > Are these configuration options the same as those provided by
> DataStream
> >> > V1,
> >> > or might there be different options compared to V1?
> >> >
> >> > I haven't considered options that don't exist in V1 yet, but we may
> have
> >> > some new options as we continue to develop.
> >> >
> >> > > Q2. The FLIP describes the interface for handling processing
> >> >  timers (ProcessingTimeManager), but it does not mention
> >> > how to delete or update an existing timer. V1 API provides TimeService
> >> > that could delete a timer. Does this mean that
> >> >  once a timer is registered, it cannot be changed?
> >> >
> >> > I think we do need to introduce a method to delete the timer, but I'm
> >> kind
> >> > of curious why we need to update the timer instead of registering a
> new
> >> > one. Anyway, I have updated the FLIP to support delete the timer.
> >> >
> >> >
> >> >
> >> > Best regards,
> >> >
> >> > Weijie
> >> >
> >> >
> >> > weijie guo  于2024年1月30日周二 14:35写道：
> >> >
> >> > > Hi Xuannan,
> >> > >
> >> > > > 1. +1 to only use XXXParititionStream if users only need to use
> the
> >> > > configurable PartitionStream.  If there are use cases for both,
>

Re: [DISCUSS] FLIP-409: DataStream V2 Building Blocks: DataStream, Partitioning and ProcessFunction

[Xintong Song]

Thanks for the updates. LGTM.

Best,

Xintong



On Mon, Feb 19, 2024 at 10:51 AM weijie guo 
wrote:

> Thanks for the reply, Xintong.
>
> Based on your comments, I made the following changes to this FLIP:
>
> 1. Renaming `TwoInputStreamProcessFunction` and
> `BroadcastTwoInputStreamProcessFunction` to
> `TwoInputNonBroadcastStreamProcessFunction` and
> `TwoInputBroadcastStreamProcessFunction`, respectively.
>
> 2. Making `NonPartitionedContext` extend `RuntimeContext`.
>
> > Some of these changes also affect FLIP-410. I noticed that FLIP-410 is
> also updated accordingly. It would be nice to also mention those changes in
> the FLIP-410 discussion thread.
>
> Yes, I've now mentioned those updates in the FLIP-410 discussion thread.
>
>
> Best regards,
>
> Weijie
>
>
> Xintong Song  于2024年2月5日周一 10:58写道：
>
> > Thanks for updating the FLIP, Weijie.
> >
> > I think separating the TwoInputProcessFunction according to whether the
> > input stream contains BroadcastStream makes sense.
> >
> > I have a few more comments.
> > 1. I'd suggest the names `TwoInputNonBroadcastStreamProcessFunction` and
> > `TwoInputBroadcastStreamProcessFunction` for the separated methods.
> > 2. I'd suggest making `NonPartitionedContext` extend `RuntimeContext`.
> > Otherwise, for all the functionalities that `RuntimeContext` provides, we
> > need to duplicate them for `NonPartitionedContext`.
> > 3. Some of these changes also affect FLIP-410. I noticed that FLIP-410 is
> > also updated accordingly. It would be nice to also mention those changes
> in
> > the FLIP-410 discussion thread.
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Sun, Feb 4, 2024 at 11:23 AM weijie guo 
> > wrote:
> >
> > > Hi Xuannan and Xintong,
> > >
> > > Good point! After further consideration, I feel that we should make the
> > > Broadcast + NonKeyed/Keyed process function different from the normal
> > > TwoInputProcessFunction. Because the record from the broadcast input
> > indeed
> > > correspond to all partitions, while the record from the non-broadcast
> > edge
> > > have explicit partitions.
> > >
> > > When we consider the data of broadcast input, it is only valid to do
> > > something on all the partitions at once, such as things like
> > > `applyToKeyedState`. Similarly, other operations(e.g, endOfInput) that
> do
> > > not determine the current partition should also only be allowed to
> > perform
> > > on all partitions. This FLIP has been updated.
> > >
> > > Best regards,
> > >
> > > Weijie
> > >
> > >
> > > Xintong Song  于2024年2月1日周四 11:31写道：
> > >
> > > > OK, I see your point.
> > > >
> > > > I think the demand for updating states and emitting outputs upon
> > > receiving
> > > > a broadcast record makes sense. However, the way
> > > > `KeyedBroadcastProcessFunction` supports this may not be optimal.
> E.g.,
> > > if
> > > > `Collector#collect` is called in `processBroadcastElement` but
> outside
> > of
> > > > `Context#applyToKeyedState`, the result can be undefined.
> > > >
> > > > Currently in this FLIP, a `TwoInputStreamProcessFunction` is not
> aware
> > of
> > > > which input is KeyedStream and which is BroadcastStream, which makes
> > > > supporting things like `applyToKeyedState` difficult. I think we can
> > > > provide a built-in function similar to
> `KeyedBroadcastProcessFunction`
> > on
> > > > top of `TwoInputStreamProcessFunction` to address this demand.
> > > >
> > > > WDYT?
> > > >
> > > >
> > > > Best,
> > > >
> > > > Xintong
> > > >
> > > >
> > > >
> > > > On Thu, Feb 1, 2024 at 10:41 AM Xuannan Su 
> > > wrote:
> > > >
> > > > > Hi Weijie and Xingtong,
> > > > >
> > > > > Thanks for the reply! Please see my comments below.
> > > > >
> > > > > > Does this mean if we want to support (KeyedStream,
> BroadcastStream)
> > > ->
> > > > > > (KeyedStream), we must make sure that no data can be output upon
> > > > > processing
> > > > > > records from the input BroadcastStream? That's probably a
> > reasonable
> > > > > > limitation.
> > &

Re: [ANNOUNCE] Flink 1.19 feature freeze & sync summary on 01/30/2024

[Xintong Song]

Thanks for the info.

My opinion would be to follow the process by default, and to make
exceptions only if there're good reasons. From your description, it sounds
like merging the PR in or after 1.19 doesn't really make a difference. In
that case, I'd suggest to merge it for the next release (i.e. merge it into
master after the 1.19 branch cutting).

Best,

Xintong



On Mon, Feb 5, 2024 at 12:52 PM Rui Fan <1996fan...@gmail.com> wrote:

> Thanks Xintong for the reply.
>
> They are Flink internal classes, and they are not used anymore.
> So I think they don't affect users, the benefit of removing them
> is to simplify Flink's code and reduce maintenance costs.
>
> If we just merge some user-related PRs recently, I could merge
> it after 1.19. Thank you again~
>
> Best,
> Rui
>
> On Mon, Feb 5, 2024 at 12:21 PM Xintong Song 
> wrote:
>
> > Hi Rui,
> >
> > Quick question, would there be any downside if this PR doesn't go into
> > 1.19? Or any user benefit from getting it into this release?
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Sun, Feb 4, 2024 at 10:16 AM Rui Fan <1996fan...@gmail.com> wrote:
> >
> > > Hi release managers,
> > >
> > > > The feature freeze of 1.19 has started now. That means that no new
> > > features
> > > > or improvements should now be merged into the master branch unless
> you
> > > ask
> > > > the release managers first, which has already been done for PRs, or
> > > pending
> > > > on CI to pass. Bug fixes and documentation PRs can still be merged.
> > >
> > > I'm curious whether the code cleanup could be merged?
> > > FLINK-31449[1] removed DeclarativeSlotManager related logic.
> > > Some other classes are not used anymore after FLINK-31449.
> > > FLINK-34345[2][3] will remove them.
> > >
> > > I checked these classes are not used in the master branch.
> > > And the PR[3] is reviewed for now, could I merge it now or
> > > after flink-1.19?
> > >
> > > Looking forward to your feedback, thanks~
> > >
> > > [1] https://issues.apache.org/jira/browse/FLINK-31449
> > > [2] https://issues.apache.org/jira/browse/FLINK-34345
> > > [3] https://github.com/apache/flink/pull/24257
> > >
> > > Best,
> > > Rui
> > >
> > > On Wed, Jan 31, 2024 at 5:20 PM Lincoln Lee 
> > > wrote:
> > >
> > >> Hi Matthias,
> > >>
> > >> Thanks for letting us know! After discussed with 1.19 release
> managers,
> > we
> > >> agreed to merge these pr.
> > >>
> > >> Thank you for the work on GHA workflows!
> > >>
> > >> Best,
> > >> Yun, Jing, Martijn and Lincoln
> > >>
> > >>
> > >> Matthias Pohl  于2024年1月30日周二 22:20写道：
> > >>
> > >> > Thanks for the update, Lincoln.
> > >> >
> > >> > fyi: I merged FLINK-32684 (deprecating AkkaOptions) [1] since we
> > agreed
> > >> in
> > >> > today's meeting that this change is still ok to go in.
> > >> >
> > >> > The beta version of the GitHub Actions workflows (FLIP-396 [2]) are
> > also
> > >> > finalized (see related PRs for basic CI [3], nightly master [4] and
> > >> nightly
> > >> > scheduling [5]). I'd like to merge the changes before creating the
> > >> > release-1.19 branch. That would enable us to see whether we miss
> > >> anything
> > >> > in the GHA workflows setup when creating a new release branch.
> > >> >
> > >> > The changes are limited to a few CI scripts that are also used for
> > Azure
> > >> > Pipelines (see [3]). The majority of the changes are GHA-specific
> and
> > >> > shouldn't affect the Azure Pipelines CI setup.
> > >> >
> > >> > Therefore, I'm requesting the approval from the 1.19 release
> managers
> > to
> > >> > go ahead with merging the mentioned PRs [3, 4, 5].
> > >> >
> > >> > Matthias
> > >> >
> > >> >
> > >> > [1] https://issues.apache.org/jira/browse/FLINK-32684
> > >> > [2]
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-396%3A+Trial+to+test+GitHub+Actions+as+an+alternative+for+Flink%27s+current+Azure+CI+infrastructure
> > >

Re: [ANNOUNCE] Flink 1.19 feature freeze & sync summary on 01/30/2024

[Xintong Song]

Hi Rui,

Quick question, would there be any downside if this PR doesn't go into
1.19? Or any user benefit from getting it into this release?

Best,

Xintong



On Sun, Feb 4, 2024 at 10:16 AM Rui Fan <1996fan...@gmail.com> wrote:

> Hi release managers,
>
> > The feature freeze of 1.19 has started now. That means that no new
> features
> > or improvements should now be merged into the master branch unless you
> ask
> > the release managers first, which has already been done for PRs, or
> pending
> > on CI to pass. Bug fixes and documentation PRs can still be merged.
>
> I'm curious whether the code cleanup could be merged?
> FLINK-31449[1] removed DeclarativeSlotManager related logic.
> Some other classes are not used anymore after FLINK-31449.
> FLINK-34345[2][3] will remove them.
>
> I checked these classes are not used in the master branch.
> And the PR[3] is reviewed for now, could I merge it now or
> after flink-1.19?
>
> Looking forward to your feedback, thanks~
>
> [1] https://issues.apache.org/jira/browse/FLINK-31449
> [2] https://issues.apache.org/jira/browse/FLINK-34345
> [3] https://github.com/apache/flink/pull/24257
>
> Best,
> Rui
>
> On Wed, Jan 31, 2024 at 5:20 PM Lincoln Lee 
> wrote:
>
>> Hi Matthias,
>>
>> Thanks for letting us know! After discussed with 1.19 release managers, we
>> agreed to merge these pr.
>>
>> Thank you for the work on GHA workflows!
>>
>> Best,
>> Yun, Jing, Martijn and Lincoln
>>
>>
>> Matthias Pohl  于2024年1月30日周二 22:20写道：
>>
>> > Thanks for the update, Lincoln.
>> >
>> > fyi: I merged FLINK-32684 (deprecating AkkaOptions) [1] since we agreed
>> in
>> > today's meeting that this change is still ok to go in.
>> >
>> > The beta version of the GitHub Actions workflows (FLIP-396 [2]) are also
>> > finalized (see related PRs for basic CI [3], nightly master [4] and
>> nightly
>> > scheduling [5]). I'd like to merge the changes before creating the
>> > release-1.19 branch. That would enable us to see whether we miss
>> anything
>> > in the GHA workflows setup when creating a new release branch.
>> >
>> > The changes are limited to a few CI scripts that are also used for Azure
>> > Pipelines (see [3]). The majority of the changes are GHA-specific and
>> > shouldn't affect the Azure Pipelines CI setup.
>> >
>> > Therefore, I'm requesting the approval from the 1.19 release managers to
>> > go ahead with merging the mentioned PRs [3, 4, 5].
>> >
>> > Matthias
>> >
>> >
>> > [1] https://issues.apache.org/jira/browse/FLINK-32684
>> > [2]
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-396%3A+Trial+to+test+GitHub+Actions+as+an+alternative+for+Flink%27s+current+Azure+CI+infrastructure
>> > [3] https://github.com/apache/flink/pull/23970
>> > [4] https://github.com/apache/flink/pull/23971
>> > [5] https://github.com/apache/flink/pull/23972
>> >
>> > On Tue, Jan 30, 2024 at 1:51 PM Lincoln Lee 
>> > wrote:
>> >
>> >> Hi everyone,
>> >>
>> >> (Since feature freeze and release sync are on the same day, we merged
>> the
>> >> announcement and sync summary together)
>> >>
>> >>
>> >> *- Feature freeze*
>> >> The feature freeze of 1.19 has started now. That means that no new
>> >> features
>> >> or improvements should now be merged into the master branch unless you
>> ask
>> >> the release managers first, which has already been done for PRs, or
>> >> pending
>> >> on CI to pass. Bug fixes and documentation PRs can still be merged.
>> >>
>> >>
>> >> *- Cutting release branch*
>> >> Currently we have three blocker issues[1][2][3], and will try to close
>> >> them this Friday.
>> >> We are planning to cut the release branch on next Monday (Feb 6th) if
>> no
>> >> new test instabilities,
>> >> and we'll make another announcement in the dev mailing list then.
>> >>
>> >>
>> >> *- Cross-team testing*
>> >> Release testing is expected to start next week as soon as we cut the
>> >> release branch.
>> >> As a prerequisite, please Before we start testing, please make sure
>> >> 1. Whether the feature needs a cross-team testing
>> >> 2. If yes, please the documentation completed
>> >> There's an umbrella ticket[4] for tracking the 1.19 testing, RM will
>> >> create all tickets for completed features listed on the 1.19 wiki
>> page[5]
>> >> and assign to the feature's Responsible Contributor,
>> >> also contributors are encouraged to create tickets following the steps
>> in
>> >> the umbrella ticket if there are other ones that need to be cross-team
>> >> tested.
>> >>
>> >> *- Release notes*
>> >>
>> >> All new features and behavior changes require authors to fill out the
>> >> 'Release Note' column in the JIRA(click the Edit button and pull the
>> page
>> >> to the center),
>> >> especially since 1.19 involves a lot of deprecation, which is important
>> >> for users and will be part of the release announcement.
>> >>
>> >> - *Sync meeting* (https://meet.google.com/vcx-arzs-trv)
>> >>
>> >> We've already switched to weekly release sync, so the next release sync
>> >> will be on Feb 6

Re: [DISCUSS] FLIP-409: DataStream V2 Building Blocks: DataStream, Partitioning and ProcessFunction

[Xintong Song]

Thanks for updating the FLIP, Weijie.

I think separating the TwoInputProcessFunction according to whether the
input stream contains BroadcastStream makes sense.

I have a few more comments.
1. I'd suggest the names `TwoInputNonBroadcastStreamProcessFunction` and
`TwoInputBroadcastStreamProcessFunction` for the separated methods.
2. I'd suggest making `NonPartitionedContext` extend `RuntimeContext`.
Otherwise, for all the functionalities that `RuntimeContext` provides, we
need to duplicate them for `NonPartitionedContext`.
3. Some of these changes also affect FLIP-410. I noticed that FLIP-410 is
also updated accordingly. It would be nice to also mention those changes in
the FLIP-410 discussion thread.

Best,

Xintong



On Sun, Feb 4, 2024 at 11:23 AM weijie guo 
wrote:

> Hi Xuannan and Xintong,
>
> Good point! After further consideration, I feel that we should make the
> Broadcast + NonKeyed/Keyed process function different from the normal
> TwoInputProcessFunction. Because the record from the broadcast input indeed
> correspond to all partitions, while the record from the non-broadcast edge
> have explicit partitions.
>
> When we consider the data of broadcast input, it is only valid to do
> something on all the partitions at once, such as things like
> `applyToKeyedState`. Similarly, other operations(e.g, endOfInput) that do
> not determine the current partition should also only be allowed to perform
> on all partitions. This FLIP has been updated.
>
> Best regards,
>
> Weijie
>
>
> Xintong Song  于2024年2月1日周四 11:31写道：
>
> > OK, I see your point.
> >
> > I think the demand for updating states and emitting outputs upon
> receiving
> > a broadcast record makes sense. However, the way
> > `KeyedBroadcastProcessFunction` supports this may not be optimal. E.g.,
> if
> > `Collector#collect` is called in `processBroadcastElement` but outside of
> > `Context#applyToKeyedState`, the result can be undefined.
> >
> > Currently in this FLIP, a `TwoInputStreamProcessFunction` is not aware of
> > which input is KeyedStream and which is BroadcastStream, which makes
> > supporting things like `applyToKeyedState` difficult. I think we can
> > provide a built-in function similar to `KeyedBroadcastProcessFunction` on
> > top of `TwoInputStreamProcessFunction` to address this demand.
> >
> > WDYT?
> >
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Thu, Feb 1, 2024 at 10:41 AM Xuannan Su 
> wrote:
> >
> > > Hi Weijie and Xingtong,
> > >
> > > Thanks for the reply! Please see my comments below.
> > >
> > > > Does this mean if we want to support (KeyedStream, BroadcastStream)
> ->
> > > > (KeyedStream), we must make sure that no data can be output upon
> > > processing
> > > > records from the input BroadcastStream? That's probably a reasonable
> > > > limitation.
> > >
> > > I don't think that the requirement for supporting (KeyedStream,
> > > BroadcastStream) -> (KeyedStream) is that no data can be output upon
> > > processing the BroadcastStream. For instance, in the current
> > > `KeyedBroadcastProcessFunction`, we use Context#applyToKeyedState to
> > > produce output results, which can be keyed in the same manner as the
> > > keyed input stream, upon processing data from the BroadcastStream.
> > > Therefore, I believe it only requires that the user must ensure that
> > > the output is keyed in the same way as the input, in this case, the
> > > same way as the keyed input stream. I think this requirement is
> > > consistent with that of (KeyedStream, KeyedStream) -> (KeyedStream).
> > > Thus, I believe that supporting (KeyedStream, BroadcastStream) ->
> > > (KeyedStream) will not introduce complexity for the users. WDYT?
> > >
> > > Best regards,
> > > Xuannan
> > >
> > >
> > > On Tue, Jan 30, 2024 at 3:12 PM weijie guo 
> > > wrote:
> > > >
> > > > Hi Xintong,
> > > >
> > > > Thanks for your reply.
> > > >
> > > > > Does this mean if we want to support (KeyedStream, BroadcastStream)
> > ->
> > > > (KeyedStream), we must make sure that no data can be output upon
> > > processing
> > > > records from the input BroadcastStream? That's probably a reasonable
> > > > limitation.
> > > >
> > > > I think so, this is the restriction that has to be imposed in order
> to
> > > > avoid re-partition(i.e. shuffle).
> > >

Re: [DISCUSS] FLIP-409: DataStream V2 Building Blocks: DataStream, Partitioning and ProcessFunction

[Xintong Song]

OK, I see your point.

I think the demand for updating states and emitting outputs upon receiving
a broadcast record makes sense. However, the way
`KeyedBroadcastProcessFunction` supports this may not be optimal. E.g., if
`Collector#collect` is called in `processBroadcastElement` but outside of
`Context#applyToKeyedState`, the result can be undefined.

Currently in this FLIP, a `TwoInputStreamProcessFunction` is not aware of
which input is KeyedStream and which is BroadcastStream, which makes
supporting things like `applyToKeyedState` difficult. I think we can
provide a built-in function similar to `KeyedBroadcastProcessFunction` on
top of `TwoInputStreamProcessFunction` to address this demand.

WDYT?


Best,

Xintong



On Thu, Feb 1, 2024 at 10:41 AM Xuannan Su  wrote:

> Hi Weijie and Xingtong,
>
> Thanks for the reply! Please see my comments below.
>
> > Does this mean if we want to support (KeyedStream, BroadcastStream) ->
> > (KeyedStream), we must make sure that no data can be output upon
> processing
> > records from the input BroadcastStream? That's probably a reasonable
> > limitation.
>
> I don't think that the requirement for supporting (KeyedStream,
> BroadcastStream) -> (KeyedStream) is that no data can be output upon
> processing the BroadcastStream. For instance, in the current
> `KeyedBroadcastProcessFunction`, we use Context#applyToKeyedState to
> produce output results, which can be keyed in the same manner as the
> keyed input stream, upon processing data from the BroadcastStream.
> Therefore, I believe it only requires that the user must ensure that
> the output is keyed in the same way as the input, in this case, the
> same way as the keyed input stream. I think this requirement is
> consistent with that of (KeyedStream, KeyedStream) -> (KeyedStream).
> Thus, I believe that supporting (KeyedStream, BroadcastStream) ->
> (KeyedStream) will not introduce complexity for the users. WDYT?
>
> Best regards,
> Xuannan
>
>
> On Tue, Jan 30, 2024 at 3:12 PM weijie guo 
> wrote:
> >
> > Hi Xintong,
> >
> > Thanks for your reply.
> >
> > > Does this mean if we want to support (KeyedStream, BroadcastStream) ->
> > (KeyedStream), we must make sure that no data can be output upon
> processing
> > records from the input BroadcastStream? That's probably a reasonable
> > limitation.
> >
> > I think so, this is the restriction that has to be imposed in order to
> > avoid re-partition(i.e. shuffle).
> > If one just want to get a keyed-stream and don't care about the data
> > distribution, then explicit KeyBy partitioning works as expected.
> >
> > > The problem is would this limitation be too implicit for the users to
> > understand.
> >
> > Since we can't check for this limitation at compile time, if we were to
> add
> > support for this case, we would have to introduce additional runtime
> checks
> > to ensure program correctness. For now, I'm inclined not to support it,
> as
> > it's hard for users to understand this restriction unless we have
> something
> > better. And we can always add it later if we do realize there's a strong
> > demand for it.
> >
> > > 1. I'd suggest renaming the method with timestamp to something like
> > `collectAndOverwriteTimestamp`. That might help users understand that
> they
> > don't always need to call this method, unless they explicitly want to
> > overwrite the timestamp.
> >
> > Make sense, I have updated this FLIP toward this new method name.
> >
> > > 2. While this method provides a way to set timestamps, how would users
> > read
> > timestamps from the records?
> >
> > Ah, good point. I will introduce a new method to get the timestamp of the
> > current record in RuntimeContext.
> >
> >
> > Best regards,
> >
> > Weijie
> >
> >
> > Xintong Song  于2024年1月30日周二 14:04写道：
> >
> > > Just trying to understand.
> > >
> > > > Is there a particular reason we do not support a
> > > > `TwoInputProcessFunction` to combine a KeyedStream with a
> > > > BroadcastStream to result in a KeyedStream? There seems to be a valid
> > > > use case where a KeyedStream is enriched with a BroadcastStream and
> > > > returns a Stream that is partitioned in the same way.
> > >
> > >
> > > > The key point here is that if the returned stream is a KeyedStream,
> we
> > > > require that the partition of  input and output be the same. As for
> the
> > > > data on the broadcast edge, it will be

Re: [DISCUSS] FLIP-410: Config, Context and Processing Timer Service of DataStream API V2

[Xintong Song]

>
> > How can users define custom metrics within the `ProcessFunction`?
> Will there be a method like `getMetricGroup` available in the
> `RuntimeContext`?
>
> I think this is a reasonable request. For extensibility, I have added the
> getMetricManager instead of getMetricGroup to RuntimeContext, we can use it
> to get the MetricGroup.
>

Why introduce a new `MetricManager` rather than just return `MetricGroup`
from `RuntimeContext`?

> Q2. The FLIP describes the interface for handling processing
>  timers (ProcessingTimeManager), but it does not mention
> how to delete or update an existing timer. V1 API provides TimeService
> that could delete a timer. Does this mean that
>  once a timer is registered, it cannot be changed?
>
> I think we do need to introduce a method to delete the timer, but I'm kind
> of curious why we need to update the timer instead of registering a new
> one. Anyway, I have updated the FLIP to support delete the timer.
>

Registering a new timer does not mean the old timer should be removed.
There could be multiple timers.

If we don't support deleting timers, developers can still decide to do
nothing upon the timer is triggered. In that case, they will need
additional logic to decide whether the timer should be skipped or not in
`onProcessingTimer`. Besides, there could also be additional performance
overhead in frequent calling and skipping the callback.

Best,

Xintong



On Tue, Jan 30, 2024 at 3:26 PM weijie guo 
wrote:

> Hi Wencong,
>
> > Q1. In the "Configuration" section, it is mentioned that
> configurations can be set continuously using the withXXX methods.
> Are these configuration options the same as those provided by DataStream
> V1,
> or might there be different options compared to V1?
>
> I haven't considered options that don't exist in V1 yet, but we may have
> some new options as we continue to develop.
>
> > Q2. The FLIP describes the interface for handling processing
>  timers (ProcessingTimeManager), but it does not mention
> how to delete or update an existing timer. V1 API provides TimeService
> that could delete a timer. Does this mean that
>  once a timer is registered, it cannot be changed?
>
> I think we do need to introduce a method to delete the timer, but I'm kind
> of curious why we need to update the timer instead of registering a new
> one. Anyway, I have updated the FLIP to support delete the timer.
>
>
>
> Best regards,
>
> Weijie
>
>
> weijie guo  于2024年1月30日周二 14:35写道：
>
> > Hi Xuannan,
> >
> > > 1. +1 to only use XXXParititionStream if users only need to use the
> > configurable PartitionStream.  If there are use cases for both,
> > perhaps we could use `ProcessConfigurableNonKeyedPartitionStream` or
> > `ConfigurableNonKeyedPartitionStream` for simplicity.
> >
> > As for why we need both, you can refer to my reply to Yunfeng's first
> > question. As for the name, I can accept
> > ProcessConfigurableNonKeyedPartitionStream or keep the status quo. But I
> > don't want to change it to ConfigurableNonKeyedPartitionStream, the
> reason
> > is the same, because the configuration is applied to the Process rather
> > than the Stream.
> >
> > > Should we allow users to set custom configurations through the
> > `ProcessConfigurable` interface and access these configurations in the
> > `ProcessFunction` via `RuntimeContext`? I believe it would be useful
> > for process function developers to be able to define custom
> > configurations.
> >
> > If I understand you correctly, you want to set custom properties for
> > processing. The current configurations are mostly for the runtime engine,
> > such as determining the underlying operator 's parallelism and SSG. But
> I'm
> > not aware of the need to pass in a custom value(independent of the
> > framework itself) and then get it at runtime from RuntimeContext. Could
> > you give some examples?
> >
> > > How can users define custom metrics within the `ProcessFunction`?
> > Will there be a method like `getMetricGroup` available in the
> > `RuntimeContext`?
> >
> > I think this is a reasonable request. For extensibility, I have added the
> > getMetricManager instead of getMetricGroup to RuntimeContext, we can use
> > it to get the MetricGroup.
> >
> >
> > Best regards,
> >
> > Weijie
> >
> >
> > weijie guo  于2024年1月30日周二 13:45写道：
> >
> >> Thanks Yunfeng,
> >>
> >> Let me try to answer your question :)
> >>
> >> > 1. Would it be better to have all XXXPartitionStream classes implement
> >> ProcessConfigurable, instead of defining both XXXPartitionStream and
> >> ProcessConfigurableAndXXXPartitionStream? I wonder whether users would
> >> need to operate on a non-configurable PartitionStream.
> >>
> >> I thought about this for a while and decided to separate DataStream from
> >> ProcessConfigurable. At the core of this is that streams and c
> >> onfigurations are completely orthogonal concepts, and configuration is
> >> only responsible for the `Process`, not the `Stream`. This is why only
> >> the `process/connectA

Re: [VOTE] FLIP-331: Support EndOfStreamTrigger and isOutputOnlyAfterEndOfStream operator attribute to optimize task deployment

[Xintong Song]

+1

Best,

Xintong



On Wed, Jan 31, 2024 at 11:41 AM Xuannan Su  wrote:

> Hi everyone,
>
> Thanks for all the feedback about the FLIP-331: Support
> EndOfStreamTrigger and isOutputOnlyAfterEndOfStream operator attribute
> to optimize task deployment [1] [2].
>
> I'd like to start a vote for it. The vote will be open for at least 72
> hours(excluding weekends,until Feb 5, 12:00AM GMT) unless there is an
> objection or an insufficient number of votes.
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-331%3A+Support+EndOfStreamTrigger+and+isOutputOnlyAfterEndOfStream+operator+attribute+to+optimize+task+deployment
> [2] https://lists.apache.org/thread/qq39rmg3f23ysx5m094s4c4cq0m4tdj5
>
>
> Best,
> Xuannan
>

Re: [DISCUSS] FLIP-409: DataStream V2 Building Blocks: DataStream, Partitioning and ProcessFunction

[Xintong Song]

Just trying to understand.

> Is there a particular reason we do not support a
> `TwoInputProcessFunction` to combine a KeyedStream with a
> BroadcastStream to result in a KeyedStream? There seems to be a valid
> use case where a KeyedStream is enriched with a BroadcastStream and
> returns a Stream that is partitioned in the same way.


> The key point here is that if the returned stream is a KeyedStream, we
> require that the partition of  input and output be the same. As for the
> data on the broadcast edge, it will be broadcast to all parallelism, we
> cannot keep the data partition consistent. For example, if a specific
> record is sent to both SubTask1 and SubTask2, after processing, the
> partition index calculated by the new KeySelector is `1`, then the data
> distribution of SubTask2 has obviously changed.


Does this mean if we want to support (KeyedStream, BroadcastStream) ->
(KeyedStream), we must make sure that no data can be output upon processing
records from the input BroadcastStream? That's probably a reasonable
limitation. The problem is would this limitation be too implicit for the
users to understand.


> I noticed that there are two `collect` methods in the Collector,
>
> one with a timestamp and one without. Could you elaborate on the
> differences between them? Additionally, in what use case would one use
> the method that includes the timestamp?
>
>
> That's a good question, and it's mostly used with time-related operators
> such as Window. First, we want to give the process function the ability to
> reset timestamps, which makes it more flexible than the original
> API. Second, we don't want to take the timestamp extraction
> operator/function as a base primitive, it's more like a high-level
> extension. Therefore, the framework must provide this functionality.
>
>
1. I'd suggest renaming the method with timestamp to something like
`collectAndOverwriteTimestamp`. That might help users understand that they
don't always need to call this method, unless they explicitly want to
overwrite the timestamp.

2. While this method provides a way to set timestamps, how would users read
timestamps from the records?


Best,

Xintong



On Tue, Jan 30, 2024 at 12:45 PM weijie guo 
wrote:

> Hi Xuannan,
>
> Thank you for your attention.
>
> > In the partitioning section, it says that "broadcast can only be
> used as a side-input of other Inputs." Could you clarify what is meant
> by "side-input"? If I understand correctly, it refer to one of the
> inputs of the `TwoInputStreamProcessFunction`. If that is the case,
> the term "side-input" may not be accurate.
>
> Yes, you got it right! I have rewrote this sentence to avoid
> misunderstanding.
>
> > Is there a particular reason we do not support a
> `TwoInputProcessFunction` to combine a KeyedStream with a
> BroadcastStream to result in a KeyedStream? There seems to be a valid
> use case where a KeyedStream is enriched with a BroadcastStream and
> returns a Stream that is partitioned in the same way.
>
> The key point here is that if the returned stream is a KeyedStream, we
> require that the partition of  input and output be the same. As for the
> data on the broadcast edge, it will be broadcast to all parallelism, we
> cannot keep the data partition consistent. For example, if a specific
> record is sent to both SubTask1 and SubTask2, after processing, the
> partition index calculated by the new KeySelector is `1`, then the data
> distribution of SubTask2 has obviously changed.
>
> > 3. There appears to be a typo in the example code. The
> `SingleStreamProcessFunction` should probably be
> `OneInputStreamProcessFunction`.
>
> Yes, good catch. I have updated this FLIP.
>
> > 4. How do we set the global configuration for the
> ExecutionEnvironment? Currently, we have the
> StreamExecutionEnvironment.getExecutionEnvironment(Configuration)
> method to provide the global configuration in the API.
>
> This is because we don't want to allow set config programmatically in the
> new API, everything best comes from configuration files. However, this may
> be too ideal, and the specific details need to be considered and discussed
> in more detail, and I propose to devote a new sub-FLIP to this issue later.
> We can easily provide the `getExecutionEnvironment(Configuration)` or
> `withConfiguration(Configuration)` method later.
>
> > I noticed that there are two `collect` methods in the Collector,
> one with a timestamp and one without. Could you elaborate on the
> differences between them? Additionally, in what use case would one use
> the method that includes the timestamp?
>
> That's a good question, and it's mostly used with time-related operators
> such as Window. First, we want to give the process function the ability to
> reset timestamps, which makes it more flexible than the original
> API. Second, we don't want to take the timestamp extraction
> operator/function as a base primitive, it's more like a high-level
> extension. Therefore, the framework m

Re: [DISCUSS] FLIP-408: [Umbrella] Introduce DataStream API V2

[Xintong Song]

Thanks for working on this, Weijie.

The design flaws of the current DataStream API (i.e., V1) have been a pain
for a long time. It's great to see efforts going on trying to resolve them.

Significant changes to such an important and comprehensive set of public
APIs deserves caution. From that perspective, the ideas of introducing a
new set of APIs that gradually replace the current one, splitting the
introducing of the new APIs into many separate FLIPs, and making
intermediate APIs @Experiemental until all of them are completed make
great sense to me.

Besides, the ideas of generalized watermark, execution hints sound quite
interesting. Looking forward to more detailed discussions in the
corresponding sub-FLIPs.

+1 for the roadmap.

Best,

Xintong



On Tue, Jan 30, 2024 at 11:00 AM weijie guo 
wrote:

> Hi Wencong:
>
> > The Processing TimerService is currently
> defined as one of the basic primitives, partly because it's understood that
> you have to choose between processing time and event time.
> The other part of the reason is that it needs to work based on the task's
> mailbox thread model to avoid concurrency issues. Could you clarify the
> second
> part of the reason?
>
> Since the processing logic of the operators takes place in the mailbox
> thread, the processing timer's callback function must also be executed in
> the mailbox to ensure thread safety.
> If we do not define the Processing TimerService as primitive, there is no
> way for the user to dispatch custom logic to the mailbox thread.
>
>
> Best regards,
>
> Weijie
>
>
> Xuannan Su  于2024年1月29日周一 17:12写道：
>
> > Hi Weijie,
> >
> > Thanks for driving the work! There are indeed many pain points in the
> > current DataStream API, which are challenging to resolve with its
> > existing design. It is a great opportunity to propose a new DataStream
> > API that tackles these issues. I like the way we've divided the FLIP
> > into multiple sub-FLIPs; the roadmap is clear and comprehensible. +1
> > for the umbrella FLIP. I am eager to see the sub-FLIPs!
> >
> > Best regards,
> > Xuannan
> >
> >
> >
> >
> > On Wed, Jan 24, 2024 at 8:55 PM Wencong Liu 
> wrote:
> > >
> > > Hi Weijie,
> > >
> > >
> > > Thank you for the effort you've put into the DataStream API ! By
> > reorganizing and
> > > redesigning the DataStream API, as well as addressing some of the
> > unreasonable
> > > designs within it, we can enhance the efficiency of job development for
> > developers.
> > > It also allows developers to design more flexible Flink jobs to meet
> > business requirements.
> > >
> > >
> > > I have conducted a comprehensive review of the DataStream API design in
> > versions
> > > 1.18 and 1.19. I found quite a few functional defects in the DataStream
> > API, such as the
> > > lack of corresponding APIs in batch processing scenarios. In the
> > upcoming 1.20 version,
> > > I will further improve the DataStream API in batch computing scenarios.
> > >
> > >
> > > The issues existing in the old DataStream API (which can be referred to
> > as V1) can be
> > > addressed from a design perspective in the initial version of V2. I
> hope
> > to also have the
> > >  opportunity to participate in the development of DataStream V2 and
> make
> > my contribution.
> > >
> > >
> > > Regarding FLIP-408, I have a question: The Processing TimerService is
> > currently
> > > defined as one of the basic primitives, partly because it's understood
> > that
> > > you have to choose between processing time and event time.
> > > The other part of the reason is that it needs to work based on the
> task's
> > > mailbox thread model to avoid concurrency issues. Could you clarify the
> > second
> > > part of the reason?
> > >
> > > Best,
> > > Wencong Liu
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > At 2023-12-26 14:42:20, "weijie guo" 
> wrote:
> > > >Hi devs,
> > > >
> > > >
> > > >I'd like to start a discussion about FLIP-408: [Umbrella] Introduce
> > > >DataStream API V2 [1].
> > > >
> > > >
> > > >The DataStream API is one of the two main APIs that Flink provides for
> > > >writing data processing programs. As an API that was introduced
> > > >practically since day-1 of the project and has been evolved for nearly
> > > >a decade, we are observing more and more problems of it. Improvements
> > > >on these problems require significant breaking changes, which makes
> > > >in-place refactor impractical. Therefore, we propose to introduce a
> > > >new set of APIs, the DataStream API V2, to gradually replace the
> > > >original DataStream API.
> > > >
> > > >
> > > >The proposal to introduce a whole set new API is complex and includes
> > > >massive changes. We are planning  to break it down into multiple
> > > >sub-FLIPs for incremental discussion. This FLIP is only used as an
> > > >umbrella, mainly focusing on motivation, goals, and overall planning.
> > > >That is to say, more design and implementation details  will be
> > > >discussed in 

Re: Re: [DISCUSS] FLIP-331: Support EndOfStreamWindows and isOutputOnEOF operator attribute to optimize task deployment

[Xintong Song]

Sorry, I just sent an incomplete email draft by mistake.

Thanks for reviving this, Xuannan. The updated FLIP LGTM. +1 for it.

Best,

Xintong



On Tue, Jan 23, 2024 at 5:51 PM Xintong Song  wrote:

> Thanks
>
> Best,
>
> Xintong
>
>
>
> On Wed, Jan 10, 2024 at 5:56 PM Xuannan Su  wrote:
>
>> Hi all,
>>
>> After several rounds of offline discussions with Xingtong and Jinhao,
>> we have decided to narrow the scope of the FLIP. It will now focus on
>> introducing OperatorAttributes that indicate whether an operator emits
>> records only after inputs have ended. We will also use the attribute
>> to optimize task scheduling for better resource utilization. Setting
>> the backlog status and optimizing the operator implementation during
>> the backlog will be deferred to future work.
>>
>> In addition to the change above, we also make the following changes to
>> the FLIP to address the problems mentioned by Dong:
>> - Public interfaces are updated to reuse the GlobalWindows.
>> - Instead of making all outputs of the upstream operators of the
>> "isOutputOnlyAfterEndOfStream=true" operator blocking, we only make
>> the output of the operator with "isOutputOnlyAfterEndOfStream=true"
>> blocking. This can prevent the second problem Dong mentioned. In the
>> future, we may introduce an extra OperatorAttributes to indicate if an
>> operator has any side output.
>>
>> I would greatly appreciate any comment or feedback you may have on the
>> updated FLIP.
>>
>> Best regards,
>> Xuannan
>>
>> On Tue, Sep 26, 2023 at 11:24 AM Dong Lin  wrote:
>> >
>> > Hi all,
>> >
>> > Thanks for the review!
>> >
>> > Becket and I discussed this FLIP offline and we agreed on several things
>> > that need to be improved with this FLIP. I will summarize our discussion
>> > with the problems and TODOs. We will update the FLIP and let you know
>> once
>> > the FLIP is ready for review again.
>> >
>> > 1) Investigate whether it is possible to update the existing
>> GlobalWindows
>> > in a backward-compatible way and re-use it for the same purpose
>> > as EndOfStreamWindows, without introducing EndOfStreamWindows as a new
>> > class.
>> >
>> > Note that GlobalWindows#getDefaultTrigger returns a NeverTrigger
>> instance
>> > which will not trigger window's computation even on end-of-inputs. We
>> will
>> > need to investigate its existing usage and see if we can re-use it in a
>> > backward-compatible way.
>> >
>> > 2) Let JM know whether any operator in the upstream of the operator with
>> > "isOutputOnEOF=true" will emit output via any side channel. The FLIP
>> should
>> > update the execution mode of those operators *only if* all outputs from
>> > those operators are emitted only at the end of input.
>> >
>> > More specifically, the upstream operator might involve a user-defined
>> > operator that might emit output directly to an external service, where
>> the
>> > emission operation is not explicitly expressed as an operator's output
>> edge
>> > and thus not visible to JM. Similarly, it is also possible for the
>> > user-defined operator to register a timer
>> > via InternalTimerService#registerEventTimeTimer and emit output to an
>> > external service inside Triggerable#onEventTime. There is a chance that
>> > users still need related logic to output data in real-time, even if the
>> > downstream operators have isOutputOnEOF=true.
>> >
>> > One possible solution to address this problem is to add an extra
>> > OperatorAttribute to specify whether this operator might output records
>> in
>> > such a way that does not go through operator's output (e.g. side
>> output).
>> > Then the JM can safely enable the runtime optimization currently
>> described
>> > in the FLIP when there is no such operator.
>> >
>> > 3) Create a follow-up FLIP that allows users to specify whether a source
>> > with Boundedness=bounded should have isProcessingBacklog=true.
>> >
>> > This capability would effectively introduce a 3rd strategy to set
>> backlog
>> > status (in addition to FLIP-309 and FLIP-328). It might be useful to
>> note
>> > that, even though the data in bounded sources are backlog data in most
>> > practical use-cases, it is not necessarily true. For example, users
>> might
>> > want to start a Flink j

Re: Re: [DISCUSS] FLIP-331: Support EndOfStreamWindows and isOutputOnEOF operator attribute to optimize task deployment

[Xintong Song]

Thanks

Best,

Xintong



On Wed, Jan 10, 2024 at 5:56 PM Xuannan Su  wrote:

> Hi all,
>
> After several rounds of offline discussions with Xingtong and Jinhao,
> we have decided to narrow the scope of the FLIP. It will now focus on
> introducing OperatorAttributes that indicate whether an operator emits
> records only after inputs have ended. We will also use the attribute
> to optimize task scheduling for better resource utilization. Setting
> the backlog status and optimizing the operator implementation during
> the backlog will be deferred to future work.
>
> In addition to the change above, we also make the following changes to
> the FLIP to address the problems mentioned by Dong:
> - Public interfaces are updated to reuse the GlobalWindows.
> - Instead of making all outputs of the upstream operators of the
> "isOutputOnlyAfterEndOfStream=true" operator blocking, we only make
> the output of the operator with "isOutputOnlyAfterEndOfStream=true"
> blocking. This can prevent the second problem Dong mentioned. In the
> future, we may introduce an extra OperatorAttributes to indicate if an
> operator has any side output.
>
> I would greatly appreciate any comment or feedback you may have on the
> updated FLIP.
>
> Best regards,
> Xuannan
>
> On Tue, Sep 26, 2023 at 11:24 AM Dong Lin  wrote:
> >
> > Hi all,
> >
> > Thanks for the review!
> >
> > Becket and I discussed this FLIP offline and we agreed on several things
> > that need to be improved with this FLIP. I will summarize our discussion
> > with the problems and TODOs. We will update the FLIP and let you know
> once
> > the FLIP is ready for review again.
> >
> > 1) Investigate whether it is possible to update the existing
> GlobalWindows
> > in a backward-compatible way and re-use it for the same purpose
> > as EndOfStreamWindows, without introducing EndOfStreamWindows as a new
> > class.
> >
> > Note that GlobalWindows#getDefaultTrigger returns a NeverTrigger instance
> > which will not trigger window's computation even on end-of-inputs. We
> will
> > need to investigate its existing usage and see if we can re-use it in a
> > backward-compatible way.
> >
> > 2) Let JM know whether any operator in the upstream of the operator with
> > "isOutputOnEOF=true" will emit output via any side channel. The FLIP
> should
> > update the execution mode of those operators *only if* all outputs from
> > those operators are emitted only at the end of input.
> >
> > More specifically, the upstream operator might involve a user-defined
> > operator that might emit output directly to an external service, where
> the
> > emission operation is not explicitly expressed as an operator's output
> edge
> > and thus not visible to JM. Similarly, it is also possible for the
> > user-defined operator to register a timer
> > via InternalTimerService#registerEventTimeTimer and emit output to an
> > external service inside Triggerable#onEventTime. There is a chance that
> > users still need related logic to output data in real-time, even if the
> > downstream operators have isOutputOnEOF=true.
> >
> > One possible solution to address this problem is to add an extra
> > OperatorAttribute to specify whether this operator might output records
> in
> > such a way that does not go through operator's output (e.g. side output).
> > Then the JM can safely enable the runtime optimization currently
> described
> > in the FLIP when there is no such operator.
> >
> > 3) Create a follow-up FLIP that allows users to specify whether a source
> > with Boundedness=bounded should have isProcessingBacklog=true.
> >
> > This capability would effectively introduce a 3rd strategy to set backlog
> > status (in addition to FLIP-309 and FLIP-328). It might be useful to note
> > that, even though the data in bounded sources are backlog data in most
> > practical use-cases, it is not necessarily true. For example, users might
> > want to start a Flink job to consume real-time data from a Kafka topic
> and
> > specify that the job stops after 24 hours, which means the source is
> > technically bounded while the data is fresh/real-time.
> >
> > This capability is more generic and can cover more use-case than
> > EndOfStreamWindows. On the other hand, EndOfStreamWindows will still be
> > useful in cases where users already need to specify this window assigner
> in
> > a DataStream program, without bothering users to decide whether it is
> safe
> > to treat data in a bounded source as backlog data.

Re: [VOTE] Accept Flink CDC into Apache Flink

[Xintong Song]

+1 (binding)

Best,

Xintong



On Tue, Jan 9, 2024 at 3:31 PM Benchao Li  wrote:

> +1 (non-binding)
>
> Feng Wang  于2024年1月9日周二 15:29写道：
> >
> > +1 non-binding
> > Regards,
> > Feng
> >
> > On Tue, Jan 9, 2024 at 3:05 PM Leonard Xu  wrote:
> >
> > > Hello all,
> > >
> > > This is the official vote whether to accept the Flink CDC code
> contribution
> > >  to Apache Flink.
> > >
> > > The current Flink CDC code, documentation, and website can be
> > > found here:
> > > code: https://github.com/ververica/flink-cdc-connectors <
> > > https://github.com/ververica/flink-cdc-connectors>
> > > docs: https://ververica.github.io/flink-cdc-connectors/ <
> > > https://ververica.github.io/flink-cdc-connectors/>
> > >
> > > This vote should capture whether the Apache Flink community is
> interested
> > > in accepting, maintaining, and evolving Flink CDC.
> > >
> > > Regarding my original proposal[1] in the dev mailing list, I firmly
> believe
> > > that this initiative aligns perfectly with Flink. For the Flink
> community,
> > > it represents an opportunity to bolster Flink's competitive edge in
> > > streaming
> > > data integration, fostering the robust growth and prosperity of the
> Apache
> > > Flink
> > > ecosystem. For the Flink CDC project, becoming a sub-project of Apache
> > > Flink
> > > means becoming an integral part of a neutral open-source community,
> > > capable of
> > > attracting a more diverse pool of contributors.
> > >
> > > All Flink CDC maintainers are dedicated to continuously contributing to
> > > achieve
> > > seamless integration with Flink. Additionally, PMC members like Jark,
> > > Qingsheng,
> > > and I are willing to infacilitate the expansion of contributors and
> > > committers to
> > > effectively maintain this new sub-project.
> > >
> > > This is a "Adoption of a new Codebase" vote as per the Flink bylaws
> [2].
> > > Only PMC votes are binding. The vote will be open at least 7 days
> > > (excluding weekends), meaning until Thursday January 18 12:00 UTC, or
> > > until we
> > > achieve the 2/3rd majority. We will follow the instructions in the
> Flink
> > > Bylaws
> > > in the case of insufficient active binding voters:
> > >
> > > > 1. Wait until the minimum length of the voting passes.
> > > > 2. Publicly reach out via personal email to the remaining binding
> voters
> > > in the
> > > voting mail thread for at least 2 attempts with at least 7 days between
> > > two attempts.
> > > > 3. If the binding voter being contacted still failed to respond after
> > > all the attempts,
> > > the binding voter will be considered as inactive for the purpose of
> this
> > > particular voting.
> > >
> > > Welcome voting !
> > >
> > > Best,
> > > Leonard
> > > [1] https://lists.apache.org/thread/o7klnbsotmmql999bnwmdgo56b6kxx9l
> > > [2]
> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120731026
>
>
>
> --
>
> Best,
> Benchao Li
>

Re: [VOTE] FLIP-405: Migrate string configuration key to ConfigOption

[Xintong Song]

+1 (binding)

Best,

Xintong



On Mon, Jan 8, 2024 at 1:48 PM Hang Ruan  wrote:

> +1(non-binding)
>
> Best,
> Hang
>
> Rui Fan <1996fan...@gmail.com> 于2024年1月8日周一 13:04写道：
>
> > +1(binding)
> >
> > Best,
> > Rui
> >
> > On Mon, Jan 8, 2024 at 1:00 PM Xuannan Su  wrote:
> >
> > > Hi everyone,
> > >
> > > Thanks for all the feedback about the FLIP-405: Migrate string
> > > configuration key to ConfigOption [1] [2].
> > >
> > > I'd like to start a vote for it. The vote will be open for at least 72
> > > hours(excluding weekends,until Jan 11, 12:00AM GMT) unless there is an
> > > objection or an insufficient number of votes.
> > >
> > >
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-405%3A+Migrate+string+configuration+key+to+ConfigOption
> > > [2] https://lists.apache.org/thread/zfw1b1g3679yn0ppjbsokfrsx9k7ybg0
> > >
> > >
> > > Best,
> > > Xuannan
> > >
> >
>

Re: [2.0] Help needed for release 2.0 work items

[Xintong Song]

>
> I would like to ask if there is any plan to review and refactor the CLI in
> Flink 2.0.
>

Not that I'm aware of.


I have the impression of seeing discussions about not using CLI options and
using only "-Dconfig.key", but I cannot find it now. I personally think
that is a good direction to go, and should also solve the problem mentioned
in your example. My biggest concern is whether there are contributors with
enough capacity to work on it.


Feel free to pick it up if you'd like to.


Best,

Xintong



On Wed, Jan 3, 2024 at 11:37 AM Zakelly Lan  wrote:

> Hi Xintong,
>
> Thanks for driving this.
>
> I would like to ask if there is any plan to review and refactor the CLI in
> Flink 2.0. I recently found that the CLI commands and parameters are
> confusing in some ways (e.g.
> https://github.com/apache/flink/pull/23253#discussion_r1405707256). It
> would be beneficial to offer a more intuitive and straightforward CLI
> command to enhance usability.
>
>
> Best,
> Zakelly
>
> On Wed, Jan 3, 2024 at 10:45 AM Xintong Song 
> wrote:
>
> > Thanks a lot for offering the help, Rui. The plan sounds good to me. I'll
> > put your name and the milestones into the 2.0 wiki page.
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Wed, Jan 3, 2024 at 10:38 AM Rui Fan <1996fan...@gmail.com> wrote:
> >
> > > Thanks Xintong for promoting the progress of Flink 2.0.
> > >
> > > If no one minds, I'd like to pick this one: Use Java’s Duration instead
> > of
> > > Flink’s Time.
> > > Could I assign FLINK-14068[1] to me?
> > >
> > > My expected progress is:
> > > - Mark org.apache.flink.api.common.time.Time and
> > >  org.apache.flink.streaming.api.windowing.time.Time
> > >  as @Deprecated in 1.19 (Must do in 1.19)
> > > - Refactor all usages of them to Java's Duration(Nice do in 1.19, must
> do
> > > in 1.20)
> > > - Remove them in 2.0
> > >
> > > Is this plan reasonable?
> > >
> > > [1] https://issues.apache.org/jira/browse/FLINK-14068
> > >
> > > Best,
> > > Rui
> > >
> > > On Wed, Jan 3, 2024 at 9:18 AM Xintong Song 
> > wrote:
> > >
> > >> Hi devs,
> > >>
> > >> The release managers have been tracking the progress of release 2.0
> work
> > >> items. Unfortunately, some of the items are not in good progress, and
> > >> either don't have a contributor or the original contributor no longer
> > has
> > >> capacity to work on them. We have already tried reaching out to some
> > >> developers, but unfortunately don't find many people with capacity.
> > >>
> > >> Therefore, we are looking for developers who want to pick them up.
> > >>
> > >> Helps are needed on:
> > >>
> > >>- Introduce dedicated MetricsScope
> > >>- Rework MetricGroup scope APIs
> > >>- Remove MetricGroup methods accepting an int as a name
> > >>- Remove brackets around variables
> > >>- Drop MetricReporter#open
> > >>- Gauge should only take subclasses of Number, rather than
> > >> everything
> > >>- Add MetricGroup#getLogicalScope
> > >>- User Java’s Duration instead of Flink’s Time
> > >>- Review and refactor the REST API
> > >>- Properly handle NaN/Infinity in OpenAPI spec
> > >>- Enforce single maxExceptions query parameter
> > >>- Drop Yarn specific get rest endpoints
> > >>- Review and refactor the metrics implementation
> > >>- Attach semantics to Gauges; refactor Counter / Meter to be Gauges
> > >> with
> > >>syntactic sugar on top
> > >>- Restructure
> > >>
> > >>
> > >> Please note that:
> > >>
> > >>- For some of the items, the milestones are already given, and
> there
> > >>might be some actions that need to be performed by Flink 1.19.
> Please
> > >> be
> > >>aware that we are only 3.5 weeks from the 1.19 feature freeze.
> > >>- There are also items which don't have any plans / milestones yet.
> > For
> > >>such items, we may want to quickly look into them to find out if
> > >> there's
> > >>anything that needs to be done in 1.19.
> > >>- See more details on the 2.0 wiki page [1]
> > >>
> > >>
> > >> If these items do not make Flink 1.19, we can discuss later what to do
> > >> with
> > >> them, either postpone release 2.0 or exclude them from this major
> > release.
> > >> But for now, let's first see what we can do by 1.19.
> > >>
> > >> Best,
> > >>
> > >> Xintong
> > >>
> > >>
> > >> [1] https://cwiki.apache.org/confluence/display/FLINK/2.0+Release
> > >>
> > >
> >
>

Re: [2.0] Help needed for release 2.0 work items

[Xintong Song]

Thanks a lot for offering the help, Rui. The plan sounds good to me. I'll
put your name and the milestones into the 2.0 wiki page.

Best,

Xintong



On Wed, Jan 3, 2024 at 10:38 AM Rui Fan <1996fan...@gmail.com> wrote:

> Thanks Xintong for promoting the progress of Flink 2.0.
>
> If no one minds, I'd like to pick this one: Use Java’s Duration instead of
> Flink’s Time.
> Could I assign FLINK-14068[1] to me?
>
> My expected progress is:
> - Mark org.apache.flink.api.common.time.Time and
>  org.apache.flink.streaming.api.windowing.time.Time
>  as @Deprecated in 1.19 (Must do in 1.19)
> - Refactor all usages of them to Java's Duration(Nice do in 1.19, must do
> in 1.20)
> - Remove them in 2.0
>
> Is this plan reasonable?
>
> [1] https://issues.apache.org/jira/browse/FLINK-14068
>
> Best,
> Rui
>
> On Wed, Jan 3, 2024 at 9:18 AM Xintong Song  wrote:
>
>> Hi devs,
>>
>> The release managers have been tracking the progress of release 2.0 work
>> items. Unfortunately, some of the items are not in good progress, and
>> either don't have a contributor or the original contributor no longer has
>> capacity to work on them. We have already tried reaching out to some
>> developers, but unfortunately don't find many people with capacity.
>>
>> Therefore, we are looking for developers who want to pick them up.
>>
>> Helps are needed on:
>>
>>- Introduce dedicated MetricsScope
>>- Rework MetricGroup scope APIs
>>- Remove MetricGroup methods accepting an int as a name
>>- Remove brackets around variables
>>- Drop MetricReporter#open
>>- Gauge should only take subclasses of Number, rather than
>> everything
>>- Add MetricGroup#getLogicalScope
>>- User Java’s Duration instead of Flink’s Time
>>- Review and refactor the REST API
>>- Properly handle NaN/Infinity in OpenAPI spec
>>- Enforce single maxExceptions query parameter
>>- Drop Yarn specific get rest endpoints
>>- Review and refactor the metrics implementation
>>- Attach semantics to Gauges; refactor Counter / Meter to be Gauges
>> with
>>syntactic sugar on top
>>- Restructure
>>
>>
>> Please note that:
>>
>>- For some of the items, the milestones are already given, and there
>>might be some actions that need to be performed by Flink 1.19. Please
>> be
>>aware that we are only 3.5 weeks from the 1.19 feature freeze.
>>- There are also items which don't have any plans / milestones yet. For
>>such items, we may want to quickly look into them to find out if
>> there's
>>anything that needs to be done in 1.19.
>>- See more details on the 2.0 wiki page [1]
>>
>>
>> If these items do not make Flink 1.19, we can discuss later what to do
>> with
>> them, either postpone release 2.0 or exclude them from this major release.
>> But for now, let's first see what we can do by 1.19.
>>
>> Best,
>>
>> Xintong
>>
>>
>> [1] https://cwiki.apache.org/confluence/display/FLINK/2.0+Release
>>
>

Re: [ANNOUNCE] New Apache Flink Committer - Alexander Fedulov

[Xintong Song]

Congratulations~!

Best,

Xintong



On Wed, Jan 3, 2024 at 9:36 AM Leonard Xu  wrote:

> Congrats, Alex!
>
> Best,
> Leonard
>
> > 2024年1月3日 上午4:11，Tang, Zhiyan (udx2na)  写道：
> >
> > big congrats to Alex and Happy New Year everyone!
> >
> > Best
> > Tony
> >
> > Get Outlook for iOS
> > 
> > From: Austin Cawley-Edwards 
> > Sent: Tuesday, January 2, 2024 2:07:49 PM
> > To: dev@flink.apache.org 
> > Cc: Alexander Fedulov 
> > Subject: Re: [ANNOUNCE] New Apache Flink Committer - Alexander Fedulov
> >
> > Congrats Alex!!
> >
> > On Tue, Jan 2, 2024 at 10:12 Feng Jin  wrote:
> >
> >> Congratulations, Alex!
> >>
> >> Best,
> >> Feng
> >>
> >> On Tue, Jan 2, 2024 at 11:04 PM Chen Yu  wrote:
> >>
> >>> Congratulations, Alex!
> >>>
> >>> Best,
> >>> Yu Chen
> >>>
> >>> 
> >>> 发件人: Zhanghao Chen 
> >>> 发送时间: 2024年1月2日 22:44
> >>> 收件人: dev
> >>> 抄送: Alexander Fedulov
> >>> 主题: Re: [ANNOUNCE] New Apache Flink Committer - Alexander Fedulov
> >>>
> >>> Congrats, Alex!
> >>>
> >>> Best,
> >>> Zhanghao Chen
> >>> 
> >>> From: Maximilian Michels 
> >>> Sent: Tuesday, January 2, 2024 20:15
> >>> To: dev 
> >>> Cc: Alexander Fedulov 
> >>> Subject: [ANNOUNCE] New Apache Flink Committer - Alexander Fedulov
> >>>
> >>> Happy New Year everyone,
> >>>
> >>> I'd like to start the year off by announcing Alexander Fedulov as a
> >>> new Flink committer.
> >>>
> >>> Alex has been active in the Flink community since 2019. He has
> >>> contributed more than 100 commits to Flink, its Kubernetes operator,
> >>> and various connectors [1][2].
> >>>
> >>> Especially noteworthy are his contributions on deprecating and
> >>> migrating the old Source API functions and test harnesses, the
> >>> enhancement to flame graphs, the dynamic rescale time computation in
> >>> Flink Autoscaling, as well as all the small enhancements Alex has
> >>> contributed which make a huge difference.
> >>>
> >>> Beyond code contributions, Alex has been an active community member
> >>> with his activity on the mailing lists [3][4], as well as various
> >>> talks and blog posts about Apache Flink [5][6].
> >>>
> >>> Congratulations Alex! The Flink community is proud to have you.
> >>>
> >>> Best,
> >>> The Flink PMC
> >>>
> >>> [1]
> >>>
> https://github.com/search?type=commits&q=author%3Aafedulov+org%3Aapache
> >>> [2]
> >>>
> >>
> https://issues.apache.org/jira/browse/FLINK-28229?jql=status%20in%20(Resolved%2C%20Closed)%20AND%20assignee%20in%20(afedulov)%20ORDER%20BY%20resolved%20DESC%2C%20created%20DESC
> >>> [3]
> https://lists.apache.org/list?dev@flink.apache.org:lte=100M:Fedulov
> >>> [4]
> https://lists.apache.org/list?u...@flink.apache.org:lte=100M:Fedulov
> >>> [5]
> >>>
> >>
> https://flink.apache.org/2020/01/15/advanced-flink-application-patterns-vol.1-case-study-of-a-fraud-detection-system/
> >>> [6]
> >>>
> >>
> https://www.ververica.com/blog/presenting-our-streaming-concepts-introduction-to-flink-video-series
> >>>
> >>
>
>

[2.0] Help needed for release 2.0 work items

[Xintong Song]

Hi devs,

The release managers have been tracking the progress of release 2.0 work
items. Unfortunately, some of the items are not in good progress, and
either don't have a contributor or the original contributor no longer has
capacity to work on them. We have already tried reaching out to some
developers, but unfortunately don't find many people with capacity.

Therefore, we are looking for developers who want to pick them up.

Helps are needed on:

   - Introduce dedicated MetricsScope
   - Rework MetricGroup scope APIs
   - Remove MetricGroup methods accepting an int as a name
   - Remove brackets around variables
   - Drop MetricReporter#open
   - Gauge should only take subclasses of Number, rather than everything
   - Add MetricGroup#getLogicalScope
   - User Java’s Duration instead of Flink’s Time
   - Review and refactor the REST API
   - Properly handle NaN/Infinity in OpenAPI spec
   - Enforce single maxExceptions query parameter
   - Drop Yarn specific get rest endpoints
   - Review and refactor the metrics implementation
   - Attach semantics to Gauges; refactor Counter / Meter to be Gauges with
   syntactic sugar on top
   - Restructure


Please note that:

   - For some of the items, the milestones are already given, and there
   might be some actions that need to be performed by Flink 1.19. Please be
   aware that we are only 3.5 weeks from the 1.19 feature freeze.
   - There are also items which don't have any plans / milestones yet. For
   such items, we may want to quickly look into them to find out if there's
   anything that needs to be done in 1.19.
   - See more details on the 2.0 wiki page [1]


If these items do not make Flink 1.19, we can discuss later what to do with
them, either postpone release 2.0 or exclude them from this major release.
But for now, let's first see what we can do by 1.19.

Best,

Xintong


[1] https://cwiki.apache.org/confluence/display/FLINK/2.0+Release

Re: [VOTE] FLIP-398: Improve Serialization Configuration And Usage In Flink

[Xintong Song]

+1 (binding)


Best,

Xintong



On Wed, Dec 27, 2023 at 2:54 PM Yong Fang  wrote:

> Hi devs,
>
> Thanks for all feedback about the FLIP-398: Improve Serialization
> Configuration And Usage In Flink [1] which has been discussed in [2].
>
> I'd like to start a vote for it. The vote will be open for at least 72
> hours unless there is an objection or insufficient votes.
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-398%3A+Improve+Serialization+Configuration+And+Usage+In+Flink
> [2] https://lists.apache.org/thread/m67s4qfrh660lktpq7yqf9docvvf5o9l
>
> Best,
> Fang Yong
>

Re: [DISCUSS] FLIP-405: Migrate string configuration key to ConfigOption

[Xintong Song]

>
> Configuration has a `public  T get(ConfigOption option)` method.
> Could we remove all `Xxx getXxx(ConfigOption configOption)` methods?



Note: all `public void setXxx(ConfigOption key, Xxx value)` methods
> can be replaced with `public  Configuration set(ConfigOption option,
> T value)` as well.


+1


Best,

Xintong



On Tue, Dec 26, 2023 at 8:44 PM Xintong Song  wrote:

> These features don't have a public option, but they work. I'm not sure
>> whether these features are used by some advanced users.
>> Actually, I think some of them are valuable! For example:
>>
>> - ConfigConstants.YARN_CONTAINER_START_COMMAND_TEMPLATE
>>   allows users to define the start command of the yarn container.
>> - FileInputFormat.ENUMERATE_NESTED_FILES_FLAG allows
>>   flink job reads all files under the directory even if it has nested
>> directories.
>>
>> This FLIP focuses on the refactor option, I'm afraid these features are
>> used
>> in some production and removing these features will affect some flink
>> jobs.
>> So I prefer to keep these features, WDTY?
>>
>
> First of all, I don't think we should support any knobs that users can
> only learn how to use from reading Flink's internal codes. From this
> perspective, for existing string-keyed knobs that are not mentioned in any
> public documentation, yes we can argue that they are functioning, but we
> can also argue that they are not really exposed to users. That means
> migrating them to ConfigOption is not a pure refactor, but would make
> something that used to be hidden from users now exposed to users. For such
> options, I personally would lean toward not exposing them. If we consider
> them as already exposed, then process-wise there's no problem in
> deprecating some infrequently-used options and removing them in a major
> version bump, and if they are proved needed later we can add them back
> anytime. On the other hand, if we consider them as not yet exposed, then
> removing them later would be a breaking change.
>
>
> Secondly, I don't really come up with any cases where users need to tune
> these knobs. E.g., why would we allow users to customize the yarn container
> start command while we already provide `env.java.opts`? And what would be
> the problem if Flink just doesn't support nested files? And even worse,
> such knobs may provide chances for users to shoot themself in the foot.
> E.g., what if %jvmmem% is missing from a user-provided container start
> command? Admittedly, there might be a small fraction of advanced users that
> know how to use these knobs. However, those users usually have their own
> custom fork of Flink, and it should not be a big problem for them to build
> such abilities by themselves.
>
>
> Taking a step back, if we decide that some of the mentioned knobs are
> really useful, we should at least provide enough descriptions to help users
> understand when and how to use these options. E.g., the current description
> for yarn container command template is far from enough, which does not
> explain what the placeholders mean, what happens if some placeholders are
> missing or if an unknown placeholder is provided.
>
>
> Best,
>
> Xintong
>
>
>
> On Tue, Dec 26, 2023 at 7:39 PM Rui Fan <1996fan...@gmail.com> wrote:
>
>> In addition to what is written in FLIP, I found some methods of
>> Configuration
>> are not necessary. And I wanna hear more thoughts from all of you.
>>
>> Configuration has a `public  T get(ConfigOption option)` method.
>> Could we remove all `Xxx getXxx(ConfigOption configOption)` methods?
>> Such as:
>> - public int getInteger(ConfigOption configOption)
>> - public String getString(ConfigOption configOption)
>> - public long getLong(ConfigOption configOption)
>> - etc
>>
>> I prefer users and flink use `get(ConfigOption option)` instead of
>> `getXxx(ConfigOption configOption)` based on some reasons:
>>
>> 1. All callers can replace it directly without any extra effort.
>>   `T get(ConfigOption option)` method can replace all
>>   `Xxx getXxx(ConfigOption configOption)` methods directly.
>> 2. Callers can call get directly, and users or flink developers
>>   don't need to care about should they call getInteger or getString.
>> 3. Flink code is easier to maintain.
>> 4. `T get(ConfigOption option)` is designed later than
>> `Xxx getXxx(ConfigOption configOption)`, I guess if
>> `T get(ConfigOption option)` is designed first,
>>all `Xxx getXxx(ConfigOption configOption)` methods
>>   aren't needed.
>>
>> Note: all `public void setXxx(ConfigOpt

Re: [DISCUSS] FLIP-405: Migrate string configuration key to ConfigOption

[Xintong Song]

can explain in the description that if not configured,
> > > `System.getProperty("log.file")` will be used, but that is not a
> default
> > > value.
> >
> > Explain it in the description is fine for me.
> >
> > > 2. So I wonder if we can simply mark them as deprecated and remove in
> > 2.0.
> >
> > These features don't have a public option, but they work. I'm not sure
> > whether these features are used by some advanced users.
> > Actually, I think some of them are valuable! For example:
> >
> > - ConfigConstants.YARN_CONTAINER_START_COMMAND_TEMPLATE
> >   allows users to define the start command of the yarn container.
> > - FileInputFormat.ENUMERATE_NESTED_FILES_FLAG allows
> >   flink job reads all files under the directory even if it has nested
> > directories.
> >
> > This FLIP focuses on the refactor option, I'm afraid these features are
> > used
> > in some production and removing these features will affect some flink
> jobs.
> > So I prefer to keep these features, WDTY?
> >
> > > 3. Simply saying "getting / setting value with string key is
> discouraged"
> > > in JavaDoc of get/setString is IMHO a bit confusing. People may have
> the
> > > question why would we keep the discouraged interfaces at all. I would
> > > suggest the following:
> > > ```
> > > We encourage users and developers to always use ConfigOption for
> getting
> > /
> > > setting the configurations if possible, for its rich description, type,
> > > default-value and other supports. The string-key-based getter / setter
> > > should only be used when ConfigOption is not applicable, e.g., the key
> is
> > > programmatically generated in runtime.
> > > ```
> >
> > Suggested comment is good for me, and I'd like to hear the thought from
> > Xuannan
> > who wrote the original comment.
> >
> > Best,
> > Rui
> >
> > On Tue, Dec 26, 2023 at 5:26 PM Xintong Song 
> > wrote:
> >
> >> Thanks for the efforts, Rui and Xuannan.
> >>
> >> I think it's a good idea to migrate string-key configuration accesses to
> >> ConfigOption-s in general.
> >>
> >> I have a few suggestions / questions regarding the FLIP.
> >>
> >> 1. I think the default value for `TASK_MANAGER_LOG_PATH_KEY` should be
> "no
> >> default". We can explain in the description that if not configured,
> >> `System.getProperty("log.file")` will be used, but that is not a default
> >> value.
> >>
> >> 2. I wonder if the following string-keys can be simply removed? They are
> >> neither set by Flink, nor documented anywhere (AFAIK) so that users know
> >> how to set them. All of them were introduced a long time ago, require
> >> significant knowledge and familiarity about Flink internals and low
> level
> >> details in order to use, and some of them are even private. So I wonder
> if
> >> we can simply mark them as deprecated and remove in 2.0.
> >> - ConfigConstants.YARN_CONTAINER_START_COMMAND_TEMPLATE
> >> - FileInputFormat.FILE_PARAMETER_KEY
> >> - FileInputFormat.ENUMERATE_NESTED_FILES_FLAG
> >> - FileOutputFormat.FILE_PARAMETER_KEY
> >> - BinaryInputFormat.BLOCK_SIZE_PARAMETER_KEY
> >> - BinaryOutputFormat.BLOCK_SIZE_PARAMETER_KEY
> >>
> >> 3. Simply saying "getting / setting value with string key is
> discouraged"
> >> in JavaDoc of get/setString is IMHO a bit confusing. People may have the
> >> question why would we keep the discouraged interfaces at all. I would
> >> suggest the following:
> >> ```
> >> We encourage users and developers to always use ConfigOption for
> getting /
> >> setting the configurations if possible, for its rich description, type,
> >> default-value and other supports. The string-key-based getter / setter
> >> should only be used when ConfigOption is not applicable, e.g., the key
> is
> >> programmatically generated in runtime.
> >> ```
> >>
> >> WDYT?
> >>
> >> Best,
> >>
> >> Xintong
> >>
> >>
> >>
> >> On Tue, Dec 26, 2023 at 4:12 PM Rui Fan <1996fan...@gmail.com> wrote:
> >>
> >> > Hi all,
> >> >
> >> > Xuannan(cced) and I would like to start a discussion on FLIP-405:
> >> > FLIP-405: Migrate string configuration key to ConfigOption[1].
> >> >
> >> > As Flink progresses to 2.0, we want to enhance the user experience
> >> > with the existing configuration. In FLIP-77, Flink introduced
> >> ConfigOption
> >> > with DataType and strongly encourage users to utilize ConfigOption
> >> > instead of string keys for accessing and setting Flink configurations.
> >> > Presently, many string configuration keys have been deprecated and
> >> > replaced with ConfigOptions; however, some string configuration
> >> > keys are still in use.
> >> >
> >> > To ensure a better experience with the existing configuration in Flink
> >> > 2.0,
> >> > this FLIP will migrate all user-facing string configuration keys to
> >> > ConfigOptions.
> >> > Additionally, we want to modify the Configuration infrastructure to
> >> > promote the use of ConfigOption over string configuration keys
> >> > among developers and users. It's mentioned in a preview thread[2].
> >> >
> >> > Looking forward to everyone's feedback and suggestions, thank you!
> >> >
> >> > [1] https://cwiki.apache.org/confluence/x/6Yr5E
> >> > [2] https://lists.apache.org/thread/zzsf7glfcdjcjm1hfo1xdwc6jp37nb3m
> >> >
> >> > Best,
> >> > Rui
> >> >
> >>
> >
>

Re: [DISCUSS] FLIP-405: Migrate string configuration key to ConfigOption

[Xintong Song]

Thanks for the efforts, Rui and Xuannan.

I think it's a good idea to migrate string-key configuration accesses to
ConfigOption-s in general.

I have a few suggestions / questions regarding the FLIP.

1. I think the default value for `TASK_MANAGER_LOG_PATH_KEY` should be "no
default". We can explain in the description that if not configured,
`System.getProperty("log.file")` will be used, but that is not a default
value.

2. I wonder if the following string-keys can be simply removed? They are
neither set by Flink, nor documented anywhere (AFAIK) so that users know
how to set them. All of them were introduced a long time ago, require
significant knowledge and familiarity about Flink internals and low level
details in order to use, and some of them are even private. So I wonder if
we can simply mark them as deprecated and remove in 2.0.
- ConfigConstants.YARN_CONTAINER_START_COMMAND_TEMPLATE
- FileInputFormat.FILE_PARAMETER_KEY
- FileInputFormat.ENUMERATE_NESTED_FILES_FLAG
- FileOutputFormat.FILE_PARAMETER_KEY
- BinaryInputFormat.BLOCK_SIZE_PARAMETER_KEY
- BinaryOutputFormat.BLOCK_SIZE_PARAMETER_KEY

3. Simply saying "getting / setting value with string key is discouraged"
in JavaDoc of get/setString is IMHO a bit confusing. People may have the
question why would we keep the discouraged interfaces at all. I would
suggest the following:
```
We encourage users and developers to always use ConfigOption for getting /
setting the configurations if possible, for its rich description, type,
default-value and other supports. The string-key-based getter / setter
should only be used when ConfigOption is not applicable, e.g., the key is
programmatically generated in runtime.
```

WDYT?

Best,

Xintong



On Tue, Dec 26, 2023 at 4:12 PM Rui Fan <1996fan...@gmail.com> wrote:

> Hi all,
>
> Xuannan(cced) and I would like to start a discussion on FLIP-405:
> FLIP-405: Migrate string configuration key to ConfigOption[1].
>
> As Flink progresses to 2.0, we want to enhance the user experience
> with the existing configuration. In FLIP-77, Flink introduced ConfigOption
> with DataType and strongly encourage users to utilize ConfigOption
> instead of string keys for accessing and setting Flink configurations.
> Presently, many string configuration keys have been deprecated and
> replaced with ConfigOptions; however, some string configuration
> keys are still in use.
>
> To ensure a better experience with the existing configuration in Flink
> 2.0,
> this FLIP will migrate all user-facing string configuration keys to
> ConfigOptions.
> Additionally, we want to modify the Configuration infrastructure to
> promote the use of ConfigOption over string configuration keys
> among developers and users. It's mentioned in a preview thread[2].
>
> Looking forward to everyone's feedback and suggestions, thank you!
>
> [1] https://cwiki.apache.org/confluence/x/6Yr5E
> [2] https://lists.apache.org/thread/zzsf7glfcdjcjm1hfo1xdwc6jp37nb3m
>
> Best,
> Rui
>

Re: [DISCUSS] Should Configuration support getting value based on String key?

[Xintong Song]

Ideally, public API changes should go through the FLIP process.

I see the point that starting a FLIP for such a tiny change might be
overkill. However, one could also argue that anything that is too trivial
to go through a FLIP should also be easy enough to quickly get through the
process.

In this particular case, since you and Xuannan are working on another FLIP
regarding the usage of string-keys in configurations, why not make removing
of the @Deprecated annotation part of that FLIP?

Best,

Xintong



On Wed, Dec 20, 2023 at 9:27 PM Rui Fan <1996fan...@gmail.com> wrote:

> Thanks Xuannan and Xintong for the quick feedback!
>
> > However, we should let the user know that we encourage using
> ConfigOptions over the string-based
> > configuration key, as Timo said, we should add the message to `String
> > getString(String key, String defaultValue)` method.
>
> Sure, we can add some comments to guide users to use ConfigOption.
>
> If so, I will remove the @Deprecated annotation for
> `getString(String key, String defaultValue)` method`, and add
> some comments  for it.
>
> Also, it's indeed a small change related to Public class(or Api),
> is voting necessary?
>
> Best,
> Rui
>
> On Wed, 20 Dec 2023 at 16:55, Xintong Song  wrote:
>
> > Sounds good to me.
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Wed, Dec 20, 2023 at 9:40 AM Xuannan Su 
> wrote:
> >
> > > Hi Rui,
> > >
> > > I am fine with keeping the `String getString(String key, String
> > > defaultValue)` if more people favor it. However, we should let the
> > > user know that we encourage using ConfigOptions over the string-based
> > > configuration key, as Timo said, we should add the message to `String
> > > getString(String key, String defaultValue)` method.
> > >
> > > Best,
> > > Xuannan
> > >
> > > On Tue, Dec 19, 2023 at 7:55 PM Rui Fan <1996fan...@gmail.com> wrote:
> > > >
> > > > > > I noticed that Configuration is used in
> > > > > > DistributedCache#writeFileInfoToConfig and readFileInfoFromConfig
> > > > > > to store some cacheFile meta-information. Their keys are
> > > > > > temporary(key name with number) and it is not convenient
> > > > > > to predefine ConfigOption.
> > > > >
> > > > >
> > > > > True, this one requires a bit more effort to migrate from
> string-key
> > to
> > > > > ConfigOption, but still should be doable. Looking at how the two
> > > mentioned
> > > > > methods are implemented and used, it seems what is really needed is
> > > > > serialization and deserialization of `DistributedCacheEntry`-s. And
> > > all the
> > > > > entries are always written / read at once. So I think we can
> > serialize
> > > the
> > > > > whole set of entries into a JSON string (or something similar), and
> > > use one
> > > > > ConfigOption with a deterministic key for it, rather than having
> one
> > > > > ConfigOption for each field in each entry. WDYT?
> > > >
> > > > Hi Xintong, thanks for the good suggestion! Most of the entries can
> be
> > > > serialized to a json string, and we can only write/read them at once.
> > > > The CACHE_FILE_BLOB_KEY is special, its type is byte[], we need to
> > > > store it by the setBytes/getBytes.
> > > >
> > > > Also, I have an offline discussion with @Xuannan Su : refactoring all
> > > code
> > > > that uses String as key requires a separate FLIP. And we will provide
> > > > detailed FLIP  later.
> > > >
> > > >
> > >
> >
> --
> > > >
> > > > Hi all, thanks everyone for the lively discussion. It's really a
> > > trade-off to
> > > > keep "String getString(String key, String defaultValue)" or not.
> > > > (It's not a right or wrong thing.)
> > > >
> > > > Judging from the discussion, most discussants can accept that keeping
> > > > `String getString(String key, String defaultValue)` and depreate the
> > > > rest of `getXxx(String key, Xxx defaultValue)`.
> > > >
> > > > cc @Xintong Song @Xuannan Su , WDYT?
> > > >
> > > > Best,
> > > > Rui
> > > >
> > > > On Fri, Dec 15, 2023 at 11:38 AM Zhu Zh

Re: [DISCUSS] Should Configuration support getting value based on String key?

[Xintong Song]

Sounds good to me.

Best,

Xintong



On Wed, Dec 20, 2023 at 9:40 AM Xuannan Su  wrote:

> Hi Rui,
>
> I am fine with keeping the `String getString(String key, String
> defaultValue)` if more people favor it. However, we should let the
> user know that we encourage using ConfigOptions over the string-based
> configuration key, as Timo said, we should add the message to `String
> getString(String key, String defaultValue)` method.
>
> Best,
> Xuannan
>
> On Tue, Dec 19, 2023 at 7:55 PM Rui Fan <1996fan...@gmail.com> wrote:
> >
> > > > I noticed that Configuration is used in
> > > > DistributedCache#writeFileInfoToConfig and readFileInfoFromConfig
> > > > to store some cacheFile meta-information. Their keys are
> > > > temporary(key name with number) and it is not convenient
> > > > to predefine ConfigOption.
> > >
> > >
> > > True, this one requires a bit more effort to migrate from string-key to
> > > ConfigOption, but still should be doable. Looking at how the two
> mentioned
> > > methods are implemented and used, it seems what is really needed is
> > > serialization and deserialization of `DistributedCacheEntry`-s. And
> all the
> > > entries are always written / read at once. So I think we can serialize
> the
> > > whole set of entries into a JSON string (or something similar), and
> use one
> > > ConfigOption with a deterministic key for it, rather than having one
> > > ConfigOption for each field in each entry. WDYT?
> >
> > Hi Xintong, thanks for the good suggestion! Most of the entries can be
> > serialized to a json string, and we can only write/read them at once.
> > The CACHE_FILE_BLOB_KEY is special, its type is byte[], we need to
> > store it by the setBytes/getBytes.
> >
> > Also, I have an offline discussion with @Xuannan Su : refactoring all
> code
> > that uses String as key requires a separate FLIP. And we will provide
> > detailed FLIP  later.
> >
> >
> --
> >
> > Hi all, thanks everyone for the lively discussion. It's really a
> trade-off to
> > keep "String getString(String key, String defaultValue)" or not.
> > (It's not a right or wrong thing.)
> >
> > Judging from the discussion, most discussants can accept that keeping
> > `String getString(String key, String defaultValue)` and depreate the
> > rest of `getXxx(String key, Xxx defaultValue)`.
> >
> > cc @Xintong Song @Xuannan Su , WDYT?
> >
> > Best,
> > Rui
> >
> > On Fri, Dec 15, 2023 at 11:38 AM Zhu Zhu  wrote:
> >>
> >> I think it's not clear whether forcing using ConfigOption would hurt
> >> the user experience.
> >>
> >> Maybe it does at the beginning, because using string keys to access
> >> Flink configuration can be simpler for new components/jobs.
> >> However, problems may happen later if the configuration usages become
> >> more complex, like key renaming, using types other than strings, and
> >> other problems that ConfigOption was invented to address.
> >>
> >> Personally I prefer to encourage the usage of ConfigOption.
> >> Jobs should use GlobalJobParameter for custom config, which is different
> >> from the Configuration interface. Therefore, Configuration is mostly
> >> used in other components/plugins, in which case the long-term
> maintenance
> >> can be important.
> >>
> >> However, since it is not a right or wrong choice, I'd also be fine
> >> to keep the `getString()` method if more devs/users are in favor of it.
> >>
> >> Thanks,
> >> Zhu
> >>
> >> Timo Walther  于2023年12月14日周四 17:41写道：
> >>
> >> > The configuration in Flink is complicated and I fear we won't have
> >> > enough capacity to substantially fix it. The introduction of
> >> > ReadableConfig, WritableConfig, and typed ConfigOptions was a right
> step
> >> > into making the code more maintainable. From the Flink side, every
> read
> >> > access should go through ConfigOption.
> >> >
> >> > However, I also understand Gyula pragmatism here because (practically
> >> > speaking) users get access `getString()` via `toMap().get()`. So I'm
> >> > fine with removing the deprecation for functionality that is available
> >> > anyways. We should, however, add the message to `getString()` that
> this
> >> > method 

Re: [VOTE] FLIP-382: Unify the Provision of Diverse Metadata for Context-like APIs

[Xintong Song]

+1 (binding)

Best,

Xintong



On Mon, Dec 18, 2023 at 3:05 PM Lijie Wang  wrote:

> +1 (binding)
>
> Best,
> Lijie
>
> Yuxin Tan  于2023年12月15日周五 17:14写道：
>
> > +1 (non-binding)
> >
> > Best,
> > Yuxin
> >
> >
> > weijie guo  于2023年12月15日周五 10:05写道：
> >
> > > +1(binding)
> > >
> > > Best regards,
> > >
> > > Weijie
> > >
> > >
> > > Wencong Liu  于2023年12月15日周五 09:13写道：
> > >
> > > > Hi dev,
> > > >
> > > > I'd like to start a vote on FLIP-382.
> > > >
> > > > Discussion thread:
> > > > https://lists.apache.org/thread/3mgsc31odtpmzzl32s4oqbhlhxd0mn6b
> > > > FLIP:
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-382%3A+Unify+the+Provision+of+Diverse+Metadata+for+Context-like+APIs
> > > >
> > > > Best regards,
> > > > Wencong Liu
> > >
> >
>

Re: [VOTE] FLIP-380: Support Full Partition Processing On Non-keyed DataStream

[Xintong Song]

+1 (binding)


Best,

Xintong



On Fri, Dec 15, 2023 at 5:15 PM weijie guo 
wrote:

> +1 (binding)
>
> Best regards,
>
> Weijie
>
>
> Wencong Liu  于2023年12月15日周五 09:15写道：
>
> > Hi dev,
> >
> > I'd like to start a vote on FLIP-380.
> >
> > Discussion thread:
> > https://lists.apache.org/thread/nn7myj7vsvytbkdrnbvj5h0homsjrn1h
> > FLIP:
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-380%3A+Support+Full+Partition+Processing+On+Non-keyed+DataStream
> >
> > Best regards,
> > Wencong Liu
>

Re: [DISCUSS] FLIP-398: Improve Serialization Configuration And Usage In Flink

[Xintong Song]

Hi Ken,

I think the main purpose of this FLIP is to change how users interact with
the knobs for customizing the serialization behaviors, from requiring code
changes to working with pure configurations. Redesigning the knobs (i.e.,
names, semantics, etc.), on the other hand, is not the purpose of this
FLIP. Preserving the existing names and semantics should also help minimize
the migration cost for existing users. Therefore, I'm in favor of not
changing them.

Concerning decoupling from Kryo, and introducing other serialization
frameworks like Fury, I think that's a bigger topic that is worth further
discussion. At the moment, I'm not aware of any community consensus on
doing so. And even if in the future we decide to do so, the changes needed
should be the same w/ or w/o this FLIP. So I'd suggest not to block this
FLIP on these issues.

WDYT?

Best,

Xintong



On Fri, Dec 15, 2023 at 1:40 AM Ken Krugler 
wrote:

> Hi Yong,
>
> Looks good, thanks for creating this.
>
> One comment - related to my recent email about Fury, I would love to see
> the v2 serialization decoupled from Kryo.
>
> As part of that, instead of using xxxKryo in methods, call them xxxGeneric.
>
> A more extreme change would be to totally rely on Fury (so no more POJO
> serializer). Fury is faster than the POJO serializer in my tests, but this
> would be a much bigger change.
>
> Though it could dramatically simplify the Flink serialization support.
>
> — Ken
>
> PS - a separate issue is how to migrate state from Kryo to something like
> Fury, which supports schema evolution. I think this might be possible, by
> having a smarter deserializer that identifies state as being created by
> Kryo, and using (shaded) Kryo to deserialize, while still writing as Fury.
>
> > On Dec 6, 2023, at 6:35 PM, Yong Fang  wrote:
> >
> > Hi devs,
> >
> > I'd like to start a discussion about FLIP-398: Improve Serialization
> > Configuration And Usage In Flink [1].
> >
> > Currently, users can register custom data types and serializers in Flink
> > jobs through various methods, including registration in code,
> > configuration, and annotations. These lead to difficulties in upgrading
> > Flink jobs and priority issues.
> >
> > In flink-2.0 we would like to manage job data types and serializers
> through
> > configurations. This FLIP will introduce a unified option for data type
> and
> > serializer and users can configure all custom data types and
> > pojo/kryo/custom serializers. In addition, this FLIP will add more
> built-in
> > serializers for complex data types such as List and Map, and optimize the
> > management of Avro Serializers.
> >
> > Looking forward to hearing from you, thanks!
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-398%3A+Improve+Serialization+Configuration+And+Usage+In+Flink
> >
> > Best,
> > Fang Yong
>
> --
> Ken Krugler
> http://www.scaleunlimited.com
> Custom big data solutions
> Flink & Pinot
>
>
>
>

Re: [DISCUSS] Should Configuration support getting value based on String key?

[Xintong Song]

Hi Gyula,

First of all, even if we remove the `getXXX(String key, XXX defaultValue)`
methods, there are still several ways to access the configuration with
string-keys.

   - If one wants to access a specific option, as Rui mentioned,
   `ConfigOptions.key("xxx").stringType().noDefaultValue()` can be used. TBH,
   I can't think of a use case where a temporally created ConfigOption is
   preferred over a predefined one. Do you have any examples for that?
   - If one wants to access the whole configuration set, then `toMap` or
   `iterator` might be helpful.

It is true that these ways are less convenient than `getXXX(String key, XXX
defaultValue)`, and that's exactly my purpose, to make the key-string less
convenient so that developers choose ConfigOption over it whenever is
possible.

there will always be cases where a more flexible
> dynamic handling is necessary without the added overhead of the toMap logic
>

I'm not sure about this. I agree there are cases where flexible and dynamic
handling is needed, but maybe "without the added overhead of the toMap
logic" is not that necessary?

I'd think of this as "encouraging developers to use ConfigOption as much as
possible" vs. "a bit less convenient in 5% of the cases". I guess there's
no right and wrong, just different engineer opinions. While I'm personally
stand with removing the string-key access methods, I'd also be fine with
the other way if there are more people in favor of it.

Best,

Xintong



On Thu, Dec 14, 2023 at 3:45 PM Gyula Fóra  wrote:

> Hi Xintong,
>
> I don’t really see the actual practical benefit from removing the getstring
> and setstring low level methods.
>
> I understand that ConfigOptions are nicer for 95% of the cases but from a
> technical point of view there will always be cases where a more flexible
> dynamic handling is necessary without the added overhead of the toMap
> logic.
>
> I think it’s the most natural thing for any config abstraction to expose
> basic get set methods with a simple key.
>
> What do you think?
>
> Cheers
> Gyula
>
> On Thu, 14 Dec 2023 at 08:00, Xintong Song  wrote:
>
> > >
> > > IIUC, you both prefer using ConfigOption instead of string keys for
> > > all use cases, even internal ones. We can even forcefully delete
> > > these @Depreated methods in Flink-2.0 to guide users or
> > > developers to use ConfigOption.
> > >
> >
> > Yes, at least from my side.
> >
> >
> > I noticed that Configuration is used in
> > > DistributedCache#writeFileInfoToConfig and readFileInfoFromConfig
> > > to store some cacheFile meta-information. Their keys are
> > > temporary(key name with number) and it is not convenient
> > > to predefine ConfigOption.
> > >
> >
> > True, this one requires a bit more effort to migrate from string-key to
> > ConfigOption, but still should be doable. Looking at how the two
> mentioned
> > methods are implemented and used, it seems what is really needed is
> > serialization and deserialization of `DistributedCacheEntry`-s. And all
> the
> > entries are always written / read at once. So I think we can serialize
> the
> > whole set of entries into a JSON string (or something similar), and use
> one
> > ConfigOption with a deterministic key for it, rather than having one
> > ConfigOption for each field in each entry. WDYT?
> >
> >
> > If everyone agrees with this direction, we can start to refactor all
> > > code that uses getXxx(String key, String defaultValue) into
> > > getXxx(ConfigOption configOption), and completely
> > > delete all getXxx(String key, String defaultValue) in 2.0.
> > > And I'm willing to pick it up~
> > >
> >
> > Exactly. Actually, Xuannan and a few other colleagues are working on
> > reviewing all the existing configuration options. We identified some
> common
> > issues that can potentially be improved, and not using string-key is one
> of
> > them. We are still summarizing the findings, and will bring them to the
> > community for discussion once ready. Helping hands is definitely
> welcomed.
> > :)
> >
> >
> > Yeah, `toMap` can solve the problem, and I also mentioned it in the
> > > initial mail.
> > >
> >
> > True. Sorry I overlooked it.
> >
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Thu, Dec 14, 2023 at 2:14 PM Rui Fan <1996fan...@gmail.com> wrote:
> >
> > > Thanks Xintong and Xuannan for the feedback!
> > >
> > > IIUC, you both prefer using ConfigOption instead of stri

Re: [VOTE] FLIP-383: Support Job Recovery for Batch Jobs

[Xintong Song]

+1 (binding)

Best,

Xintong



On Thu, Dec 14, 2023 at 3:15 PM Lijie Wang  wrote:

> Hi devs, Thanks for all feedback about the FLIP-383: Support Job Recovery
> for Batch Jobs[1]. This FLIP was discussed in [2].
>
> I'd like to start a vote for it. The vote will be open for at least 72
> hours (until December 19th 12:00 GMT) unless there is an objection or
> insufficient votes. [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-383%3A+Support+Job+Recovery+for+Batch+Jobs
>
> [2] https://lists.apache.org/thread/074z237c07vtj74685nxo6bttkq3kshz
>   Best, Lijie
>

Re: [DISCUSS] Should Configuration support getting value based on String key?

[Xintong Song]

>
> IIUC, you both prefer using ConfigOption instead of string keys for
> all use cases, even internal ones. We can even forcefully delete
> these @Depreated methods in Flink-2.0 to guide users or
> developers to use ConfigOption.
>

Yes, at least from my side.


I noticed that Configuration is used in
> DistributedCache#writeFileInfoToConfig and readFileInfoFromConfig
> to store some cacheFile meta-information. Their keys are
> temporary(key name with number) and it is not convenient
> to predefine ConfigOption.
>

True, this one requires a bit more effort to migrate from string-key to
ConfigOption, but still should be doable. Looking at how the two mentioned
methods are implemented and used, it seems what is really needed is
serialization and deserialization of `DistributedCacheEntry`-s. And all the
entries are always written / read at once. So I think we can serialize the
whole set of entries into a JSON string (or something similar), and use one
ConfigOption with a deterministic key for it, rather than having one
ConfigOption for each field in each entry. WDYT?


If everyone agrees with this direction, we can start to refactor all
> code that uses getXxx(String key, String defaultValue) into
> getXxx(ConfigOption configOption), and completely
> delete all getXxx(String key, String defaultValue) in 2.0.
> And I'm willing to pick it up~
>

Exactly. Actually, Xuannan and a few other colleagues are working on
reviewing all the existing configuration options. We identified some common
issues that can potentially be improved, and not using string-key is one of
them. We are still summarizing the findings, and will bring them to the
community for discussion once ready. Helping hands is definitely welcomed.
:)


Yeah, `toMap` can solve the problem, and I also mentioned it in the
> initial mail.
>

True. Sorry I overlooked it.


Best,

Xintong



On Thu, Dec 14, 2023 at 2:14 PM Rui Fan <1996fan...@gmail.com> wrote:

> Thanks Xintong and Xuannan for the feedback!
>
> IIUC, you both prefer using ConfigOption instead of string keys for
> all use cases, even internal ones. We can even forcefully delete
> these @Depreated methods in Flink-2.0 to guide users or
> developers to use ConfigOption.
>
> Using ConfigOption is feasible in all scenarios because ConfigOption
> can be easily created via
> `ConfigOptions.key("xxx").stringType().noDefaultValue()` even if
> there is no predefined ConfigOption.
>
> I noticed that Configuration is used in
> DistributedCache#writeFileInfoToConfig and readFileInfoFromConfig
> to store some cacheFile meta-information. Their keys are
> temporary(key name with number) and it is not convenient
> to predefine ConfigOption.
>
> If everyone agrees with this direction, we can start to refactor all
> code that uses getXxx(String key, String defaultValue) into
> getXxx(ConfigOption configOption), and completely
> delete all getXxx(String key, String defaultValue) in 2.0.
> And I'm willing to pick it up~
>
> To Xintong:
>
> > I think a `toMap` as suggested by Zhu and Xuannan should solve the
> > problem. Alternatively, we may also consider providing an iterator for
> the
> > Configuration.
>
> Yeah, `toMap` can solve the problem, and I also mentioned it in the
> initial mail.
>
> Also Providing an iterator is fine, but we don't have a strong
> requirement for now. Simple is better, how about considering it
> if we have other strong requirements in the future?
>
> Looking forwarding to your feedback, thanks~
>
> Best,
> Rui
>
> On Thu, Dec 14, 2023 at 11:36 AM Xintong Song 
> wrote:
>
>> I'm leaning towards not allowing string-key based configuration access in
>> the long term.
>>
>> I see the Configuration being used in two different ways:
>>
>> 1. Writing / reading a specific option. In such cases, ConfigOption has
>> many advantages compared to string-key, such as explicit value type,
>> descriptions, default values, deprecated / fallback keys. I think we
>> should
>> encourage, and maybe even enforce, choosing ConfigOption over string-keys
>> in such specific option access scenarios. That also applies to internal
>> usages, for which the description may not be necessary because we won't
>> generate documentation from it but we can still benefit from other
>> advantages.
>>
>> 2. Iterating over all the configured options. In such cases, it is
>> currently impractical to find the proper ConfigOption for each configured
>> option. I think a `toMap` as suggested by Zhu and Xuannan should solve the
>> problem. Alternatively, we may also consider providing an iterator for the
>> Configuration.
>>
>> I think if we want to encou

Re: [DISCUSS] Should Configuration support getting value based on String key?

[Xintong Song]

I'm leaning towards not allowing string-key based configuration access in
the long term.

I see the Configuration being used in two different ways:

1. Writing / reading a specific option. In such cases, ConfigOption has
many advantages compared to string-key, such as explicit value type,
descriptions, default values, deprecated / fallback keys. I think we should
encourage, and maybe even enforce, choosing ConfigOption over string-keys
in such specific option access scenarios. That also applies to internal
usages, for which the description may not be necessary because we won't
generate documentation from it but we can still benefit from other
advantages.

2. Iterating over all the configured options. In such cases, it is
currently impractical to find the proper ConfigOption for each configured
option. I think a `toMap` as suggested by Zhu and Xuannan should solve the
problem. Alternatively, we may also consider providing an iterator for the
Configuration.

I think if we want to encourage using ConfigOption in case-1, the most
effective way is to not allow accessing a specific option with a
string-key, so that developers not awaring of this discussion won't
accidentally use it. On the other hand, case-2 is a much rarer use case
compared to case-1, and given the fact that there are alternative
approaches, I think we should not compromise case-1 for it.

WDYT?

Best,

Xintong



On Thu, Dec 14, 2023 at 10:25 AM Xuannan Su  wrote:

> Hi Rui,
>
> We are currently revisiting all the configurations for Flink 2.0, and
> it turns out that many string-based configurations in
> `ConfigConstants` are deprecated and have been replaced by
> `ConfigOptions`. Since `ConfigOptions` offers many advantages over
> string-based configurations for the end user, I believe we should
> encourage users to set and get the Flink configuration exclusively
> with `ConfigOption`. And we are going to eventually replace all the
> string-based configurations with `ConfigOptions` for this use case.
>
> For the first use case you mentioned, I think they are all internal usage,
> and we should aim to replace them with ConfigOptions gradually.
> Meanwhile, we may consider making those getters/setters for internal
> use only while the replacement is in progress.
>
> For the second use case, IIUC, you need to iterate over all the
> configurations to replace some old configuration keys with new ones. I
> believe  `toMap` is suitable for this scenario.
>
> Best,
> Xuannan
>
>
>
> On Wed, Dec 13, 2023 at 9:04 PM Rui Fan <1996fan...@gmail.com> wrote:
> >
> > Thanks Zhu for the quick response!
> >
> > > It is not a blocker of the deprecation, epsecially given that they are
> > not standard
> > > configuration and are just using Configuration class for convenience.
> >
> > Yes, it's not a blocker of deprecation.
> >
> > > These are internal usages and we can have an internal getter method for
> > them.
> >
> > For case1, do you mean we reuse the old getString method as the internal
> > getter method or add a new getter method?
> >
> > Anyway, it's fine for me if we have an internal getter method. As I
> > understand,
> > the public method without any annotation will be the internal method,
> right?
> >
> > > Not sure why it's required to convert all keys or values. If it is used
> > > to create strings for dynamic properties or config files to deploy
> jobs,
> > > toMap()/toFileWritableMap() may be a better choice. They are already
> > > used in this kind of scenarios.
> >
> > For case2, it's really a special scenario, and toMap() is fine for case2.
> > Case2 uses the getString instead of toMap due to getString is easier.
> > Also, kubernetes-operator is also a internal usage, if case1 is solved,
> > case2 also can use the internal getter method.So we can focus on case1.
> >
> > Thank you
> >
> > Best,
> > Rui
> >
> > On Wed, Dec 13, 2023 at 8:01 PM Zhu Zhu  wrote:
> >
> > > Hi Rui,
> > >
> > > I'd like to understand why there is a strong requirement for these
> > > deprecated
> > > methods. The ConfigOption param methods help to do the type conversion
> so
> > > that users do not need to do it by themselves.
> > >
> > > For the 2 reasons to keep these methods mentioned above:
> > >
> > > > 1. A lot of scenarios don't define the ConfigOption, they using
> > > String as the key and value directly, such as: StreamConfig,
> > > TaskConfig, DistributedCache, etc.
> > >
> > > These are internal usages and we can have an internal getter method for
> > > them.
> > > It is not a blocker of the deprecation, epsecially given that they are
> not
> > > standard
> > > configuration and are just using Configuration class for convenience.
> > >
> > > > 2. Some code wanna convert all keys or values, this converting
> > > is generic, so the getString(String key, String defaultValue) is
> needed.
> > > Such as: kubernetes-operator [3].
> > >
> > > Not sure why it's required to convert all keys or values. If it is used
> > > to create strings for dynamic properties or c

Re: [VOTE] FLIP-396: Trial to test GitHub Actions as an alternative for Flink's current Azure CI infrastructure

[Xintong Song]

+1 (binding)

Best,

Xintong



On Thu, Dec 14, 2023 at 10:05 AM Jiabao Sun 
wrote:

> Thanks Matthias for this hard work!
>
> +1(non-binding)
>
> Best,
> Jiabao
>
>
> > 2023年12月14日 09:57，Leonard Xu  写道：
> >
> > +1(binding)
> >
> >
> > Best,
> > Leonard
> >
> >> 2023年12月13日 下午10:59，Benchao Li  写道：
> >>
> >> +1 (binding)
> >>
> >> Thanks Matthias for driving it!
> >>
> >> Etienne Chauchot  于2023年12月13日周三 21:35写道：
> >>>
> >>> Thanks Matthias for your hard work !
> >>>
> >>> +1 (binding)
> >>>
> >>> Best
> >>>
> >>> Etienne
> >>>
> >>> Le 12/12/2023 à 11:23, Lincoln Lee a écrit :
>  +1 (binding)
> 
>  Thanks for driving this!
> 
>  Best,
>  Lincoln Lee
> 
> 
>  Yun Tang  于2023年12月12日周二 17:52写道：
> 
> > Thanks for Matthias driving this work!
> >
> > +1 (binding)
> >
> > Best
> > Yun Tang
> > 
> > From: Yangze Guo
> > Sent: Tuesday, December 12, 2023 16:12
> > To:dev@flink.apache.org  
> > Subject: Re: [VOTE] FLIP-396: Trial to test GitHub Actions as an
> > alternative for Flink's current Azure CI infrastructure
> >
> > +1 (binding)
> >
> > Best,
> > Yangze Guo
> >
> > On Tue, Dec 12, 2023 at 3:51 PM Yuxin Tan
> wrote:
> >> +1 (non binding)
> >> Thanks for the effort.
> >>
> >> Best,
> >> Yuxin
> >>
> >>
> >> Samrat Deb  于2023年12月12日周二 15:25写道：
> >>
> >>> +1 (non binding)
> >>> Thanks for driving
> >>>
> >>> On Tue, 12 Dec 2023 at 11:59 AM, Sergey Nuyanzin<
> snuyan...@gmail.com>
> >>> wrote:
> >>>
>  +1 (binding)
> 
>  Thanks for driving this
> 
>  On Tue, Dec 12, 2023, 07:22 Rui Fan<1996fan...@gmail.com>  wrote:
> 
> > +1(binding)
> >
> > Best,
> > Rui
> >
> > On Tue, Dec 12, 2023 at 11:58 AM weijie guo <
> > guoweijieres...@gmail.com
> > wrote:
> >
> >> Thanks Matthias for this efforts.
> >>
> >> +1(binding)
> >>
> >>
> >> Best regards,
> >>
> >> Weijie
> >>
> >>
> >> Matthias Pohl  于2023年12月11日周一
> >>> 21:51写道：
> >>> Hi everyone,
> >>> I'd like to start a vote on FLIP-396 [1]. It covers enabling
> > GitHub
> >> Actions
> >>> (GHA) in Apache Flink. This means that GHA workflows will run
> > aside
> > from
> >>> the usual Azure CI workflows in a trial phase (which ends
> > earliest
>  with
> >> the
> >>> release of Flink 1.19). Azure CI will still serve as the
> > project's
> > ground
> >>> of truth until the community decides in a final vote to switch
> > to
> >>> GHA
> > or
> >>> stick to Azure CI.
> >>>
> >>> The related discussion thread can be found in [2].
> >>>
> >>> The vote will remain open for at least 72 hours and only
> > concluded
> >>> if
> >> there
> >>> are no objections and enough (i.e. at least 3) binding votes.
> >>>
> >>> Matthias
> >>>
> >>> [1]
> >>>
> >>>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-396%3A+Trial+to+test+GitHub+Actions+as+an+alternative+for+Flink%27s+current+Azure+CI+infrastructure
> >>> [2]
> >>> https://lists.apache.org/thread/h4cmv7l3y8mxx2t435dmq4ltco4sbrgb
> >>> --
> >>>
> >>> [image: Aiven]
> >>>
> >>> *Matthias Pohl*
> >>> Opensource Software Engineer, *Aiven*
> >>> matthias.p...@aiven.io  |  +49 170 9869525
> >>> aiven.io|   <
> >> https://www.facebook.com/aivencloud
> >>>  <
> https://twitter.com/aiven_io>
> >>> *Aiven Deutschland GmbH*
> >>> Alexanderufer 3-7, 10117 Berlin
> >>> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> >>> Amtsgericht Charlottenburg, HRB 209739 B
> >>>
> >>
> >>
> >>
> >> --
> >>
> >> Best,
> >> Benchao Li
>
>

Re: [VOTE] FLIP-335: Deprecation and removal of Flink's Time classes in favor of Java's Duration

[Xintong Song]

+1 (binding)

I guess I missed the FLIP discussion. But the FLIP LGTM. Thanks for driving
this, Matthias.

Best,

Xintong



On Tue, Dec 12, 2023 at 8:33 PM Jing Ge  wrote:

> +1(binding)
>
> Thanks.
>
> Best regards,
> Jing
>
> On Tue, Dec 12, 2023 at 1:12 PM Zhu Zhu  wrote:
>
> > +1 (binding)
> >
> > Thanks,
> > Zhu
> >
> > Matthias Pohl  于2023年12月12日周二 01:32写道：
> >
> > > Hi everyone,
> > > I'd like to start a vote on FLIP-335 [1]. It covers the deprecation
> (and
> > > Flink 2.0-related removal) of Flink's Time classes in favor of Java's
> > > Duration class.
> > >
> > > The related discussion thread can be found in [2].
> > >
> > > The vote will remain open for at least 72 hours and only concluded if
> > there
> > > are no objections and enough (i.e. at least 3) binding votes.
> > >
> > > Matthias
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-335%3A+Removing+Flink%27s+Time+classes
> > > [2] https://lists.apache.org/thread/48ysrg1rrtl8s1twg9wmx35l201hnc2w
> > >
> > > --
> > >
> > > [image: Aiven] 
> > >
> > > *Matthias Pohl*
> > > Opensource Software Engineer, *Aiven*
> > > matthias.p...@aiven.io|  +49 170 9869525
> > > aiven.io    |   <
> > https://www.facebook.com/aivencloud
> > > >
> > >      <
> > > https://twitter.com/aiven_io>
> > > *Aiven Deutschland GmbH*
> > > Alexanderufer 3-7, 10117 Berlin
> > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > > Amtsgericht Charlottenburg, HRB 209739 B
> > >
> >
>

Re: [DISCUSS] FLIP-398: Improve Serialization Configuration And Usage In Flink

[Xintong Song]

Thanks for working on this, Yong.

The usability of the serialization mechanism has indeed been a pain for a
long time.

The proposed changes look good to me. +1 from my side.

Best,

Xintong



On Thu, Dec 7, 2023 at 10:36 AM Yong Fang  wrote:

> Hi devs,
>
> I'd like to start a discussion about FLIP-398: Improve Serialization
> Configuration And Usage In Flink [1].
>
> Currently, users can register custom data types and serializers in Flink
> jobs through various methods, including registration in code,
> configuration, and annotations. These lead to difficulties in upgrading
> Flink jobs and priority issues.
>
> In flink-2.0 we would like to manage job data types and serializers through
> configurations. This FLIP will introduce a unified option for data type and
> serializer and users can configure all custom data types and
> pojo/kryo/custom serializers. In addition, this FLIP will add more built-in
> serializers for complex data types such as List and Map, and optimize the
> management of Avro Serializers.
>
> Looking forward to hearing from you, thanks!
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-398%3A+Improve+Serialization+Configuration+And+Usage+In+Flink
>
> Best,
> Fang Yong
>

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

[Xintong Song]

A big +1 for this proposal.

Thanks Leonard and the Flink CDC community. While there are many further
details to be discussed, I believe this proposal is aligned with the
long-term interests for both communities.

Best,

Xintong



On Thu, Dec 7, 2023 at 12:06 PM Samrat Deb  wrote:

> That's really cool :)
> +1 for the great addition
>
> Bests,
> Samrat
>
> On Thu, 7 Dec 2023 at 9:20 AM, Jingsong Li  wrote:
>
> > Wow, Cool, Nice
> >
> > CDC is playing an increasingly important role.
> >
> > +1
> >
> > Best,
> > Jingsong
> >
> > On Thu, Dec 7, 2023 at 11:25 AM Leonard Xu  wrote:
> > >
> > > Dear Flink devs,
> > >
> > > As you may have heard, we at Alibaba (Ververica) are planning to donate
> > CDC Connectors for the Apache Flink project[1] to the Apache Flink
> > community.
> > >
> > > CDC Connectors for Apache Flink comprise a collection of source
> > connectors designed specifically for Apache Flink. These connectors[2]
> > enable the ingestion of changes from various databases using Change Data
> > Capture (CDC), most of these CDC connectors are powered by Debezium[3].
> > They support both the DataStream API and the Table/SQL API, facilitating
> > the reading of database snapshots and continuous reading of transaction
> > logs with exactly-once processing, even in the event of failures.
> > >
> > >
> > > Additionally, in the latest version 3.0, we have introduced many
> > long-awaited features. Starting from CDC version 3.0, we've built a
> > Streaming ELT Framework available for streaming data integration. This
> > framework allows users to write their data synchronization logic in a
> > simple YAML file, which will automatically be translated into a Flink
> > DataStreaming job. It emphasizes optimizing the task submission process
> and
> > offers advanced functionalities such as whole database synchronization,
> > merging sharded tables, and schema evolution[4].
> > >
> > >
> > > I believe this initiative is a perfect match for both sides. For the
> > Flink community, it presents an opportunity to enhance Flink's
> competitive
> > advantage in streaming data integration, promoting the healthy growth and
> > prosperity of the Apache Flink ecosystem. For the CDC Connectors project,
> > becoming a sub-project of Apache Flink means being part of a neutral
> > open-source community, which can attract a more diverse pool of
> > contributors.
> > >
> > > Please note that the aforementioned points represent only some of our
> > motivations and vision for this donation. Specific future operations need
> > to be further discussed in this thread. For example, the sub-project name
> > after the donation; we hope to name it Flink-CDC aiming to streaming data
> > intergration through Apache Flink, following the naming convention of
> > Flink-ML; And this project is managed by a total of 8 maintainers,
> > including 3 Flink PMC members and 1 Flink Committer. The remaining 4
> > maintainers are also highly active contributors to the Flink community,
> > donating this project to the Flink community implies that their
> permissions
> > might be reduced. Therefore, we may need to bring up this topic for
> further
> > discussion within the Flink PMC. Additionally, we need to discuss how to
> > migrate existing users and documents. We have a user group of nearly
> 10,000
> > people and a multi-version documentation site need to migrate. We also
> need
> > to plan for the migration of CI/CD processes and other specifics.
> > >
> > >
> > > While there are many intricate details that require implementation, we
> > are committed to progressing and finalizing this donation process.
> > >
> > >
> > > Despite being Flink’s most active ecological project (as evaluated by
> > GitHub metrics), it also boasts a significant user base. However, I
> believe
> > it's essential to commence discussions on future operations only after
> the
> > community reaches a consensus on whether they desire this donation.
> > >
> > >
> > > Really looking forward to hear what you think!
> > >
> > >
> > > Best,
> > > Leonard (on behalf of the Flink CDC Connectors project maintainers)
> > >
> > > [1] https://github.com/ververica/flink-cdc-connectors
> > > [2]
> >
> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html
> > > [3] https://debezium.io
> > > [4]
> >
> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html
> >
>

Re: [DISCUSS] FLIP-383: Support Job Recovery for Batch Jobs

[Xintong Song]

@Paul,


Do you mean the scenario where users call `evn.execute()` multiple times in
the `main()` method? I believe that is not supported currently when HA is
enabled, for the exact same reason you mentioned that Flink is not aware of
which jobs are executed and which are not.


On the other hand, if an external scheduler is used to submit multiple jobs
to a session cluster, Flink already has a JobResultStore for persisting
information about successfully completed jobs, so that only in-completed
jobs will be recovered. See FLIP-194[1] for more details.


Best,

Xintong


[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-194%3A+Introduce+the+JobResultStore

On Tue, Dec 5, 2023 at 6:01 PM Xintong Song  wrote:

> Thanks for addressing my comments, Lijie. LGTM
>
> Best,
>
> Xintong
>
>
>
> On Tue, Dec 5, 2023 at 2:56 PM Paul Lam  wrote:
>
>> Hi Lijie,
>>
>> Recovery for batch jobs is no doubt a long-awaited feature. Thanks for
>> the proposal!
>>
>> I’m concerned about the multi-job scenario. In session mode, users could
>> use web submission to upload and run jars which may produce multiple
>> Flink jobs. However, these jobs may not be submitted at once and run in
>> parallel. Instead, they could be dependent on other jobs like a DAG. The
>> schedule of the jobs is controlled by the user's main method.
>>
>> IIUC, in the FLIP, the main method is lost after the recovery, and only
>> submitted jobs would be recovered. Is that right?
>>
>> Best,
>> Paul Lam
>>
>> > 2023年11月2日 18:00，Lijie Wang  写道：
>> >
>> > Hi devs,
>> >
>> > Zhu Zhu and I would like to start a discussion about FLIP-383: Support
>> Job
>> > Recovery for Batch Jobs[1]
>> >
>> > Currently, when Flink’s job manager crashes or gets killed, possibly
>> due to
>> > unexpected errors or planned nodes decommission, it will cause the
>> > following two situations:
>> > 1. Failed, if the job does not enable HA.
>> > 2. Restart, if the job enable HA. If it’s a streaming job, the job will
>> be
>> > resumed from the last successful checkpoint. If it’s a batch job, it
>> has to
>> > run from beginning, all previous progress will be lost.
>> >
>> > In view of this, we think the JM crash may cause great regression for
>> batch
>> > jobs, especially long running batch jobs. This FLIP is mainly to solve
>> this
>> > problem so that batch jobs can recover most job progress after JM
>> crashes.
>> > In this FLIP, our goal is to let most finished tasks not need to be
>> re-run.
>> >
>> > You can find more details in the FLIP-383[1]. Looking forward to your
>> > feedback.
>> >
>> > [1]
>> >
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-383%3A+Support+Job+Recovery+for+Batch+Jobs
>> >
>> > Best,
>> > Lijie
>>
>>

Re: [DISCUSS] FLIP-383: Support Job Recovery for Batch Jobs

[Xintong Song]

Thanks for addressing my comments, Lijie. LGTM

Best,

Xintong



On Tue, Dec 5, 2023 at 2:56 PM Paul Lam  wrote:

> Hi Lijie,
>
> Recovery for batch jobs is no doubt a long-awaited feature. Thanks for
> the proposal!
>
> I’m concerned about the multi-job scenario. In session mode, users could
> use web submission to upload and run jars which may produce multiple
> Flink jobs. However, these jobs may not be submitted at once and run in
> parallel. Instead, they could be dependent on other jobs like a DAG. The
> schedule of the jobs is controlled by the user's main method.
>
> IIUC, in the FLIP, the main method is lost after the recovery, and only
> submitted jobs would be recovered. Is that right?
>
> Best,
> Paul Lam
>
> > 2023年11月2日 18:00，Lijie Wang  写道：
> >
> > Hi devs,
> >
> > Zhu Zhu and I would like to start a discussion about FLIP-383: Support
> Job
> > Recovery for Batch Jobs[1]
> >
> > Currently, when Flink’s job manager crashes or gets killed, possibly due
> to
> > unexpected errors or planned nodes decommission, it will cause the
> > following two situations:
> > 1. Failed, if the job does not enable HA.
> > 2. Restart, if the job enable HA. If it’s a streaming job, the job will
> be
> > resumed from the last successful checkpoint. If it’s a batch job, it has
> to
> > run from beginning, all previous progress will be lost.
> >
> > In view of this, we think the JM crash may cause great regression for
> batch
> > jobs, especially long running batch jobs. This FLIP is mainly to solve
> this
> > problem so that batch jobs can recover most job progress after JM
> crashes.
> > In this FLIP, our goal is to let most finished tasks not need to be
> re-run.
> >
> > You can find more details in the FLIP-383[1]. Looking forward to your
> > feedback.
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-383%3A+Support+Job+Recovery+for+Batch+Jobs
> >
> > Best,
> > Lijie
>
>

Re: [DISCUSS] FLIP-383: Support Job Recovery for Batch Jobs

[Xintong Song]

Thanks for the proposal, Lijie and Zhu.

I have been having offline discussions with the Apache Celeborn folks
regarding integrating Apache Celeborn into Flink's Hybrid Shuffle mode. One
thing coming from those discussions that might relate to this FLIP is that
Celeborn maintains some internal states inside its LifecycleManager (think
of this as a component resident in Flink's Shuffle Master), which would
also need persistent and recovery in order for the partitions to be reused
after a JM crash. Given that Flink supports pluggable shuffle services,
there could be other custom shuffle services with similar demands. I wonder
if it makes sense to also add interfaces that take snapshots from Shuffle
Master once a while, and provide such snapshots to Shuffle Master upon
recovery?

Best,

Xintong



On Thu, Nov 30, 2023 at 5:48 PM Lijie Wang  wrote:

> Hi Guowei,
>
> Thanks for your feedback.
>
> >> As far as I know, there are multiple job managers on standby in some
> scenarios. In this case, is your design still effective?
> I think it's still effective. There will only be one leader. After becoming
> the leader, the startup process of JobMaster is the same as only one
> jobmanger restarts, so I think the current process should also be
> applicable to multi-jobmanager situation. We will also do some tests to
> cover this case.
>
> >> How do you rule out that there might still be some states in the memory
> of the original operator coordinator?
> Current restore process is the same as steraming jobs restore from
> checkpoint(call the same methods) after failover, which is widely used in
> production, so I think there is no problem.
>
> >> Additionally, using NO_CHECKPOINT seems a bit odd. Why not use a normal
> checkpoint ID greater than 0 and record it in the event store?
> We use -1(NO_CHECKPOINT) to distinguish it from a normal checkpoint, -1
> indicates that this is a snapshot for the no-checkpoint/batch scenarios.
>
> Besides, considering that currently some operator coordinators may not
> support taking snapshots in the no-checkpint/batch scenarios (or don't
> support passing -1 as a checkpoint id), we think it is better to let the
> developer explicitly specify whether it supports snapshots in the batch
> scenario. Therefore, we intend to introduce the "SupportsBatchSnapshot"
> interface for split enumerator and the "supportsBatchSnapshot" method for
> operator coordinator. You can find more details in FLIP "Introduce
> SupportsBatchSnapshot interface" and "JobEvent" sections.
>
> Looking forward to your further feedback.
>
> Best,
> Lijie
>
> Guowei Ma  于2023年11月19日周日 10:47写道：
>
> > Hi,
> >
> >
> > This is a very good proposal, as far as I know, it can solve some very
> > critical production operations in certain scenarios. I have two minor
> > issues:
> >
> > As far as I know, there are multiple job managers on standby in some
> > scenarios. In this case, is your design still effective? I'm unsure if
> you
> > have conducted any tests. For instance, standby job managers might take
> > over these failed jobs more quickly.
> > Regarding the part about the operator coordinator, how can you ensure
> that
> > the checkpoint mechanism can restore the state of the operator
> coordinator:
> > For example:
> > How do you rule out that there might still be some states in the memory
> of
> > the original operator coordinator? After all, the implementation was done
> > under the assumption of scenarios where the job manager doesn't fail.
> > Additionally, using NO_CHECKPOINT seems a bit odd. Why not use a normal
> > checkpoint ID greater than 0 and record it in the event store?
> > If the issues raised in point 2 cannot be resolved in the short term,
> would
> > it be possible to consider not supporting failover with a source job
> > manager?
> >
> > Best,
> > Guowei
> >
> >
> > On Thu, Nov 2, 2023 at 6:01 PM Lijie Wang 
> > wrote:
> >
> > > Hi devs,
> > >
> > > Zhu Zhu and I would like to start a discussion about FLIP-383: Support
> > Job
> > > Recovery for Batch Jobs[1]
> > >
> > > Currently, when Flink’s job manager crashes or gets killed, possibly
> due
> > to
> > > unexpected errors or planned nodes decommission, it will cause the
> > > following two situations:
> > > 1. Failed, if the job does not enable HA.
> > > 2. Restart, if the job enable HA. If it’s a streaming job, the job will
> > be
> > > resumed from the last successful checkpoint. If it’s a batch job, it
> has
> > to
> > > run from beginning, all previous progress will be lost.
> > >
> > > In view of this, we think the JM crash may cause great regression for
> > batch
> > > jobs, especially long running batch jobs. This FLIP is mainly to solve
> > this
> > > problem so that batch jobs can recover most job progress after JM
> > crashes.
> > > In this FLIP, our goal is to let most finished tasks not need to be
> > re-run.
> > >
> > > You can find more details in the FLIP-383[1]. Looking forward to your
> > > feedback.
> > >
> > > [1]
> > >
> > >
> 

Re: [DISCUSS] FLIP-395: Migration to GitHub Actions

[Xintong Song]

Thanks for the efforts, Matthias.


I think it would be helpful if we can at the end migrate the CI to an
ASF-managed Github Action, as long as it provides us a similar computation
capacity and stability. Given that the proposal is only to start a trial
and investigate whether the migration is feasible, I don't see much concern
in this.


I have only one suggestion and one question.

- Regarding the migration plan, I wonder if we should not disable the CI
bot until we fully decide to migrate to Github Actions? In case the nightly
runs don't really work well, it might be debatable whether we should
maintain the CI in two places (i.e. PRs on Github Actions and cron builds
on Azure).

- What exactly are the changes that would affect contributors during the
trial period? Is it only an additional CI report that you can potentially
just ignore? Or would there be some larger impacts, e.g. you cannot merge a
PR if the Github Action CI is not passed (I don't know, I just made this
up)?


Best,

Xintong



On Wed, Nov 29, 2023 at 8:07 PM Yuxin Tan  wrote:

> Ok, Thanks for the update and the explanations.
>
> Best,
> Yuxin
>
>
> Matthias Pohl  于2023年11月29日周三 15:43写道：
>
> > >
> > > According to the Flip, the new tests will support arm env.
> > > I believe that's good news for arm users. I have a minor
> > > question here. Will it be a blocker before migrating the new
> > > tests? If not,  If not, when can we expect arm environment
> > > support to be implemented? Thanks.
> >
> >
> > Thanks for your feedback, everyone.
> >
> > About the ARM support. I want to underline that this FLIP is not about
> > migrating to GitHub Actions but to start a trial run in the Apache Flink
> > repository. That would allow us to come up with a proper decision whether
> > GitHub Actions is what we want. I admit that the title is a bit
> > "clickbaity". I updated the FLIP's title and its Motivation to make
> things
> > clear.
> >
> > The FLIP suggests starting a trial period until 1.19 is released to try
> > things out. A proper decision on whether we want to migrate would be made
> > at the end of the 1.19 release cycle.
> >
> > About the ARM support: This related content of the FLIP is entirely based
> > on documentation from Apache INFRAs side. INFRA seems to offer this ARM
> > support for their ephemeral runners. The ephemeral runners are in the
> > testing stage, i.e. these runners are still experimental. Apache INFRA
> asks
> > Apache projects to join this test.
> >
> > Whether the ARM support is actually possible to achieve within Flink is
> > something we have to figure out as part of the trial run. One conclusion
> of
> > the trial run could be that we still move ahead with GHA but don't use
> arm
> > machines due to some blocking issues.
> >
> > Matthias
> >
> >
> >
> > On Wed, Nov 29, 2023 at 4:46 AM Yuxin Tan 
> wrote:
> >
> > > Hi, Matthias,
> > >
> > > Thanks for driving this.
> > > +1 from my side.
> > >
> > > According to the Flip, the new tests will support arm env.
> > > I believe that's good news for arm users. I have a minor
> > > question here. Will it be a blocker before migrating the new
> > > tests? If not,  If not, when can we expect arm environment
> > > support to be implemented? Thanks.
> > >
> > > Best,
> > > Yuxin
> > >
> > >
> > > Márton Balassi  于2023年11月29日周三 03:09写道：
> > >
> > > > Thanks, Matthias. Big +1 from me.
> > > >
> > > > On Tue, Nov 28, 2023 at 5:30 PM Matthias Pohl
> > > >  wrote:
> > > >
> > > > > Thanks for the pointer. I'm planning to join that meeting.
> > > > >
> > > > > On Tue, Nov 28, 2023 at 4:16 PM Etienne Chauchot <
> > echauc...@apache.org
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > FYI there is the ASF infra roundtable soon. One of the subjects
> for
> > > > this
> > > > > > session is GitHub Actions. It could be worth passing by:
> > > > > >
> > > > > > December 6th, 2023 at 1700 UTC on the #Roundtablechannel on
> Slack.
> > > > > >
> > > > > > For information about theroundtables, and about how to join,
> > > > > > see:https://infra.apache.org/roundtable.html
> > > > > > 
> > > > > >
> > > > > > Best
> > > > > >
> > > > > > Etienne
> > > > > >
> > > > > > Le 24/11/2023 à 14:16, Maximilian Michels a écrit :
> > > > > > > Thanks for reviving the efforts here Matthias! +1 for the
> > > transition
> > > > > > > to GitHub Actions.
> > > > > > >
> > > > > > > As for ASF Infra Jenkins, it works fine. Jenkins is extremely
> > > > > > > feature-rich. Not sure about the spare capacity though. I know
> > that
> > > > > > > for Apache Beam, Google donated a bunch of servers to get
> > > additional
> > > > > > > build capacity.
> > > > > > >
> > > > > > > -Max
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Nov 23, 2023 at 10:30 AM Matthias Pohl
> > > > > > >   wrote:
> > > > > > >> Btw. even though we've been focusing on GitHub Actions with
> this
> > > > FLIP,
> > > > > > I'm
> > > > > > >> curious whether somebody

Re: [VOTE] Release flink-shaded 16.2, release candidate #1

[Xintong Song]

+1 (binding)

- verified checksum and signature
- verified license and notice file
- verified source archive does not contain any binary
- reviewed the release notes and web pr

Best,

Xintong



On Mon, Nov 13, 2023 at 4:21 PM Yuxin Tan  wrote:

> Thanks weijie for driving the new release!
>
> +1 (non-binding)
>
> - Built from source code succeeded
> - Verified signatures
> - Verified hashsums
> - Reviewed the web PR
>
> Best,
> Yuxin
>
>
> weijie guo  于2023年11月13日周一 15:57写道：
>
> > Hi everyone,
> >
> > Please review and vote on the release candidate #1 for the version
> > 16.2, as follows:
> >
> > [ ] +1, Approve the release
> >
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> >
> >
> >
> >
> > The complete staging area is available for your review, which includes:
> >
> > * JIRA release notes [1],
> >
> > * the official Apache source release to be deployed to dist.apache.org
> >  [2], which are signed with the key with fingerprint
> > 8D56AE6E7082699A4870750EA4E8C4C05EE6861F [3],
> >
> > * all artifacts to be deployed to the Maven Central Repository [4],
> >
> > * source code tag "release-16.2-rc1" [5],
> >
> > * website pull request listing the new release [6].
> >
> >
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> > approval, with at least 3 PMC affirmative votes.
> >
> >
> >
> > Thanks,
> >
> > Release Manager
> >
> >
> >
> > [1] https://issues.apache.org/jira/projects/FLINK/versions/12353810
> >
> > [2] https://dist.apache.org/repos/dist/dev/flink/flink-shaded-16.2-rc1
> >
> > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> >
> > [4]
> > https://repository.apache.org/content/repositories/orgapacheflink-1671/
> >
> > [5] https://github.com/apache/flink-shaded/releases/tag/release-16.2-rc1
> >
> > [6] https://github.com/apache/flink-web/pull/697
> >
>