Re: 2.0 podcast

[Márton Balassi]

Hi Rich,

Thanks for reaching out, that is a great idea. Unfortunately I am not aware
of Flink folks that are joining the Denver conference this year.

Best,
Marton

On Thu, Oct 3, 2024 at 9:39 PM Rich Bowen  wrote:

> Hi, folks,
>
> I understand you have 2.0 coming up pretty soon. I was wondering if any of
> you will be in Denver for Community Over Code, and be willing to do an
> interview with me for feathercast.apache.org?
>
> Thanks!
>
> --Rich
>

Re: [VOTE] FLIP-474: Store operator name and UID in state metadata

[Márton Balassi]

+1 (binding)

Thanks for clarifying the big picture [1], that helps a lot.

[1]
https://docs.google.com/document/d/1Du1-TShoOjaNDCahs3sgLWIpYkXzJPdSkgHcWLpELyw/edit?usp=sharing

On Mon, Aug 26, 2024 at 10:02 AM Gabor Somogyi 
wrote:

> Hi All,
>
> I would like to start a vote on FLIP-474: Store operator name and UID in
> state metadata [1].
> The discussion thread can be found here [2].
>
> During the discussion a clear need formed to have an umbrella document
> about the global view
> which manifested in Flink state observability umbrella [3].
>
> The vote will be open for at least 72 hours unless there are any objections
> or insufficient votes.
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-474%3A+Store+operator+name+and+UID+in+state+metadata
> [2] https://lists.apache.org/thread/n927lv2r7r2rr71bbc379v9km6lo4v3t
> [3]
>
> https://docs.google.com/document/d/1Du1-TShoOjaNDCahs3sgLWIpYkXzJPdSkgHcWLpELyw/edit?usp=sharing
>
> BR,
> G
>

Re: [DISCUSS] FLIP-474: Store operator name and UID in state metadata

[Márton Balassi]

Hi team,

Thank you for the proposal, Gabor and the review, Zakelly. Let me provide
more context on the intended outcome, let us agree on the desirability of
that first and then we can come back to the technical details.

The pain point that we observed with multiple users is that their
operations teams cannot effectively read the Flink state. These folks are
usually not Java developers (unlike the team who wrote the application in
the first place), so we cannot reasonably expect them to be able to
efficiently use the State Processor API, especially not in high pressure
situations.

Providing a FlinkSQL interface for these folks would be ideal, especially
over specific checkpoints/savepoints. Spark has recently introduced this
feature [1] and we have users who are considering switching to Spark as a
result.

You can consider the motivation to be a better alternative to the
deprecated queryable state effort, which was never fully embraced, and
rightfully so it lacked security and put unnecessary load on the live Flink
application. I like Gabor's intended approach much more.

What do you think?

[1]
https://www.databricks.com/blog/announcing-state-reader-api-new-statestore-data-source

On Thu, Aug 8, 2024 at 12:06 PM Zakelly Lan  wrote:

> Hi Gabor,
>
> Thanks for the proposal! However, I find it a little strange. Are you
> saying the user can set the operator uid but then doesn't know what they
> set when debugging? Otherwise, is the `OperatorIdentifier.forUid("my-uid")`
> feasible? I understand your point about potential cross-team work, but the
> person may not be able to debug code that was not written by them. Things
> get complex in this scenario. Could you provide more details about the
> issue you are facing?
>
> Regarding the checkpoint, it is not designed to be self-contained or
> human-readable. I suggest not introducing such columns for debugging
> purposes.
>
>
> Best,
> Zakelly
>
> On Wed, Aug 7, 2024 at 10:07 PM Gabor Somogyi 
> wrote:
>
> > Hi Devs,
> >
> > I would like to start a discussion on FLIP-474: Store operator name and
> UID
> > in state metadata[1].
> >
> > In short users are interested in what kind of operators are inside a
> > checkpoint data which can be enhanced from user experience perspective.
> The
> > details can be found in FLIP-474[1].
> >
> > Please share your thoughts on this.
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-474%3A+Store+operator+name+and+UID+in+state+metadata
> >
> > BR,
> > G
> >
>

Re: [VOTE] FLIP-464: Merge "flink run" and "flink run-application"

[Márton Balassi]

+1 (binding)

On Wed, Jun 12, 2024 at 7:25 PM Őrhidi Mátyás 
wrote:

> Sounds reasonable,
> +1
>
> Cheers,
> Matyas
>
>
> On Wed, Jun 12, 2024 at 8:54 AM Mate Czagany  wrote:
>
> > Hi,
> >
> > Thank you for driving this Ferenc,
> > +1 (non-binding)
> >
> > Regards,
> > Mate
> >
> > Ferenc Csaky  ezt írta (időpont: 2024. jún.
> > 12., Sze, 17:23):
> >
> > > Hello devs,
> > >
> > > I would like to start a vote about FLIP-464 [1]. The FLIP is about to
> > > merge back the
> > > "flink run-application" functionality to "flink run", so the latter
> will
> > > be capable to deploy jobs in
> > > all deployment modes. More details in the FLIP. Discussion thread [2].
> > >
> > > The vote will be open for at least 72 hours (until 2024 March 23 14:03
> > > UTC) unless there
> > > are any objections or insufficient votes.
> > >
> > > Thanks,Ferenc
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=311626179
> > > [2] https://lists.apache.org/thread/xh58xs0y58kqjmfvd4yor79rv6dlcg5g
> >
>

Re: Flink Kubernetes Operator 1.9.0 release planning

[Márton Balassi]

+1 for cutting the release and Gyula as the release manager.

On Tue, Jun 11, 2024 at 10:41 AM David Radley 
wrote:

> I agree – thanks for driving this Gyula.
>
> From: Rui Fan <1996fan...@gmail.com>
> Date: Tuesday, 11 June 2024 at 02:52
> To: dev@flink.apache.org 
> Cc: Mate Czagany 
> Subject: [EXTERNAL] Re: Flink Kubernetes Operator 1.9.0 release planning
> Thanks Gyula for driving this release!
>
> > I suggest we cut the release branch this week after merging current
> > outstanding smaller PRs.
>
> It makes sense to me.
>
> Best,
> Rui
>
> On Mon, Jun 10, 2024 at 3:05 PM Gyula Fóra  wrote:
>
> > Hi all!
> >
> > I want to kick off the discussion / release process for the Flink
> > Kubernetes Operator 1.9.0 version.
> >
> > The last, 1.8.0, version was released in March and since then we have
> had a
> > number of important fixes. Furthermore there are some bigger pieces of
> > outstanding work in the form of open PRs such as the Savepoint CRD work
> > which should only be merged to 1.10.0 to gain more exposure/stability.
> >
> > I suggest we cut the release branch this week after merging current
> > outstanding smaller PRs.
> >
> > I volunteer as the release manager but if someone else would like to do
> it,
> > I would also be happy to assist.
> >
> > Please let me know what you think.
> >
> > Cheers,
> > Gyula
> >
>
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>

Re: [DISCUSS] Planning Flink 1.20

[Márton Balassi]

Thanks for kicking this off, folks.

+1 for the timeline and the release manager candidates (Weijie, Rui
Fan,Ufuk/Robert).

On Mon, Mar 25, 2024 at 12:54 PM Leonard Xu  wrote:

> Wow, happy to see Ufuk and Robert join the release managers group.
>
> +1 for the release manager candidates(Weijie, Rui Fan,Ufuk and Robert)
> from my side.
>
>
> Best,
> Leonard
>
>
>
> > 2024年3月25日 下午6:09，Robert Metzger  写道：
> >
> > Hi, thanks for starting the discussion.
> >
> > +1 for the proposed timeline and the three proposed release managers.
> >
> > I'm happy to join the release managers group as well, as a backup for
> Ufuk
> > (unless there are objections about the number of release managers)
> >
> > On Mon, Mar 25, 2024 at 11:04 AM Ufuk Celebi  wrote:
> >
> >> Hey all,
> >>
> >> I'd like to join the release managers for 1.20 as well. I'm looking
> >> forward to getting more actively involved again.
> >>
> >> Cheers,
> >>
> >> Ufuk
> >>
> >> On Sun, Mar 24, 2024, at 11:27 AM, Ahmed Hamdy wrote:
> >>> +1 for the proposed timeline and release managers.
> >>> Best Regards
> >>> Ahmed Hamdy
> >>>
> >>>
> >>> On Fri, 22 Mar 2024 at 07:41, Xintong Song 
> >> wrote:
> >>>
>  +1 for the proposed timeline and Weijie & Rui as the release managers.
> 
>  I think it would be welcomed if another 1-2 volunteers join as the
> >> release
>  managers, but that's not a must. We used to have only 1-2 release
> >> managers
>  for each release,
> 
>  Best,
> 
>  Xintong
> 
> 
> 
>  On Fri, Mar 22, 2024 at 2:55 PM Jark Wu  wrote:
> 
> > Thanks for kicking this off.
> >
> > +1 for the volunteered release managers (Weijie Guo, Rui Fan) and the
> > targeting date (feature freeze: June 15).
> >
> > Best,
> > Jark
> >
> >
> >
> >
> >
> > On Fri, 22 Mar 2024 at 14:00, Rui Fan <1996fan...@gmail.com> wrote:
> >
> >> Thanks Leonard for this feedback and help!
> >>
> >> Best,
> >> Rui
> >>
> >> On Fri, Mar 22, 2024 at 12:36 PM weijie guo <
> >> guoweijieres...@gmail.com
> >
> >> wrote:
> >>
> >>> Thanks Leonard!
> >>>
>  I'd like to help you if you need some help like permissions
> >> from
>  PMC
> >>> side, please feel free to ping me.
> >>>
> >>> Nice to know. It'll help a lot!
> >>>
> >>> Best regards,
> >>>
> >>> Weijie
> >>>
> >>>
> >>> Leonard Xu  于2024年3月22日周五 12:09写道：
> >>>
>  +1 for the proposed release managers (Weijie Guo, Rui Fan),
> >> both the
> > two
>  candidates are pretty active committers thus I believe they
> >> know the
>  community development process well. The recent releases have
> >> four
> >> release
>  managers, and I am also looking forward to having other
> >> volunteers
>  join the management of Flink 1.20.
> 
>  +1 for targeting date (feature freeze: June 15, 2024),
> >> referring to
> > the
>  release cycle of recent versions, release cycle of 4 months
> >> makes
> > sense
> >> to
>  me.
> 
> 
>  I'd like to help you if you need some help like permissions
> >> from PMC
>  side, please feel free to ping me.
> 
>  Best,
>  Leonard
> 
> 
> > 2024年3月19日 下午5:35，Rui Fan <1996fan...@gmail.com> 写道：
> >
> > Hi Weijie,
> >
> > Thanks for kicking off 1.20! I'd like to join you and
> >> participate
>  in
> >> the
> > 1.20 release.
> >
> > Best,
> > Rui
> >
> > On Tue, Mar 19, 2024 at 5:30 PM weijie guo <
> > guoweijieres...@gmail.com
> >>>
> > wrote:
> >
> >> Hi everyone,
> >>
> >> With the release announcement of Flink 1.19, it's a good
> >> time to
> > kick
>  off
> >> discussion of the next release 1.20.
> >>
> >>
> >> - Release managers
> >>
> >>
> >> I'd like to volunteer as one of the release managers this
> >> time.
>  It
> >> has
>  been
> >> good practice to have a team of release managers from
> >> different
> >> backgrounds, so please raise you hand if you'd like to
> >> volunteer
> > and
>  get
> >> involved.
> >>
> >>
> >>
> >> - Timeline
> >>
> >>
> >> Flink 1.19 has been released. With a target release cycle of
> >> 4
> >> months,
> >> we propose a feature freeze date of *June 15, 2024*.
> >>
> >>
> >>
> >> - Collecting features
> >>
> >>
> >> As usual, we've created a wiki page[1] for collecting new
>  features
> > in
>  1.20.
> >>
> >>
> >> In addition, we already have a number of FLIPs that have been
>  voted
> >> or

Re: [VOTE] FLIP-439: Externalize Kudu Connector from Bahir

[Márton Balassi]

+1(binding)

On Thu, Mar 21, 2024 at 1:24 PM Leonard Xu  wrote:

> +1(binding)
>
> Best,
> Leonard
>
> > 2024年3月21日 下午5:21，Martijn Visser  写道：
> >
> > +1 (binding)
> >
> > On Thu, Mar 21, 2024 at 8:01 AM gongzhongqiang <
> gongzhongqi...@apache.org>
> > wrote:
> >
> >> +1 (non-binding)
> >>
> >> Bests,
> >> Zhongqiang Gong
> >>
> >> Ferenc Csaky  于2024年3月20日周三 22:11写道：
> >>
> >>> Hello devs,
> >>>
> >>> I would like to start a vote about FLIP-439 [1]. The FLIP is about to
> >>> externalize the Kudu
> >>> connector from the recently retired Apache Bahir project [2] to keep it
> >>> maintainable and
> >>> make it up to date as well. Discussion thread [3].
> >>>
> >>> The vote will be open for at least 72 hours (until 2024 March 23 14:03
> >>> UTC) unless there
> >>> are any objections or insufficient votes.
> >>>
> >>> Thanks,
> >>> Ferenc
> >>>
> >>> [1]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-439%3A+Externalize+Kudu+Connector+from+Bahir
> >>> [2] https://attic.apache.org/projects/bahir.html
> >>> [3] https://lists.apache.org/thread/oydhcfkco2kqp4hdd1glzy5vkw131rkz
> >>
>
>

Re: [VOTE] Apache Flink Kubernetes Operator Release 1.8.0, release candidate #1

[Márton Balassi]

+1 (binding)

As per Gyula's suggestion above verified with "
ghcr.io/apache/flink-kubernetes-operator:91d67d9 ".

- Verified Helm repo works as expected, points to correct image tag, build,
version
- Verified basic examples + checked operator logs everything looks as
expected
- Verified hashes, signatures and source release contains no binaries
- Ran built-in tests, built jars + docker image from source successfully
- Upgraded the operator and the CRD from 1.7.0 to 1.8.0

Best,
Marton

On Wed, Mar 20, 2024 at 9:10 PM Mate Czagany  wrote:

> Hi,
>
> +1 (non-binding)
>
> - Verified checksums
> - Verified signatures
> - Verified no binaries in source distribution
> - Verified Apache License and NOTICE files
> - Executed tests
> - Built container image
> - Verified chart version and appVersion matches
> - Verified Helm chart can be installed with default values
> - Verify that RC repo works as Helm repo
>
> Best Regards,
> Mate
>
> Alexander Fedulov  ezt írta (időpont: 2024.
> márc. 19., K, 23:10):
>
> > Hi Max,
> >
> > +1
> >
> > - Verified SHA checksums
> > - Verified GPG signatures
> > - Verified that the source distributions do not contain binaries
> > - Verified built-in tests (mvn clean verify)
> > - Verified build with Java 11 (mvn clean install -DskipTests -T 1C)
> > - Verified that Helm and operator files contain Apache licenses (rg -L
> > --files-without-match "http://www.apache.org/licenses/LICENSE-2.0"; .).
> >  I am not sure we need to
> > include ./examples/flink-beam-example/dependency-reduced-pom.xml
> > and ./flink-autoscaler-standalone/dependency-reduced-pom.xml though
> > - Verified that chart and appVersion matches the target release (91d67d9)
> > - Verified that Helm chart can be installed from the local Helm folder
> > without overriding any parameters
> > - Verified that Helm chart can be installed from the RC repo without
> > overriding any parameters (
> >
> >
> https://dist.apache.org/repos/dist/dev/flink/flink-kubernetes-operator-1.8.0-rc1
> > )
> > - Verified docker container build
> >
> > Best,
> > Alex
> >
> >
> > On Mon, 18 Mar 2024 at 20:50, Maximilian Michels  wrote:
> >
> > > @Rui @Gyula Thanks for checking the release!
> > >
> > > >A minor correction is that [3] in the email should point to:
> > > >ghcr.io/apache/flink-kubernetes-operator:91d67d9 . But the helm chart
> > and
> > > > everything is correct. It's a typo in the vote email.
> > >
> > > Good catch. Indeed, for the linked Docker image 8938658 points to
> > > HEAD^ of the rc branch, 91d67d9 is the HEAD. There are no code changes
> > > between those two commits, except for updating the version. So the
> > > votes are not impacted, especially because votes are casted against
> > > the source release which, as you pointed out, contains the correct
> > > image ref.
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Mon, Mar 18, 2024 at 9:54 AM Gyula Fóra 
> wrote:
> > > >
> > > > Hi Max!
> > > >
> > > > +1 (binding)
> > > >
> > > >  - Verified source release, helm chart + checkpoints / signatures
> > > >  - Helm points to correct image
> > > >  - Deployed operator, stateful example and executed upgrade +
> savepoint
> > > > redeploy
> > > >  - Verified logs
> > > >  - Flink web PR looks good +1
> > > >
> > > > A minor correction is that [3] in the email should point to:
> > > > ghcr.io/apache/flink-kubernetes-operator:91d67d9 . But the helm
> chart
> > > and
> > > > everything is correct. It's a typo in the vote email.
> > > >
> > > > Thank you for preparing the release!
> > > >
> > > > Cheers,
> > > > Gyula
> > > >
> > > > On Mon, Mar 18, 2024 at 8:26 AM Rui Fan <1996fan...@gmail.com>
> wrote:
> > > >
> > > > > Thanks Max for driving this release!
> > > > >
> > > > > +1(non-binding)
> > > > >
> > > > > - Downloaded artifacts from dist ( svn co
> > > > >
> > > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/flink/flink-kubernetes-operator-1.8.0-rc1/
> > > > > )
> > > > > - Verified SHA512 checksums : ( for i in *.tgz; do echo $i;
> sha512sum
> > > > > --check $i.sha512; done )
> > > > > - Verified GPG signatures : ( $ for i in *.tgz; do echo $i; gpg
> > > --verify
> > > > > $i.asc $i )
> > > > > - Build the source with java-11 and java-17 ( mvn -T 20 clean
> install
> > > > > -DskipTests )
> > > > > - Verified the license header during build the source
> > > > > - Verified that chart and appVersion matches the target release
> (less
> > > the
> > > > > index.yaml and Chart.yaml )
> > > > > - RC repo works as Helm repo( helm repo add
> > > flink-operator-repo-1.8.0-rc1
> > > > >
> > > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/flink/flink-kubernetes-operator-1.8.0-rc1/
> > > > > )
> > > > > - Verified Helm chart can be installed  ( helm install
> > > > > flink-kubernetes-operator
> > > > > flink-operator-repo-1.8.0-rc1/flink-kubernetes-operator --set
> > > > > webhook.create=false )
> > > > > - Submitted the autoscaling demo, the autoscaler works well with
> > > *memory
> > > 

Re: [DISCUSS] FLIP Suggestion: Externalize Kudu Connector from Bahir

[Márton Balassi]

Hi Feri,

+1. Thanks for initiating this, the FLIP proposal looks good to me.

On Wed, Mar 6, 2024 at 4:24 PM Ferenc Csaky 
wrote:

> Hello devs,
>
> Opening this thread to discuss a FLIP [1] about externalizing the Kudu
> connector, as recently
> the Apache Bahir project were moved to the attic [2]. Some details were
> discussed already
> in another thread [3]. I am proposing to externalize this connector and
> keep it maintainable,
> and up to date.
>
> Best regards,
> Ferenc
>
> [1]
> https://docs.google.com/document/d/1vHF_uVe0FTYCb6PRVStovqDeqb_C_FKjt2P5xXa7uhE
> [2] https://bahir.apache.org/
> [3] https://lists.apache.org/thread/2nb8dxxfznkyl4hlhdm3vkomm8rk4oyq

Re: [VOTE] FLIP-314: Support Customized Job Lineage Listener

[Márton Balassi]

+1 (binding)

Marton

On Wed, Feb 28, 2024 at 5:14 PM Gyula Fóra  wrote:

> +1 (binding)
>
> Gyula
>
> On Wed, Feb 28, 2024 at 11:10 AM Maciej Obuchowski  >
> wrote:
>
> > +1 (non-binding)
> >
> > Best,
> > Maciej Obuchowski
> >
> > śr., 28 lut 2024 o 10:29 Zhanghao Chen 
> > napisał(a):
> >
> > > +1 (non-binding)
> > >
> > > Best,
> > > Zhanghao Chen
> > > 
> > > From: Yong Fang 
> > > Sent: Wednesday, February 28, 2024 10:12
> > > To: dev 
> > > Subject: [VOTE] FLIP-314: Support Customized Job Lineage Listener
> > >
> > > Hi devs,
> > >
> > > I would like to restart a vote about FLIP-314: Support Customized Job
> > > Lineage Listener[1].
> > >
> > > Previously, we added lineage related interfaces in FLIP-314. Before the
> > > interfaces were developed and merged into the master, @Maciej and
> > > @Zhenqiu provided valuable suggestions for the interface from the
> > > perspective of the lineage system. So we updated the interfaces of
> > FLIP-314
> > > and discussed them again in the discussion thread [2].
> > >
> > > So I am here to initiate a new vote on FLIP-314, the vote will be open
> > for
> > > at least 72 hours unless there is an objection or insufficient votes
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-314%3A+Support+Customized+Job+Lineage+Listener
> > > [2] https://lists.apache.org/thread/wopprvp3ww243mtw23nj59p57cghh7mc
> > >
> > > Best,
> > > Fang Yong
> > >
> >
>

Re: [DISCUSS] Apache Bahir retired

[Márton Balassi]

Hi team,

Thanks for bringing this up, Feri. I am +1 for maintaining the Kudu
connector as an external Flink connector.

As per the legal/trademark questions this is actually fair game because one
does not donate code to a specific Apache project, technically it is
donated to the Apache Software foundation. Consequently moving between ASF
projects is fine, I would add a line to the NOTICE file stating that this
code originally lived in Bahir once we forked it.

Although I did not find an easy to link precedent this is also implied in
the Attic Bahir site [1] ("notify us if you fork outside Apache") and in
this [2] Apache community dev list chat. We should notify the Attic team in
any case. :-)

[1] https://attic.apache.org/projects/bahir.html
[2] https://lists.apache.org/thread/p31mz4x4dcvd43f026d5p05rpglzfyrt

On Tue, Feb 27, 2024 at 10:09 AM Ferenc Csaky 
wrote:

> Thank you Leonard for sharing your thoughts on this topic.
>
> I agree that complying with the Flink community connector
> development process would be a must, if there are no legal or
> copyright issues, I would be happy to take that task for this
> particular case.
>
> I am no legal/copyright expert myslef, but Bahir uses the Apache
> 2.0 license as well, so I believe it should be possible without too many
> complications, but I try to look for help on that front.
>
> FYI we are using and supporting a downstream fork of the Kudu connector on
> top of Flink 1.18 without any major modifications, so it is pretty up to
> date upstream as well.
>
> Regards,
> Ferenc
>
>
>
>
> On Monday, February 26th, 2024 at 10:29, Leonard Xu 
> wrote:
>
> >
> >
> > Hey, Ferenc
> >
> > Thanks for initiating this discussion. Apache Bahir is a great project
> that provided significant assistance to many Apache Flink/Spark users. It's
> pity news that it has been retired.
> >
> > I believe that connectivity is crucial for building the ecosystem of the
> Flink such a computing engine. The community, or at least I, would actively
> support the introduction and maintenance of new connectors. Therefore,
> adding a Kudu connector or other connectors from Bahir makes sense to me,
> as long as we adhere to the development process for connectors in the Flink
> community[1].
> > I recently visited the Bahir Flink repository. Although the last release
> of Bahir Flink was in August ’22[2] which is compatible with Flink 1.14,
> its latest code is compatible with Flink 1.17[3]. So, based on the existing
> codebase, developing an official Apache Flink connector for Kudu or other
> connectors should be manageable. One point to consider is that if we're not
> developing a connector entirely from scratch but based on an existing
> repository, we must ensure that there are no copyright issues. Here, "no
> issues" means satisfying both Apache Bahir's and Apache Flink's copyright
> requirements. Honestly, I'm not an expert in copyright or legal matters. If
> you're interested in contributing to the Kudu connector, it might be
> necessary to attract other experienced community members to participate in
> this aspect.
> >
> > Best,
> > Leonard
> >
> > [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP+Connector+Template
> > [2] https://github.com/apache/bahir-flink/releases/tag/v1.1.0
> > [3] https://github.com/apache/bahir-flink/blob/master/pom.xml#L116
> >
> >
> >
> > > 2024年2月22日 下午6:37，Ferenc Csaky ferenc.cs...@pm.me.INVALID 写道：
> > >
> > > Hello devs,
> > >
> > > Just saw that the Bahir project is retired [1]. Any plans on what's
> happening with the Flink connectors that were part of this project? We
> specifically use the Kudu connector and integrate it to our platform at
> Cloudera, so we would be okay to maintain it. Would it be possible to carry
> it over as separate connector repu under the Apache umbrella similarly as
> it happened with the external connectors previously?
> > >
> > > Thanks,
> > > Ferenc
>

Re: [VOTE] Accept Flink CDC into Apache Flink

[Márton Balassi]

+1 (binding)

On Tue, Jan 9, 2024 at 10:15 AM Leonard Xu  wrote:

> +1(binding)
>
> Best,
> Leonard
>
> > 2024年1月9日 下午5:08，Yangze Guo  写道：
> >
> > +1 (non-binding)
> >
> > Best,
> > Yangze Guo
> >
> > On Tue, Jan 9, 2024 at 5:06 PM Robert Metzger 
> wrote:
> >>
> >> +1 (binding)
> >>
> >>
> >> On Tue, Jan 9, 2024 at 9:54 AM Guowei Ma  wrote:
> >>
> >>> +1 (binding)
> >>> Best,
> >>> Guowei
> >>>
> >>>
> >>> On Tue, Jan 9, 2024 at 4:49 PM Rui Fan <1996fan...@gmail.com> wrote:
> >>>
>  +1 (non-binding)
> 
>  Best,
>  Rui
> 
>  On Tue, Jan 9, 2024 at 4:41 PM Hang Ruan 
> wrote:
> 
> > +1 (non-binding)
> >
> > Best,
> > Hang
> >
> > gongzhongqiang  于2024年1月9日周二 16:25写道：
> >
> >> +1 non-binding
> >>
> >> Best,
> >> Zhongqiang
> >>
> >> Leonard Xu  于2024年1月9日周二 15:05写道：
> >>
> >>> Hello all,
> >>>
> >>> This is the official vote whether to accept the Flink CDC code
> >> contribution
> >>> to Apache Flink.
> >>>
> >>> The current Flink CDC code, documentation, and website can be
> >>> found here:
> >>> code: https://github.com/ververica/flink-cdc-connectors <
> >>> https://github.com/ververica/flink-cdc-connectors>
> >>> docs: https://ververica.github.io/flink-cdc-connectors/ <
> >>> https://ververica.github.io/flink-cdc-connectors/>
> >>>
> >>> This vote should capture whether the Apache Flink community is
> > interested
> >>> in accepting, maintaining, and evolving Flink CDC.
> >>>
> >>> Regarding my original proposal[1] in the dev mailing list, I firmly
> >> believe
> >>> that this initiative aligns perfectly with Flink. For the Flink
> >> community,
> >>> it represents an opportunity to bolster Flink's competitive edge in
> >>> streaming
> >>> data integration, fostering the robust growth and prosperity of the
> >> Apache
> >>> Flink
> >>> ecosystem. For the Flink CDC project, becoming a sub-project of
>  Apache
> >>> Flink
> >>> means becoming an integral part of a neutral open-source community,
> >>> capable of
> >>> attracting a more diverse pool of contributors.
> >>>
> >>> All Flink CDC maintainers are dedicated to continuously
> >>> contributing
>  to
> >>> achieve
> >>> seamless integration with Flink. Additionally, PMC members like
> >>> Jark,
> >>> Qingsheng,
> >>> and I are willing to infacilitate the expansion of contributors and
> >>> committers to
> >>> effectively maintain this new sub-project.
> >>>
> >>> This is a "Adoption of a new Codebase" vote as per the Flink bylaws
> > [2].
> >>> Only PMC votes are binding. The vote will be open at least 7 days
> >>> (excluding weekends), meaning until Thursday January 18 12:00 UTC,
> >>> or
> >>> until we
> >>> achieve the 2/3rd majority. We will follow the instructions in the
> > Flink
> >>> Bylaws
> >>> in the case of insufficient active binding voters:
> >>>
>  1. Wait until the minimum length of the voting passes.
>  2. Publicly reach out via personal email to the remaining binding
> >> voters
> >>> in the
> >>> voting mail thread for at least 2 attempts with at least 7 days
>  between
> >>> two attempts.
>  3. If the binding voter being contacted still failed to respond
>  after
> >>> all the attempts,
> >>> the binding voter will be considered as inactive for the purpose of
> > this
> >>> particular voting.
> >>>
> >>> Welcome voting !
> >>>
> >>> Best,
> >>> Leonard
> >>> [1]
> >>> https://lists.apache.org/thread/o7klnbsotmmql999bnwmdgo56b6kxx9l
> >>> [2]
> >>>
> >>
> >
> 
> >>>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120731026
> >>
> >
> 
> >>>
>
>

Re: [DISCUSS] Externalized Python Connector Release/Dependency Process

[Márton Balassi]

+1

Thanks, Danny - I really appreciate you taking the time for the in-depth
investigation. Please proceed, looking forward to your experience.

On Mon, Jan 8, 2024 at 6:04 PM Martijn Visser 
wrote:

> Thanks for investigating Danny. It looks like the best direction to go to
> :)
>
> On Mon, Jan 8, 2024 at 5:56 PM Péter Váry 
> wrote:
> >
> > Thanks Danny for working on this!
> >
> > It would be good to do this in a way that the different connectors could
> > reuse as much code as possible, so if possible put most of the code to
> the
> > flink connector shared utils repo [1]
> >
> > +1 from for the general direction (non-binding)
> >
> > Thanks,
> > Peter
> >
> > [1] https://github.com/apache/flink-connector-shared-utils
> >
> >
> > Danny Cranmer  ezt írta (időpont: 2024. jan.
> 8.,
> > H, 17:31):
> >
> > > Hello all,
> > >
> > > I have been working with Péter and Marton on externalizing python
> > > connectors [1] from the main repo to the connector repositories. We
> have
> > > the code moved and the CI running tests for Kafka and AWS Connectors.
> I am
> > > now looking into the release process.
> > >
> > > When we undertake a Flink release we perform the following steps [2],
> > > regarding Python: 1/ run python build on CI, 2/ download Wheels
> artifacts,
> > > 3/ upload artifacts to the dist and 4/ deploy to pypi. The plan is to
> > > follow the same steps for connectors, using Github actions instead of
> Azure
> > > pipeline.
> > >
> > > Today we have a single pypi project for pyflink that contains all the
> Flink
> > > libs, apache-flink [3]. I propose we create a new pypi project per
> > > connector using the existing connector version, and following naming
> > > convention: apache-, for example:
> > > apache-flink-connector-aws, apache-flink-connector-kafka. Therefore to
> use
> > > a DataStream API connector in python, users would need to first
> install the
> > > lib, for example "python -m pip install apache-flink-connector-aws".
> > >
> > > Once we have consensus I will update the release process and perform a
> > > release of the flink-connector-aws project to test it end-to-end. I
> look
> > > forward to any feedback.
> > >
> > > Thanks,
> > > Danny
> > >
> > > [1] https://issues.apache.org/jira/browse/FLINK-33528
> > > [2]
> > >
> https://cwiki.apache.org/confluence/display/FLINK/Creating+a+Flink+Release
> > > [3] https://pypi.org/project/apache-flink/
> > >
>

Re: [DISCUSS] FLIP-316: Introduce SQL Driver

[Márton Balassi]

Thanks, Paul.

Ferenc and I have been looking into unblocking the Kubernetes path via an
updated implementation for FLINK-28915 to ship the jars conveniently there.
You can expect an updated PR there next week. Looking forward to your
findings in the YARN POC.

On Mon, Dec 11, 2023 at 4:01 AM Paul Lam  wrote:

> Hi Ferenc,
>
> Sorry for my late reply.
>
> > Is any active work happening on this FLIP? As far as I see there
> > are blockers that needs to happen first to implement regarding
> > artifact distribution.
>
> You’re right. There’s a block in K8s application mode, but none in
> YARN application. I’m doing a POC on YARN application mode
> before starting a vote thread.
>
> I’ve been busy lately, but the FLIP is active for sure. The situation
> will change in a couple of weeks.
>
> Thank you for reaching out! I’ll let you know when the POC is completed.
>
> Best,
> Paul Lam
>
> > 2023年11月21日 06:01，Ferenc Csaky  写道：
> >
> > Hello devs,
> >
> > Is any active work happening on this FLIP? As far as I see there
> > are blockers that needs to happen first to implement regarding
> > artifact distribution.
> >
> > Is this work in halt completetly or some efforts are going into
> > resolve the blockers first or something?
> >
> > Our platform would benefit this feature a lot, we have a kind of
> > working custom implementation at the moment, but it is uniquely
> > adapted to our app and platform.
> >
> > I could help out to move this forward.
> >
> > Best,
> > Ferenc
> >
> >
> >
> > On Friday, June 30th, 2023 at 04:53, Paul Lam  > wrote:
> >
> >
> >>
> >>
> >> Hi Jing,
> >>
> >> Thanks for your input!
> >>
> >>> Would you like to add
> >>> one section to describe(better with script/code example) how to use it
> in
> >>> these two scenarios from users' perspective?
> >>
> >>
> >> OK. I’ll update the FLIP with the code snippet after I get the POC
> branch done.
> >>
> >>> NIT: the pictures have transparent background when readers click on
> it. It
> >>> would be great if you can replace them with pictures with white
> background.
> >>
> >>
> >> Fixed. Thanks for pointing that out :)
> >>
> >> Best,
> >> Paul Lam
> >>
> >>> 2023年6月27日 06:51，Jing Ge j...@ververica.com.INVALID 写道：
> >>>
> >>> Hi Paul,
> >>>
> >>> Thanks for driving it and thank you all for the informative
> discussion! The
> >>> FLIP is in good shape now. As described in the FLIP, SQL Driver will be
> >>> mainly used to run Flink SQLs in two scenarios: 1. SQL client/gateway
> in
> >>> application mode and 2. external system integration. Would you like to
> add
> >>> one section to describe(better with script/code example) how to use it
> in
> >>> these two scenarios from users' perspective?
> >>>
> >>> NIT: the pictures have transparent background when readers click on
> it. It
> >>> would be great if you can replace them with pictures with white
> background.
> >>>
> >>> Best regards,
> >>> Jing
> >>>
> >>> On Mon, Jun 26, 2023 at 1:31 PM Paul Lam   mailto:paullin3...@gmail.com> wrote:
> >>>
>  Hi Shengkai,
> 
> > * How can we ship the json plan to the JobManager?
> 
>  The Flink K8s module should be responsible for file distribution. We
> could
>  introduce
>  an option like `kubernetes.storage.dir`. For each flink cluster, there
>  would be a
>  dedicated subdirectory, with the pattern like
>  `${kubernetes.storage.dir}/${cluster-id}`.
> 
>  All resources-related options (e.g. pipeline jars, json plans) that
> are
>  configured with
>  scheme `file://`   >
> would be uploaded to the resource directory
>  and downloaded to the
>  jobmanager, before SQL Driver accesses the files with the original
>  filenames.
> 
> > * Classloading strategy
> 
>  We could directly specify the SQL Gateway jar as the jar file in
>  PackagedProgram.
>  It would be treated like a normal user jar and the SQL Driver is
> loaded
>  into the user
>  classloader. WDYT?
> 
> > * Option `$internal.sql-gateway.driver.sql-config` is string type
> > I think it's better to use Map type here
> 
>  By Map type configuration, do you mean a nested map that contains all
>  configurations?
> 
>  I hope I've explained myself well, it’s a file that contains the
> extra SQL
>  configurations, which would be shipped to the jobmanager.
> 
> > * PoC branch
> 
>  Sure. I’ll let you know once I get the job done.
> 
>  Best,
>  Paul Lam
> 
> > 2023年6月26日 14:27，Shengkai Fang  fskm...@gmail.com> mailto:fskm...@gmail.com> 写道：
> >
> > Hi, Paul.
> >
> > Thanks for your update. I have a few questions about the new design:
> >
> > * How can we ship the json plan to the JobManager?
> >
> > The current design only exposes an option about the URL of the json
> > plan. It seems the gateway is responsible to upload to an external
> stroage.
>

Re: [VOTE] FLIP-372: Allow TwoPhaseCommittingSink WithPreCommitTopology to alter the type of the Committable

[Márton Balassi]

+1 (binding)

On Mon 18. Dec 2023 at 09:34, Péter Váry 
wrote:

> Hi everyone,
>
> Since there were no further comments on the discussion thread [1], I would
> like to start the vote for FLIP-372 [2].
>
> The FLIP started as a small new feature, but in the discussion thread and
> in a similar parallel thread [3] we opted for a somewhat bigger change in
> the Sink V2 API.
>
> Please read the FLIP and cast your vote.
>
> The vote will remain open for at least 72 hours and only concluded if there
> are no objections and enough (i.e. at least 3) binding votes.
>
> Thanks,
> Peter
>
> [1] - https://lists.apache.org/thread/344pzbrqbbb4w0sfj67km25msp7hxlyd
> [2] -
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-372%3A+Allow+TwoPhaseCommittingSink+WithPreCommitTopology+to+alter+the+type+of+the+Committable
> [3] - https://lists.apache.org/thread/h6nkgth838dlh5s90sd95zd6hlsxwx57
>

Re: [DISCUSS] Allow TwoPhaseCommittingSink WithPreCommitTopology to alter the type of the Committable

[Márton Balassi]

+1

Thanks, Peter. Based on the consensus in the recent thread on FLIP-371 [1]
I agree that this is the right approach. I made some minor edits to the
FLIP, which looks good to me now.

[1] https://lists.apache.org/thread/h6nkgth838dlh5s90sd95zd6hlsxwx57

On Wed, Dec 13, 2023 at 5:30 PM Gyula Fóra  wrote:

> Thank you Peter Vary for updating the FLIP based on the discussions.
> I really like the improvements introduced by the mixin interfaces which now
> aligns much better with the source and table connectors.
>
> While this introduces some breaking changes to the existing connectors,
> this is a technical debt that we need to resolve as soon as possible and
> fully before 2.0.
>
> +1 from my side.
>
> I am cc'ing some folks participating in the other threads, sorry about that
> :)
>
> Cheers,
> Gyula
>
> On Wed, Dec 13, 2023 at 4:14 PM Péter Váry 
> wrote:
>
> > I have updated the FLIP-372 [1] based on the comments from multiple
> > sources. Moved to the mixin approach as this seems to be the consensus
> > based on this thread [2]
> > Also created a draft implementation [3] PR, so I can test the changes and
> > default implementations (no new tests yet)
> > Please provide your feedback, so I can address your questions, comments
> and
> > then we can move forward to voting.
> >
> > Thanks,
> > Peter
> >
> > [1] -
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-372%3A+Allow+TwoPhaseCommittingSink+WithPreCommitTopology+to+alter+the+type+of+the+Committable
> > [2] - https://lists.apache.org/thread/h6nkgth838dlh5s90sd95zd6hlsxwx57
> > [3] - https://github.com/apache/flink/pull/23912
> >
> > Péter Váry  ezt írta (időpont: 2023. dec.
> > 11.,
> > H, 14:28):
> >
> > > We identified another issue with the current Sink API in a thread [1]
> > > related to FLIP-371 [2]. Currently it is not possible to evolve the
> > > Sink.createWriter method with deprecation, because StatefulSink and
> > > TwoPhaseCommittingSink has methods with the same name and parameters,
> but
> > > narrowed down return type (StatefulSinkWriter,
> PrecommittingSinkWriter).
> > >
> > > To make the Sink API evolvable, we minimally have to remove these.
> > >
> > > The participants there also pointed out, that the Source API also uses
> > > mixin interfaces (SupportsHandleExecutionAttemptSourceEvent,
> > > SupportsIntermediateNoMoreSplits) in some cases. My observation is that
> > it
> > > has inheritance as well (ReaderOutput, ExternallyInducedSourceReader)
> > >
> > > I have created a draft API along these lines in a branch [3] where only
> > > the last commit is relevant [4]. This implementation would follow the
> > same
> > > patterns as the current Source API.
> > >
> > > I see two different general approaches here, and I would like to hear
> > your
> > > preferences:
> > > - Keep the changes minimal, stick to the current Sink API design. We
> > > introduce the required new combination of interfaces
> > > (TwoPhaseCommttingSinkWithPreCommitTopology,
> > > WithPostCommitTopologyWithPreCommitTopology), and do not change the API
> > > structure.
> > >  - Pros:
> > >   - Minimal change - smaller rewrite on the connector side
> > >   - Type checks happen on compile time
> > >  - Cons:
> > >   - Harder to evolve
> > >   - The number of interfaces increases with the possible
> > > combinations
> > >   - Inconsistent API patterns wrt Source API - harder for
> > > developers to understand
> > > - Migrate to a model similar to the Source API. We create mixin
> > interfaces
> > > for SupportsCommitter, SupportsWriterState, SupportsPreCommitTopology,
> > > SupportsPostCommitTopology, SupportsPreWriteTopology.
> > > - Pros:
> > > - Better evolvability
> > > - Consistent with the Source API
> > > - Cons:
> > > - The connectors need to change their inheritance patterns
> (after
> > > the deprecation period) if they are using any of the more complicated
> > Sinks.
> > > - Type checks need to use `instanceof`, which could happen on
> DAG
> > > generation time. Also, if the developer fails to correctly match the
> > > generic types on the mixin interfaces, the error will only surface
> during
> > > execution time - when the job tries to process the first record
> > >
> > > I personally prefer the Mixin approach for easier evolvability and
> better
> > > consistency, but I would like to hear your thoughts, so I can flash out
> > the
> > > chosen approach in FLIP-372
> > >
> > > Thanks,
> > > Peter
> > >
> > > [1] - https://lists.apache.org/thread/h6nkgth838dlh5s90sd95zd6hlsxwx57
> > > [2] -
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-371%3A+Provide+initialization+context+for+Committer+creation+in+TwoPhaseCommittingSink
> > > [3] - https://github.com/pvary/flink/tree/mixin_demo
> > > [4] -
> > >
> >
> https://github.com/pvary/flink/commit/acfc09f4c846f983f633bbf0c902aea49aa6b156
> > >
> > >
> > > On Fri, Nov 24, 2023, 11:38 Gyula Fóra  

Re: [VOTE] FLIP-401: REST API JSON response deserialization unknown field tolerance

[Márton Balassi]

+1 (binding)

On Tue, Dec 12, 2023 at 4:16 PM Rodrigo Meneses  wrote:

> +1
>
> On Tue, Dec 12, 2023 at 6:58 AM Maximilian Michels  wrote:
>
> > +1 (binding)
> >
> > On Tue, Dec 12, 2023 at 2:23 PM Peter Huang 
> > wrote:
> > >
> > > +1 Non-binding
> > >
> > >
> > > Peter Huang
> > >
> > > Őrhidi Mátyás 于2023年12月12日 周二下午9:14写道：
> > >
> > > > +1
> > > > Matyas
> > > >
> > > > On Mon, Dec 11, 2023 at 10:26 PM Gyula Fóra 
> > wrote:
> > > >
> > > > > +1
> > > > >
> > > > > Gyula
> > > > >
> > > > > On Mon, Dec 11, 2023 at 1:26 PM Gabor Somogyi <
> > gabor.g.somo...@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > I'd like to start a vote on FLIP-401: REST API JSON response
> > > > > > deserialization unknown field tolerance [1] which has been
> > discussed in
> > > > > > this thread [2].
> > > > > >
> > > > > > The vote will be open for at least 72 hours unless there is an
> > > > objection
> > > > > or
> > > > > > not enough votes.
> > > > > >
> > > > > > BR,
> > > > > > G
> > > > > >
> > > > > > [1]
> > > > > >
> > > > > >
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-401%3A+REST+API+JSON+response+deserialization+unknown+field+tolerance
> > > > > > [2]
> > https://lists.apache.org/thread/s52w9cf60d6s10bpzv9qjczpl6m394rz
> > > > > >
> > > > >
> > > >
> >
>

Re: [DISCUSS] FLIP-401: REST API JSON response deserialization unknown field tolerance

[Márton Balassi]

+1 This greatly improves interfacing with multiple Flink versions, e.g.
upgrades from the Kubernetes Operator.

On Mon, Dec 11, 2023 at 12:36 PM Gyula Fóra  wrote:

> Thanks Gabor!
>
> +1 from my side, this sounds like a reasonable change that will
> improve integration and backward compatibility.
>
> Let's keep this open for at least another day before starting the vote.
>
> Cheers,
> Gyula
>
> On Thu, Dec 7, 2023 at 10:48 AM Gabor Somogyi 
> wrote:
>
> > Hi All,
> >
> > I'd like to start a discussion of FLIP-401: REST API JSON response
> > deserialization unknown field tolerance[1].
> >
> > *Problem statement:*
> >
> > At the moment Flink is not ignoring unknown fields when parsing REST
> > responses. An example for such a class is *JobDetailsInfo* but this
> applies
> > to all others. It would be good to add this support to increase
> > compatibility.
> >
> > The real life use-case is when the Flink k8s operator wants to handle 2
> > jobs with 2 different Flink versions where the newer version has added a
> > new field to any REST response. Such case the operator has basically 2
> > options:
> > * Use the old Flink version -> Such case exception comes because new
> field
> > comes but it's not expected
> > * Use the new Flink version -> Such case exception comes because new
> field
> > is not coming but expected
> >
> > To hack around this issue it requires quite some ugly code parts in the
> > operator.
> >
> > *Proposed solution:*
> >
> > Ignore all unknown fields in case of REST response JSON deserialization.
> > Important to know that strict serialization would stay the same as-is.
> >
> > Actual object mapper configuration which is intended to be changed can be
> > found here[2].
> >
> > Please share your opinion on this topic.
> >
> > BR,
> > G
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-401%3A+REST+API+JSON+response+deserialization+unknown+field+tolerance
> > [2]
> >
> >
> https://github.com/apache/flink/blob/3060ccd49cc8d19634b431dbf0f09ac875d0d422/flink-runtime/src/main/java/org/apache/flink/runtime/rest/util/RestMapperUtils.java#L31-L38
> >
>

Re: SQL return type change from 1.17 to 1.18

[Márton Balassi]

Thanks, for raising this Peter. +1 for reverting the change.

Given the response from Timo and Aitozi, I believe it would be best if we
could ship reverting the change in 1.18.1.

On Thu, Dec 7, 2023 at 2:47 PM Aitozi  wrote:

> Hi Peter, Timo
> Sorry for this breaking change, I didn't notice that was a breaking
> change.
> I'm +1 to revert the FLINK-33523
>
> Regards,
> aitozi
>
> Timo Walther  于2023年12月7日周四 20:41写道：
>
> > Hi Peter,
> >
> > thanks for reaching out to the Flink community. This is indeed a serious
> > issue. As the author of the Flink type system, DataType and many related
> > utilities I strongly vote for reverting FLINK-33523:
> >
> > - It changes the Flink type system without a FLIP.
> > - It breaks backwards compatibility with UDFs and connectors.
> >
> > Regards,
> > Timo
> >
> > On 07.12.23 07:38, Péter Váry wrote:
> > > Hi Team,
> > >
> > > We are working on upgrading the Iceberg-Flink connector from 1.17 to
> > 1.18,
> > > and found that some of our tests are failing. Prabhu Joseph created a
> > jira
> > > [1] to discuss this issue, along with short example code.
> > >
> > > In a nutshell:
> > > - Create a table with an 'ARRAY' column
> > > - Run a select which returns this column
> > > - The return type changes:
> > >  - From 'Object[]' - in 1.17
> > >  - To 'int[]' - in 1.18
> > >
> > > The change is introduced by this jira [2].
> > >
> > > While I understand the reasoning behind this change, this will break
> some
> > > users existing workflow as evidenced by Xingcan Cui finding this
> > > independently [3].
> > >
> > > What is the opinion of the community about this change?
> > > - Do we want to revert the change?
> > > - Do we ask the owners of the change to make this behavior
> configurable?
> > > - Do we accept this behavior change in a minor release?
> > >
> > > Thanks,
> > > Peter
> > >
> > > [1] - https://issues.apache.org/jira/browse/FLINK-33523 - DataType
> > > ARRAY fails to cast into Object[]
> > > [2] - https://issues.apache.org/jira/browse/FLINK-31835 - DataTypeHint
> > > don't support Row>
> > > [3] - https://issues.apache.org/jira/browse/FLINK-33547 - SQL
> primitive
> > > array type after upgrading to Flink 1.18.0
> > >
> >
> >
>

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

[Márton Balassi]

Hi Leonard,

Thank you for the excellent work you and the team working on the CDC
connectors project have been doing so far. I am +1 of having them under
Flink's umbrella.

On Thu, Dec 7, 2023 at 10:26 AM Etienne Chauchot 
wrote:

> Big +1, thanks this will be a very useful addition to Flink.
>
> Best
>
> Etienne
>
> Le 07/12/2023 à 09:26, Hang Ruan a écrit :
> > +1 for contributing CDC Connectors  to Apache Flink.
> >
> > Best,
> > Hang
> >
> > Yuxin Tan  于2023年12月7日周四 16:05写道：
> >
> >> Cool, +1 for contributing CDC Connectors to Apache Flink.
> >>
> >> Best,
> >> Yuxin
> >>
> >>
> >> Jing Ge  于2023年12月7日周四 15:43写道：
> >>
> >>> Awesome! +1
> >>>
> >>> Best regards,
> >>> Jing
> >>>
> >>> On Thu, Dec 7, 2023 at 8:34 AM Sergey Nuyanzin
> >>> wrote:
> >>>
>  thanks for working on this and driving it
> 
>  +1
> 
>  On Thu, Dec 7, 2023 at 7:26 AM Feng Jin
> wrote:
> 
> > This is incredibly exciting news, a big +1 for this.
> >
> > Thank you for the fantastic work on Flink CDC. We have created
> >>> thousands
>  of
> > real-time integration jobs using Flink CDC connectors.
> >
> >
> > Best,
> > Feng
> >
> > On Thu, Dec 7, 2023 at 1:45 PM gongzhongqiang <
> >>> gongzhongqi...@apache.org
> > wrote:
> >
> >> It's very exciting to hear the news.
> >> +1 for adding CDC Connectors  to Apache Flink !
> >>
> >>
> >> Best,
> >> Zhongqiang
> >>
> >> Leonard Xu  于2023年12月7日周四 11:25写道：
> >>
> >>> Dear Flink devs,
> >>>
> >>>
> >>> As you may have heard, we at Alibaba (Ververica) are planning to
>  donate
> >> CDC Connectors for the Apache Flink project
> >>> *[1]* to the Apache Flink community.
> >>>
> >>>
> >>>
> >>> CDC Connectors for Apache Flink comprise a collection of source
> >> connectors designed specifically for Apache Flink. These connectors
> >>> *[2]*
> >>>   enable the ingestion of changes from various databases using
> >>> Change
> >> Data Capture (CDC), most of these CDC connectors are powered by
>  Debezium
> >>> *[3]*
> >>> . They support both the DataStream API and the Table/SQL API,
> >> facilitating the reading of database snapshots and continuous
> >> reading
>  of
> >> transaction logs with exactly-once processing, even in the event of
> >> failures.
> >>>
> >>>
> >>> Additionally, in the latest version 3.0, we have introduced many
> >> long-awaited features. Starting from CDC version 3.0, we've built a
> >> Streaming ELT Framework available for streaming data integration.
> >>> This
> >> framework allows users to write their data synchronization logic
> >> in a
> >> simple YAML file, which will automatically be translated into a
> >> Flink
> >> DataStreaming job. It emphasizes optimizing the task submission
> >>> process
> > and
> >> offers advanced functionalities such as whole database
> >>> synchronization,
> >> merging sharded tables, and schema evolution
> >>> *[4]*.
> >>>
> >>>
> >>>
> >>>
> >>> I believe this initiative is a perfect match for both sides. For
> >>> the
> >> Flink community, it presents an opportunity to enhance Flink's
> > competitive
> >> advantage in streaming data integration, promoting the healthy
> >> growth
>  and
> >> prosperity of the Apache Flink ecosystem. For the CDC Connectors
>  project,
> >> becoming a sub-project of Apache Flink means being part of a
> >> neutral
> >> open-source community, which can attract a more diverse pool of
> >> contributors.
> >>>
> >>> Please note that the aforementioned points represent only some of
> >>> our
> >> motivations and vision for this donation. Specific future
> >> operations
>  need
> >> to be further discussed in this thread. For example, the
> >> sub-project
>  name
> >> after the donation; we hope to name it Flink-CDC
> >>> aiming to streaming data intergration through Apache Flink,
> >>> following the naming convention of Flink-ML; And this project is
> > managed
> >> by a total of 8 maintainers, including 3 Flink PMC members and 1
> >>> Flink
> >> Committer. The remaining 4 maintainers are also highly active
> > contributors
> >> to the Flink community, donating this project to the Flink
> >> community
> >> implies that their permissions might be reduced. Therefore, we may
> >>> need
> > to
> >> bring up this topic for further discussion within the Flink PMC.
> >> Additionally, we need to discuss how to migrate existing users and
> >> documents. We have a user group of nearly 10,000 people and a
> > multi-version
> >> documentation site need to migrate. We also need to plan for the
> > migration
> >> of CI/CD processes and other specifics.
> >>>
> >>>
> >>> While there are many intricate details that require
> >> implem

Re: [DISCUSS] Resolve diamond inheritance of Sink.createWriter

[Márton Balassi]

> Cheers,
> > > > Gyula
> > > >
> > > >
> > > > On Fri, Dec 1, 2023 at 10:43 AM Péter Váry <
> > peter.vary.apa...@gmail.com>
> > > > wrote:
> > > >
> > > > > Thanks Becket for your reply!
> > > > >
> > > > > *On Option 1:*
> > > > > - I personally consider API inconsistencies more important, since
> > they
> > > > will
> > > > > remain with us "forever", but this is up to the community. I can
> > > > implement
> > > > > whichever solution we decide upon.
> > > > >
> > > > > *Option 2:*
> > > > > - I don't think this specific issue merits a rewrite, but if we
> > decide
> > > to
> > > > > change our approach, then it's a different story.
> > > > >
> > > > > *Evolvability:*
> > > > > This discussion reminds me of a similar discussion on FLIP-372 [1],
> > > where
> > > > > we are trying to decide if we should use mixin interfaces, or use
> > > > interface
> > > > > inheritance.
> > > > > With the mixin approach, we have a more flexible interface, but we
> > > can't
> > > > > check the generic types of the interfaces/classes on compile time,
> or
> > > > even
> > > > > when we create the DAG. The first issue happens when we call the
> > method
> > > > and
> > > > > fail.
> > > > > The issue here is similar:
> > > > > - *StatefulSink* needs a writer with a method to `*snapshotState*`
> > > > > - *TwoPhaseCommittingSink* needs a writer with `*prepareCommit*`
> > > > > - If there is a Sink which is stateful and needs to commit, then it
> > > needs
> > > > > both of these methods.
> > > > >
> > > > > If we use the mixin solution here, we lose the possibility to check
> > the
> > > > > types in compile time. We could do the type check in runtime using
> `
> > > > > *instanceof*`, so we are better off than with the FLIP-372 example
> > > above,
> > > > > but still lose any important possibility. I personally prefer the
> > mixin
> > > > > approach, but that would mean we rewrite the Sink API again -
> likely
> > a
> > > > > SinkV3. Are we ready to move down that path?
> > > > >
> > > > > Thanks,
> > > > > Peter
> > > > >
> > > > > [1] -
> > https://lists.apache.org/thread/344pzbrqbbb4w0sfj67km25msp7hxlyd
> > > > >
> > > > > On Thu, Nov 30, 2023, 14:53 Becket Qin 
> wrote:
> > > > >
> > > > > > Hi folks,
> > > > > >
> > > > > > Sorry for replying late on the thread.
> > > > > >
> > > > > > For this particular FLIP, I see two solutions:
> > > > > >
> > > > > > Option 1:
> > > > > > 1. On top of the the current status, rename
> > > > > > *org.apache.flink.api.connector.sink2.InitContext *to
> > > > > > *CommonInitContext (*should
> > > > > > probably be package private*)*.
> > > > > > 2. Change the name *WriterInitContext* back to *InitContext, *and
> > > > revert
> > > > > > the deprecation. We can change the parameter name to
> writerContext
> > if
> > > > we
> > > > > > want to.
> > > > > > Admittedly, this does not have full symmetric naming of the
> > > > InitContexts
> > > > > -
> > > > > > we will have CommonInitContext / InitContext /
> CommitterInitContext
> > > > > instead
> > > > > > of InitContext / WriterInitContext / CommitterInitContext.
> However,
> > > the
> > > > > > naming seems clear without much confusion. Personally, I can live
> > > with
> > > > > > that, treating the class InitContext as a non-ideal legacy class
> > name
> > > > > > without much material harm.
> > > > > >
> > > > > > Option 2:
> > > > > > Theoretically speaking, if we really want to reach the perfect
> > state
> > > > > while
> > > > > > being backwards compatible, we can create a brand new set of Sink
> > > > > > int

Re: [DISCUSS] Resolve diamond inheritance of Sink.createWriter

[Márton Balassi]

Thanks, Martijn and Peter.

In terms of the concrete issue:

   - I am following up with the author of FLIP-321 [1] (Becket) to update
   the docs [2] to reflect the right state.
   - I see two reasonable approaches in terms of proceeding with the
   specific changeset:


   1. We allow the exception from FLIP-321 for this change and let the
  PublicEvolving API change happen between Flink 1.18 and 1.19, which is
  consistent with current state of the relevant documentation. [2]
We commit
  to helping the connector repos make the necessary (one liner) changes.
  2. We revert back to the original implementation plan as explicitly
  voted on in FLIP-371 [3]. That has no API breaking changes.
However we end
  up with an inconsistently named API with duplicated internal
methods. Peter
  has also discovered additional bad patterns during his work in FLIP-372
  [3], the total of these changes could be handled in a separate FLIP that
  would do multiple PublicEvolving breaking changes to clean up the API.

In terms of the general issues:

   - I agree that if a PR review of an accepted FLIP newly introduces a
   breaking API change that warrants an update to the mailing list discussion
   and possibly even a new vote.
   - I agree with the general sentiment of FLIP-321 to provide stronger API
   guarantees with the minor note that if we have changes in mind we should
   prioritize them now such that they can be validated by Flink 2.0.
   - I agree that ideally the connector repos should build against the
   latest release and not the master branch.

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-321%3A+Introduce+an+API+deprecation+process
[2]
https://nightlies.apache.org/flink/flink-docs-master/docs/ops/upgrading/#api-compatibility-guarantees
[3]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-371%3A+Provide+initialization+context+for+Committer+creation+in+TwoPhaseCommittingSink
[4]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-372%3A+Allow+TwoPhaseCommittingSink+WithPreCommitTopology+to+alter+the+type+of+the+Committable

Best,
Marton

On Mon, Nov 27, 2023 at 7:23 PM Péter Váry 
wrote:

> I think we should try to separate the discussion in a few different topics:
>
>- Concrete issue
>   - How to solve this problem in 1.19 and wrt the affected createWriter
>   interface
>   - Update the documentation [1], so FLIP-321 is visible for every
>   contributor
>- Generic issue
>   - API stability
>   - Connector dependencies
>
>
> *CreateWriter interface*
> The change on the createWriter is not strictly required for the
> functionality defined by the requirements on the FLIP.
> If the only goal is only to have a backward compatible API, we can simply
> create a separate `*CommitterInitContext*` object and do not touch the
> writer `*InitContext*`, like it was done in the original PR [2].
> The issue is that this would result in an implementation which has
> duplicated methods/implementations (internal issue only), and has
> inconsistent naming (issue for external users).
>
> If we want to create an API which is consistent (and I agree with the
> reviewer's comments), then we need to rename the parameter type (
> *WriterInitContext*) for the createWriter method.
> I have tried to keep the backward compatibility with creating a new method
> and providing a default implementation for this new method which would call
> the original method after converting the WriterInitContext to InitContext.
>
> This is failed because the following details:
>
>- *org.apache.flink.api.connector.sink2.Sink* defines
> `*SinkWriter
>createWriter(InitContext context)`*
>- *org.apache.flink.api.connector.sink2.StatefulSink* narrows it
> down to *`StatefulSinkWriterWriterStateT> createWriter(InitContext context)`*
>- *org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink* narrows
>it down to *`PrecommittingSinkWriter
>createWriter(WriterInitContext context)`*
>-
>
>  
> *org.apache.flink.streaming.runtime.operators.sink.TestSinkV2.TestStatefulSinkV2*
>implements *StatefulSink* and *TwoPhaseCommittingSink* too
>
> *TestStatefulSinkV2* is a good example where we can not achieve backward
> compatibility, since the the compiler will fail with unrelated default
> methods [3]
>
> I am open for any suggestions how to move to the new API, and keep the
> backward compatibility. If we do not find a way to keep backward
> compatibility, and we decide that we would like to honour FLIP-321, then we
> can reverting to the original solution and keep only the changes for the `
> *createCommitter*` method.
>
> *Update the documentation*
> I have not found only one place in the docs [1], where we talk about the
> compatibility guarantees.
> Based FLIP-321 and the result of the discussion here, we should update this
> page.
>
> *API stability*
> I agree with the general sentiment of FLIP-321 to keep the changes backward
> compatib

Re: [DISCUSS] FLIP-395: Migration to GitHub Actions

[Márton Balassi]

Thanks, Matthias. Big +1 from me.

On Tue, Nov 28, 2023 at 5:30 PM Matthias Pohl
 wrote:

> Thanks for the pointer. I'm planning to join that meeting.
>
> On Tue, Nov 28, 2023 at 4:16 PM Etienne Chauchot 
> wrote:
>
> > Hi all,
> >
> > FYI there is the ASF infra roundtable soon. One of the subjects for this
> > session is GitHub Actions. It could be worth passing by:
> >
> > December 6th, 2023 at 1700 UTC on the #Roundtablechannel on Slack.
> >
> > For information about theroundtables, and about how to join,
> > see:https://infra.apache.org/roundtable.html
> > 
> >
> > Best
> >
> > Etienne
> >
> > Le 24/11/2023 à 14:16, Maximilian Michels a écrit :
> > > Thanks for reviving the efforts here Matthias! +1 for the transition
> > > to GitHub Actions.
> > >
> > > As for ASF Infra Jenkins, it works fine. Jenkins is extremely
> > > feature-rich. Not sure about the spare capacity though. I know that
> > > for Apache Beam, Google donated a bunch of servers to get additional
> > > build capacity.
> > >
> > > -Max
> > >
> > >
> > > On Thu, Nov 23, 2023 at 10:30 AM Matthias Pohl
> > >   wrote:
> > >> Btw. even though we've been focusing on GitHub Actions with this FLIP,
> > I'm
> > >> curious whether somebody has experience with Apache Infra's Jenkins
> > >> deployment. The discussion I found about Jenkins [1] is quite
> out-dated
> > >> (2014). I haven't worked with it myself but could imagine that there
> are
> > >> some features provided through plugins which are missing in GitHub
> > Actions.
> > >>
> > >> [1]https://lists.apache.org/thread/vs81xdhn3q777r7x9k7wd4dyl9kvoqn4
> > >>
> > >> On Tue, Nov 21, 2023 at 4:19 PM Matthias Pohl
> > >> wrote:
> > >>
> > >>> That's a valid point. I updated the FLIP accordingly:
> > >>>
> >  Currently, the secrets (e.g. for S3 access tokens) are maintained by
> >  certain PMC members with access to the corresponding configuration
> in
> > the
> >  Azure CI project. This responsibility will be moved to Apache Infra.
> > They
> >  are in charge of handling secrets in the Apache organization. As a
> >  consequence, updating secrets is becoming a bit more complicated.
> > This can
> >  be still considered an improvement from a legal standpoint because
> the
> >  responsibility is transferred from an individual company (i.e.
> > Ververica
> >  who's the maintainer of the Azure CI project) to the Apache
> > Foundation.
> > >>>
> > >>> On Tue, Nov 21, 2023 at 3:37 PM Martijn Visser<
> > martijnvis...@apache.org>
> > >>> wrote:
> > >>>
> >  Hi Matthias,
> > 
> >  Thanks for the write-up and for the efforts on this. I really hope
> >  that we can move away from Azure towards GHA for a better
> integration
> >  as well (directly seeing if a PR can be merged due to CI passing for
> >  example).
> > 
> >  The one thing I'm missing in the FLIP is how we would setup the
> >  secrets for the nightly runs (for the S3 tests, potential tests with
> >  external services etc). My guess is we need to provide the secret to
> >  ASF Infra and then we would be able to refer to them in a pipeline?
> > 
> >  Best regards,
> > 
> >  Martijn
> > 
> >  On Tue, Nov 21, 2023 at 3:05 PM Matthias Pohl
> >    wrote:
> > > I realized that I mixed up FLIP IDs. FLIP-395 is already reserved
> > [1]. I
> > > switched to FLIP-396 [2] for the sake of consistency. 8)
> > >
> > > [1]
> https://lists.apache.org/thread/wjd3nbvg6nt93lb0sd52f0lzls6559tv
> > > [2]
> > >
> > 
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-396%3A+Migration+to+GitHub+Actions
> > > On Tue, Nov 21, 2023 at 2:58 PM Matthias Pohl<
> matthias.p...@aiven.io
> > >
> > > wrote:
> > >
> > >> Hi everyone,
> > >>
> > >> The Flink community discussed migrating from Azure CI to GitHub
> >  Actions
> > >> quite some time ago [1]. The efforts around that stalled due to
> >  limitations
> > >> around self-hosted runner support from Apache Infra’s side. There
> >  were some
> > >> recent developments on that topic. Apache Infra is experimenting
> > with
> > >> ephemeral runners now which might enable us to move ahead with
> > GitHub
> > >> Actions.
> > >>
> > >> The goal is to join the trial phase for ephemeral runners and
> >  experiment
> > >> with our CI workflows in terms of stability and performance. At
> the
> >  end we
> > >> can decide whether we want to abandon Azure CI and move to GitHub
> >  Actions
> > >> or stick to the former one.
> > >>
> > >> Nico Weidner and Chesnay laid the groundwork on this topic in the
> >  past. I
> > >> picked up the work they did and continued experimenting with it in
> > my
> >  own
> > >> fork XComp/flink [2] the past few weeks. The workflows are in a
> > state
> >  where
> > >> I think that we start moving the relevan

Re: [VOTE] Apache Flink Kubernetes Operator Release 1.7.0, release candidate #1

[Márton Balassi]

+1 (binding)

- Verified Helm repo works as expected, points to correct image tag, build,
version
- Verified basic examples + checked operator logs everything looks as
expected
- Verified hashes, signatures and source release contains no binaries
- Ran built-in tests, built jars + docker image from source successfully
- Upgraded the operator and the CRD from 1.6.1 to 1.7.0

Best,
Marton

On Mon, Nov 20, 2023 at 2:03 PM Gyula Fóra  wrote:

> +1 (binding)
>
> Verified:
>  - Release files, maven repo contents, checksums, signature
>  - Verified and installed from Helm chart
>  - Ran basic stateful example and verified
>- Upgrade flow
>- No errors in logs
>- Autoscaler (turn on/off, verify configmap cleared correctly)
>- In-place scaling with 1.18 and adaptive scheduler
>  - Built from source with Java 11 & 17
>  - Checked release notes
>
> Cheers,
> Gyula
>
> On Fri, Nov 17, 2023 at 1:59 PM Rui Fan <1996fan...@gmail.com> wrote:
>
> > +1(non-binding)
> >
> > - Downloaded artifacts from dist
> > - Verified SHA512 checksums
> > - Verified GPG signatures
> > - Build the source with java-11 and java-17
> > - Verified the license header
> > - Verified that chart and appVersion matches the target release
> > - RC repo works as Helm rep(helm repo add flink-operator-repo-1.7.0-rc1
> >
> >
> https://dist.apache.org/repos/dist/dev/flink/flink-kubernetes-operator-1.7.0-rc1/
> > )
> > - Verified Helm chart can be installed  (helm install
> > flink-kubernetes-operator
> > flink-operator-repo-1.7.0-rc1/flink-kubernetes-operator --set
> > webhook.create=false)
> > - Submitted the autoscaling demo, the autoscaler works well with rescale
> > api (kubectl apply -f autoscaling.yaml)
> > - Download Autoscaler standalone: wget
> >
> >
> https://repository.apache.org/content/repositories/orgapacheflink-1672/org/apache/flink/flink-autoscaler-standalone/1.7.0/flink-autoscaler-standalone-1.7.0.jar
> > - Ran Autoscaler standalone locally, it works well with rescale api
> >
> > Best,
> > Rui
> >
> > On Fri, Nov 17, 2023 at 2:45 AM Mate Czagany  wrote:
> >
> > > +1 (non-binding)
> > >
> > > - Checked signatures, checksums
> > > - No binaries found in the source release
> > > - Verified all source files contain the license header
> > > - All pom files point to the correct version
> > > - Verified Helm chart version and appVersion
> > > - Verified Docker image tag
> > > - Ran flink-autoscaler-standalone JAR downloaded from the maven
> > repository
> > > - Tested autoscaler upscales correctly on load with Flink 1.18
> rescaling
> > > API
> > >
> > > Thanks,
> > > Mate
> > >
> > > Gyula Fóra  ezt írta (időpont: 2023. nov. 15., Sze,
> > > 16:37):
> > >
> > > > Hi Everyone,
> > > >
> > > > Please review and vote on the release candidate #1 for the version
> > 1.7.0
> > > of
> > > > Apache Flink Kubernetes Operator,
> > > > as follows:
> > > > [ ] +1, Approve the release
> > > > [ ] -1, Do not approve the release (please provide specific comments)
> > > >
> > > > **Release Overview**
> > > >
> > > > As an overview, the release consists of the following:
> > > > a) Kubernetes Operator canonical source distribution (including the
> > > > Dockerfile), to be deployed to the release repository at
> > dist.apache.org
> > > > b) Kubernetes Operator Helm Chart to be deployed to the release
> > > repository
> > > > at dist.apache.org
> > > > c) Maven artifacts to be deployed to the Maven Central Repository
> > > > d) Docker image to be pushed to dockerhub
> > > >
> > > > **Staging Areas to Review**
> > > >
> > > > The staging areas containing the above mentioned artifacts are as
> > > follows,
> > > > for your review:
> > > > * All artifacts for a,b) can be found in the corresponding dev
> > repository
> > > > at dist.apache.org [1]
> > > > * All artifacts for c) can be found at the Apache Nexus Repository
> [2]
> > > > * The docker image for d) is staged on github [3]
> > > >
> > > > All artifacts are signed with the key 21F06303B87DAFF1 [4]
> > > >
> > > > Other links for your review:
> > > > * JIRA release notes [5]
> > > > * source code tag "release-1.7.0-rc1" [6]
> > > > * PR to update the website Downloads page to
> > > > include Kubernetes Operator links [7]
> > > >
> > > > **Vote Duration**
> > > >
> > > > The voting time will run for at least 72 hours.
> > > > It is adopted by majority approval, with at least 3 PMC affirmative
> > > votes.
> > > >
> > > > **Note on Verification**
> > > >
> > > > You can follow the basic verification guide here[8].
> > > > Note that you don't need to verify everything yourself, but please
> make
> > > > note of what you have tested together with your +- vote.
> > > >
> > > > Cheers!
> > > > Gyula Fora
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/flink/flink-kubernetes-operator-1.7.0-rc1/
> > > > [2]
> > > >
> > https://repository.apache.org/content/repositories/orgapacheflink-1672/
> > > > [3] ghcr.io/apache/flink-kubernetes-operator:ccb10b8
> > > > [4] h

Re: Send slack community invite link

[Márton Balassi]

Here you go, this is valid for 30 days:
https://join.slack.com/t/apache-flink/shared_invite/zt-276wzpx1c-DpF_IYPeZomOS3ChYkc4SA

On Sun, Nov 12, 2023 at 8:17 AM Neelabh Shukla 
wrote:

> Hey Team,
> Can someone send me the slack invite link?
>
> Thanks,
> Neelabh
>

Re: [VOTE] Apache Flink Kubernetes Operator Release 1.6.1, release candidate #1

[Márton Balassi]

Thank you, team. @David Radley: Not having Rui's key signed is not ideal,
but is acceptable for the release.

+1 (binding)

- Verified Helm repo works as expected, points to correct image tag, build,
version
- Verified basic examples + checked operator logs everything looks as
expected
- Verified hashes, signatures and source release contains no binaries
- Ran built-in tests, built jars + docker image from source successfully

Best,
Marton

On Thu, Oct 26, 2023 at 12:24 PM David Radley 
wrote:

> Hi,
> I downloaded the artifacts.
>
>   *   I did an install of the operator and ran the basic sample
>   *   I checked the checksums
>   *   Checked the GPG signatures
>   *   Ran the UI
>   *   Ran a Twistlock scan
>   *   I installed 1.6 then did a helm upgrade
>   *   I have not managed to do the source build and subsequent install yet.
> I wanted to check these 2 things are what you would expect:
>
>   1.  I followed link
> https://github.com/apache/flink-kubernetes-operator/pkgs/container/flink-kubernetes-operator/139454270?tag=51eeae1
> And notice that it does not have a description . Is this correct?
>
>   1.  I get this in the gpg verification . Is this ok?
>
>
> gpg --verify flink-kubernetes-operator-1.6.1-src.tgz.asc
>
> gpg: assuming signed data in 'flink-kubernetes-operator-1.6.1-src.tgz'
>
> gpg: Signature made Fri 20 Oct 2023 04:07:48 PDT
>
> gpg:using RSA key B2D64016B940A7E0B9B72E0D7D0528B28037D8BC
>
> gpg: Good signature from "Rui Fan fan...@apache.org fan...@apache.org>" [unknown]
>
> gpg: WARNING: This key is not certified with a trusted signature!
>
> gpg:  There is no indication that the signature belongs to the
> owner.
>
> Primary key fingerprint: B2D6 4016 B940 A7E0 B9B7  2E0D 7D05 28B2 8037 D8BC
>
>
>
>
> Hi Everyone,
>
> Please review and vote on the release candidate #1 for the version 1.6.1 of
> Apache Flink Kubernetes Operator,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> **Release Overview**
>
> As an overview, the release consists of the following:
> a) Kubernetes Operator canonical source distribution (including the
> Dockerfile), to be deployed to the release repository at dist.apache.org
> b) Kubernetes Operator Helm Chart to be deployed to the release repository
> at dist.apache.org
> c) Maven artifacts to be deployed to the Maven Central Repository
> d) Docker image to be pushed to dockerhub
>
> **Staging Areas to Review**
>
> The staging areas containing the above mentioned artifacts are as follows,
> for your review:
> * All artifacts for a,b) can be found in the corresponding dev repository
> at dist.apache.org [1]
> * All artifacts for c) can be found at the Apache Nexus Repository [2]
> * The docker image for d) is staged on github [3]
>
> All artifacts are signed with the
> key B2D64016B940A7E0B9B72E0D7D0528B28037D8BC [4]
>
> Other links for your review:
> * source code tag "release-1.6.1-rc1" [5]
> * PR to update the website Downloads page to
> include Kubernetes Operator links [6]
> * PR to update the doc version of flink-kubernetes-operator[7]
>
> **Vote Duration**
>
> The voting time will run for at least 72 hours.
> It is adopted by majority approval, with at least 3 PMC affirmative votes.
>
> **Note on Verification**
>
> You can follow the basic verification guide here[8].
> Note that you don't need to verify everything yourself, but please make
> note of what you have tested together with your +- vote.
>
> [1]
>
> https://dist.apache.org/repos/dist/dev/flink/flink-kubernetes-operator-1.6.1-rc1/
> [2]
> https://repository.apache.org/content/repositories/orgapacheflink-1663/
> [3]
>
> https://github.com/apache/flink-kubernetes-operator/pkgs/container/flink-kubernetes-operator/139454270?tag=51eeae1
> [4] https://dist.apache.org/repos/dist/release/flink/KEYS
> [5]
> https://github.com/apache/flink-kubernetes-operator/tree/release-1.6.1-rc1
> [6] https://github.com/apache/flink-web/pull/690
> [7] https://github.com/apache/flink-kubernetes-operator/pull/687
> [8]
>
> https://cwiki.apache.org/confluence/display/FLINK/Verifying+a+Flink+Kubernetes+Operator+Release
>
> Best,
> Rui
>
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>

Re: Re: [DISCUSS] Remove legacy Paimon (TableStore) doc link from Flink web navigation

[Márton Balassi]

Hi all,

Thanks for your responses.
@Jingsong Li: Thanks for the reference to the web PR, I missed that.

@Yun Tang: Thanks, I prefer simply removing the TableStore link from the
documentation navigation of Flink, as it is not a subproject of Flink
anymore - it is now its own project. It has had 2 of its own releases over
a ~half a year period.
I am all for having a proper links to Paimon, for example we could create a
"Sister Projects" subsection in the About section of the flink.apache.org
webpage and have a paragraph of intro/links there or simply add Paimon
related content to the Table connector docs [1]. We can make these 2
changes separately, but ideally they should be merged relatively close in
time.

Would you be open to updating your original PR [2] to simply remove the
links or would you like me to do it instead? I am happy to review your
change if you have a proposal of where to include Paimon.

[1]
https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/table/overview/
[2] https://github.com/apache/flink-web/pull/665

On Sat, Oct 21, 2023 at 7:33 AM Yun Tang  wrote:

> Hi  devs,
>
> I am supporting to update the links. As the PR [1] author, I originally
> wanted to keep the legacy link to the Flink table store (0.3) as that was
> part of the history of Flink. And I plan to add the new Paimon's main
> branch link to tell users from Flink that this is the latest version. WDYT?
>
> [1] https://github.com/apache/flink-web/pull/665
>
> Best
> Yun Tang
>
> 
> From: junjie201...@gmail.com 
> Sent: Friday, October 20, 2023 12:20
> To: Jing Ge ; dev 
> Cc: dev 
> Subject: Re: Re: [DISCUSS] Remove legacy Paimon (TableStore) doc link from
> Flink web navigation
>
> +1
>
> On Tue, Oct 17, 2023 at 11:13 AM Yong Fang  wrote:
>
> > +1
> >
> > On Tue, Oct 17, 2023 at 4:52 PM Leonard Xu  wrote:
> >
> > > +1
> > >
> > >
> > > > 2023年10月17日 下午4:50，Martijn Visser  写道：
> > > >
> > > > +1
> > > >
> > > > On Tue, Oct 17, 2023 at 10:34 AM Jingsong Li  >
> > > wrote:
> > > >>
> > > >> Hi marton,
> > > >>
> > > >> Thanks for driving. +1
> > > >>
> > > >> There is a PR to remove legacy Paimon
> > > >> https://github.com/apache/flink-web/pull/665 , but it hasn't been
> > > >> updated for a long time.
> > > >>
> > > >> Best,
> > > >> Jingsong
> > > >>
> > > >> On Tue, Oct 17, 2023 at 4:28 PM Márton Balassi  >
> > > wrote:
> > > >>>
> > > >>> Hi Flink & Paimon devs,
> > > >>>
> > > >>> The Flink webpage documentation navigation section still lists the
> > > outdated TableStore 0.3 and master docs as subproject docs (see
> > > attachment). I am all for advertising Paimon as a sister project of
> > Flink,
> > > but the current state is misleading in multiple ways.
> > > >>>
> > > >>> I would like to remove these obsolete links if the communities
> agree.
> > > >>>
> > > >>> Best,
> > > >>> Marton
> > >
> > >
> >
>

[DISCUSS] Remove legacy Paimon (TableStore) doc link from Flink web navigation

[Márton Balassi]

Hi Flink & Paimon devs,

The Flink webpage documentation navigation section still lists the outdated
TableStore 0.3 and master docs as subproject docs (see attachment). I am
all for advertising Paimon as a sister project of Flink, but the current
state is misleading in multiple ways.

I would like to remove these obsolete links if the communities agree.

Best,
Marton

Re: [VOTE] FLIP-371: Provide initialization context for Committer creation in TwoPhaseCommittingSink

[Márton Balassi]

+1 (binding)

Marton

On Wed, Oct 11, 2023 at 8:20 PM Gyula Fóra  wrote:

> Thanks , Peter.
>
> +1
>
> Gyula
>
> On Wed, 11 Oct 2023 at 14:47, Péter Váry 
> wrote:
>
> > Hi all,
> >
> > Thank you to everyone for the feedback on FLIP-371[1].
> > Based on the discussion thread [2], I think we are ready to take a vote
> to
> > contribute this to Flink.
> > I'd like to start a vote for it.
> > The vote will be open for at least 72 hours (excluding weekends, unless
> > there is an objection or an insufficient number of votes).
> >
> > Thanks,
> > Peter
> >
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-371%3A+Provide+initialization+context+for+Committer+creation+in+TwoPhaseCommittingSink
> > [2] https://lists.apache.org/thread/v3mrspdlrqrzvbwm0lcgr0j4v03dx97c
> >
>

Re: [DISCUSS] FLIP-371: SinkV2 Committer imporvements

[Márton Balassi]

Thanks, Peter. I agree that this is needed for Iceberg and beneficial for
other connectors too.

+1

On Wed, Oct 4, 2023 at 3:56 PM Péter Váry 
wrote:

> Hi Team,
>
> In my previous email[1] I have described our challenges migrating the
> existing Iceberg SinkFunction based implementation, to the new SinkV2 based
> implementation.
>
> As a result of the discussion around that topic, I have created the first
> [2] of the FLIP-s addressing the missing features there.
>
> The main goal of the FLIP-371 is to extend the currently existing Committer
> API by providing an initial context on Committer creation. This context
> will contain - among other, less interesting data -
> the SinkCommitterMetricGroup which could be used to store the generic
> commit related metrics, and also provide a way for the Committer to publish
> its own metrics.
>
> The feature has already been requested through FLINK-25857 [3].
>
> Please use this thread to discuss the FLIP related questions, proposals,
> feedback.
>
> Thanks,
> Peter
>
> - [1] https://lists.apache.org/thread/h3kg7jcgjstpvwlhnofq093vk93ylgsn
> - [2]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-371%3A+SinkV2+Committer+imporvements
> - [3] https://issues.apache.org/jira/browse/FLINK-25857
>

Re: [DISCUSS] Drop python 3.7 support in 1.19

[Márton Balassi]

Hi Gabor,

Thanks for bringing this up. Similarly to when we dropped Python 3.6 due to
its end of life (and added 3.10) in Flink 1.17 [1,2], it makes sense to
proceed to remove 3.7 and add 3.11 instead.

+1.

[1] https://issues.apache.org/jira/browse/FLINK-27929
[2] https://github.com/apache/flink/pull/21699

Best,
Marton

On Fri, Sep 1, 2023 at 10:39 AM Gabor Somogyi 
wrote:

> Hi All,
>
> I've analyzed through part of the pyflink code and found some improvement
> possibilities.
> I would like to hear voices on the idea.
>
> Intention:
> * upgrade several python related versions to eliminate end-of-life issues
> and keep up with bugfixes
> * start to add python arm64 support
>
> Actual situation:
> * Flink supports the following python versions: 3.7, 3.8, 3.9, 3.10
> * We use miniconda 4.7.10 (python package management system and environment
> management system) which supports the following python versions: 3.7, 3.8,
> 3.9, 3.10
> * Our python framework is not supporting anything but x86_64
>
> Issues:
> * Python 3.7.17 is the latest security patch of the 3.7 line. This version
> is end-of-life and is no longer supported:
> https://www.python.org/downloads/release/python-3717/
> * Miniconda 4.7.10 is released on 2019-07-29 which is 4 years old already
> and not supporting too many architectures (x86_64 and ppc64le)
> * The latest miniconda which has real multi-arch feature set supports the
> following python versions: 3.8, 3.9, 3.10, 3.11 and no 3.7 support
>
> Suggestion to solve the issues:
> * In 1.19 drop python 3.7 support and upgrade miniconda to the latest
> version which opens the door to other platform + python 3.11 support
>
> Please note python 3.11 support is not initiated/discussed here.
>
> BR,
> G
>

Re: [VOTE] Apache Flink Kubernetes Operator Release 1.6.0, release candidate #2

[Márton Balassi]

Thank you, team.

+1 (binding)

- Verified Helm repo works as expected, points to correct image tag, build,
version
- Verified basic examples + checked operator logs everything looks as
expected
- Verified hashes, signatures and source release contains no binaries
- Ran built-in tests, built jars + docker image from source successfully

Best,
Marton

On Mon, Aug 14, 2023 at 1:24 PM Rui Fan <1996fan...@gmail.com> wrote:

> Thanks Gyula for the release!
>
> +1 (non-binding)
>
> - Compiled and tested the source code via mvn verify
> - Verified the signatures
> - Downloaded the image
> - Deployed helm chart to test cluster
> - Ran example job
>
> Best,
> Rui
>
> On Mon, Aug 14, 2023 at 3:58 PM Gyula Fóra  wrote:
>
> > +1 (binding)
> >
> > Verified:
> >  - Hashes, signatures, source files contain no binaries
> >  - Maven repo contents look good
> >  - Verified helm chart, image, deployed stateful and autoscaling
> examples.
> > Operator logs look good
> >
> > Cheers,
> > Gyula
> >
> > On Thu, Aug 10, 2023 at 3:03 PM Gyula Fóra  wrote:
> >
> > > Hi Everyone,
> > >
> > > Please review and vote on the release candidate #2 for the
> > > version 1.6.0 of Apache Flink Kubernetes Operator,
> > > as follows:
> > > [ ] +1, Approve the release
> > > [ ] -1, Do not approve the release (please provide specific comments)
> > >
> > > **Release Overview**
> > >
> > > As an overview, the release consists of the following:
> > > a) Kubernetes Operator canonical source distribution (including the
> > > Dockerfile), to be deployed to the release repository at
> dist.apache.org
> > > b) Kubernetes Operator Helm Chart to be deployed to the release
> > repository
> > > at dist.apache.org
> > > c) Maven artifacts to be deployed to the Maven Central Repository
> > > d) Docker image to be pushed to dockerhub
> > >
> > > **Staging Areas to Review**
> > >
> > > The staging areas containing the above mentioned artifacts are as
> > follows,
> > > for your review:
> > > * All artifacts for a,b) can be found in the corresponding dev
> repository
> > > at dist.apache.org [1]
> > > * All artifacts for c) can be found at the Apache Nexus Repository [2]
> > > * The docker image for d) is staged on github [3]
> > >
> > > All artifacts are signed with the key 21F06303B87DAFF1 [4]
> > >
> > > Other links for your review:
> > > * JIRA release notes [5]
> > > * source code tag "release-1.6.0-rc2" [6]
> > > * PR to update the website Downloads page to
> > > include Kubernetes Operator links [7]
> > >
> > > **Vote Duration**
> > >
> > > The voting time will run for at least 72 hours.
> > > It is adopted by majority approval, with at least 3 PMC affirmative
> > votes.
> > >
> > >
> > > **Note on Verification**
> > >
> > > You can follow the basic verification guide here[8].
> > > Note that you don't need to verify everything yourself, but please make
> > > note of what you have tested together with your +- vote.
> > >
> > > Cheers!
> > > Gyula Fora
> > >
> > > [1]
> > >
> >
> https://dist.apache.org/repos/dist/dev/flink/flink-kubernetes-operator-1.6.0-rc2/
> > > [2]
> > >
> https://repository.apache.org/content/repositories/orgapacheflink-1649/
> > > [3] ghcr.io/apache/flink-kubernetes-operator:ebb1fed
> > > [4] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > [5]
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12353230
> > > [6]
> > >
> >
> https://github.com/apache/flink-kubernetes-operator/tree/release-1.6.0-rc2
> > > [7] https://github.com/apache/flink-web/pull/666
> > > [8]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Verifying+a+Flink+Kubernetes+Operator+Release
> > >
> >
>

Re: [DISCUSS] Proposing an LTS Release for the 1.x Line

[Márton Balassi]

Hi team,

+1 for supporting the last 1.x for a longer than usual period of time and
limiting it to bugfixes. I would suggest supporting it for double the usual
amount of time (4 minor releases).

On Tue, Jul 25, 2023 at 9:25 AM Konstantin Knauf  wrote:

> Hi Alex,
>
> yes, I think, it makes sense to support the last 1.x release longer than
> usual. This should be limited to bugfixes in my opinion.
>
> Best,
>
> Konstantin
>
> Am Di., 25. Juli 2023 um 07:07 Uhr schrieb Xintong Song <
> tonysong...@gmail.com>:
>
> > Hi Alex,
> >
> > Providing a longer supporting period for the last 1.x minor release makes
> > sense to me.
> >
> > I think we need to be more specific about what LTS means here.
> >
> >- IIUC, that means for the last 1.x minor release, we will keep
> >providing 1.x.y / 1.x.z bugfix release. This is a stronger support
> > compared
> >to regular minor releases which by default are only supported for 2
> > minor
> >release cycles.
> >- Do we only provide bug fixes for the LTS release, or do we also
> allow
> >backporting features to that release?
> >- How long exactly shall we support the LTS release?
> >
> > And maybe we can make this a general convention for last minor releases
> for
> > all major releases, rather than only discuss it for the 2.0 version bump.
> >
> > @Leonard,
> >
> > I'd like to clarify that there are no community decisions yet on release
> > 2.0 after 1.19. It is possible to have 1.20 before 2.0.
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Tue, Jul 25, 2023 at 11:54 AM Leonard Xu  wrote:
> >
> > > +1, it’s pretty necessary especially we deprecated so many APIs in 1.18
> > > and plan to remove in 2.0.
> > >
> > > The 1.19 should be a proper version for LTS Release.
> > >
> > > Best,
> > > Leonard
> > >
> > >
> > > > On Jul 25, 2023, at 3:30 AM, Alexander Fedulov <
> > > alexander.fedu...@gmail.com> wrote:
> > > >
> > > > Hello everyone,
> > > >
> > > > Recently, there were a lot of discussions about the deprecation of
> > > various
> > > > APIs for the upcoming 2.0 release. It appears there are two main
> > > motivations
> > > > with opposing directions, causing these discussions to remain
> > unsettled.
> > > On
> > > > one hand, there's a desire to finally trim a wide range of legacy
> APIs,
> > > some
> > > > lingering around since the beginning of the 1.x release line (as far
> > > back as
> > > > 2016). On the other hand, there is a commitment to uphold our
> > guarantees
> > > to
> > > > the users, ensuring a smooth transition.
> > > >
> > > > I believe we could reconcile these two motivations. My proposition is
> > to
> > > > designate the final release of the 1.x timeline as a Long-Term
> Support
> > > (LTS)
> > > > release. By doing so, we would:
> > > >
> > > > 1. Enable more efficient cleanup and be liberated to introduce more
> > > breaking
> > > >   changes, paving the way for greater innovation in the 2.0 release.
> > > > 2. Sustain a positive user experience by granting enough time for the
> > > > changes
> > > >   introduced in 2.0 to stabilize, allowing users to confidently
> > > transition
> > > >   their production code to the new release.
> > > >
> > > > I look forward to hearing your thoughts on this proposal.
> > > >
> > > > Best Regards,
> > > > Alex
> > >
> > >
> >
>
>
> --
> https://twitter.com/snntrable
> https://github.com/knaufk
>

Re: Kubernetes Operator 1.6.0 release planning

[Márton Balassi]

Thanks, Gyula.

+1 (binding)

On Thu, Jul 20, 2023 at 5:01 AM Samrat Deb  wrote:

> thank you gyula ,
> for driving it.
> +1(non binding)
>
>
> Bests,
> Samrat
>
> On Thu, 20 Jul 2023 at 8:02 AM, Rui Fan <1996fan...@gmail.com> wrote:
>
> > Thanks Gyula for driving this release.
> >
> > +1 for the timeline
> >
> > Best,
> > Rui Fan
> >
> > On Wed, Jul 19, 2023 at 11:03 PM Gyula Fóra  wrote:
> >
> > > Hi Devs!
> > >
> > > Based on our release schedule, it is about time for the next Flink K8s
> > > Operator minor release.
> > >
> > > There are still some minor work items to be completed this week, but I
> > > suggest aiming for next Wednesday (July 26th) as the 1.6.0 release-cut
> -
> > > RC1 date.
> > >
> > > I am volunteering as the release manager but if someone else wants to
> do
> > > it, I would also be happy to simply give assistance :)
> > >
> > > Please let me know if you agree or disagree with the suggested
> timeline.
> > >
> > > Cheers,
> > > Gyula
> > >
> >
>

Re: [ANNOUNCE] Apache Flink Kubernetes Operator 1.5.0 released

[Márton Balassi]

Thanks, awesome! :-)

On Wed, May 17, 2023 at 2:24 PM Gyula Fóra  wrote:

> The Apache Flink community is very happy to announce the release of Apache
> Flink Kubernetes Operator 1.5.0.
>
> The Flink Kubernetes Operator allows users to manage their Apache Flink
> applications and their lifecycle through native k8s tooling like kubectl.
>
> Release highlights:
>  - Autoscaler improvements
>  - Operator stability, observability improvements
>
> Release blogpost:
>
> https://flink.apache.org/2023/05/17/apache-flink-kubernetes-operator-1.5.0-release-announcement/
>
> The release is available for download at:
> https://flink.apache.org/downloads.html
>
> Maven artifacts for Flink Kubernetes Operator can be found at:
> https://search.maven.org/artifact/org.apache.flink/flink-kubernetes-operator
>
> Official Docker image for Flink Kubernetes Operator applications can be
> found at: https://hub.docker.com/r/apache/flink-kubernetes-operator
>
> The full release notes are available in Jira:
> https://issues.apache.org/jira/projects/FLINK/versions/12352931
>
> We would like to thank all contributors of the Apache Flink community who
> made this release possible!
>
> Regards,
> Gyula Fora
>

Re: [VOTE] Apache Flink Kubernetes Operator Release 1.5.0, release candidate #2

[Márton Balassi]

Thank you, team.

+1 (binding)

- Verified Helm repo works as expected, points to correct image tag, build,
version
- Verified basic examples + checked operator logs everything looks as
expected
- Verified hashes, signatures and source release contains no binaries
- Ran built-in tests, built jars + docker image from source successfully

Best,
Marton

On Tue, May 16, 2023 at 3:25 PM Maximilian Michels  wrote:

> +1 (binding)
>
> 1. Downloaded the archives, checksums, and signatures
> 2. Verified the signatures and checksums
> 3. Extract and inspect the source code for binaries
> 4. Verified license files / headers
> 5. Compiled and tested the source code via mvn verify
> 6. Deployed helm chart to test cluster
> 7. Ran example job
>
> -Max
>
> On Tue, May 16, 2023 at 3:10 AM Jim Busche  wrote:
> >
> > +1 (Non-binding)
> > I tested the following:
> >
> > - helm repo install from flink-kubernetes-operator-1.5.0-helm.tgz (See
> note 1 below)
> > - podman Dockerfile build from source, looked good. (See note 2 below)
> > - twistlock vulnerability scans of proposed
> ghcr.io/apache/flink-kubernetes-operator:be07be7 looks good, except for
> known Snake item.
> > - UI, basic sample, basic session jobs look good. Logs look as expected.
> > - Checksums looked good
> > - Tested OLM build/install on OpenShift 4.10.54 and OpenShift 4.12.7
> >
> > Note 1: To install on OpenShift, I had to add an extra flink-operator
> clusterrole resource.  See
> https://github.com/apache/flink-kubernetes-operator/pull/600 and issue
> https://issues.apache.org/jira/browse/FLINK-32103
> >
> > Note 2: For some reason, I can't use podman on Red Hat 8 to build Flink,
> but the Podman from Red Hat 9.0 worked fine.
> >
> >
> > Thanks, Jim
>

Re: [VOTE] Apache Flink Kubernetes Operator Release 1.5.0, release candidate #1

[Márton Balassi]

Hi Jim and Ted,

Thanks for the quick response. For Openshift issue I would assume that
adding the RBAC suggested here [1] would solve the problem, it seems fine
to me.

For the missing taskmanager could you please share the relevant logs from
your jobmanager pod that is already show running? Thanks!

[1]
https://github.com/FairwindsOps/rbac-manager/issues/180#issuecomment-752706810

On Fri, May 12, 2023 at 8:40 AM Hao t Chang  wrote:

> Seems missing taskmanager pod. I tried the following:
> kubectl create -f
> https://github.com/jetstack/cert-manager/releases/download/v1.8.2/cert-manager.yaml
> helm install 1.5rc1
> https://dist.apache.org/repos/dist/dev/flink/flink-kubernetes-operator-1.5.0-rc1/flink-kubernetes-operator-1.5.0-helm.tgz
> kubectl create -f
> https://raw.githubusercontent.com/apache/flink-kubernetes-operator/release-1.5/examples/basic.yaml
> kubectl get po
> NAME READY   STATUSRESTARTS
>  AGE
> basic-example-67bbc79dd9-blfn4   1/1 Running   0
> 2m28s
> flink-kubernetes-operator-7bd6dcdfd4-2rshp   2/2 Running   0
> 4m49s
>

Re: Kubernetes Operator 1.5.0 release planning

[Márton Balassi]

+1, thanks.

On Mon, May 1, 2023 at 4:23 PM Őrhidi Mátyás 
wrote:

> +1 SGTM.
>
> Cheers,
> Matyas
>
> On Wed, Apr 26, 2023 at 11:43 AM Hao t Chang  wrote:
>
> > Agree. I will help.
> >
> >
>

Re: [VOTE] Apache Flink Kubernetes Operator Release 1.4.0, release candidate #1

[Márton Balassi]

Thank you, Gyula.

+1 (binding)

- Verified Helm repo works as expected, points to correct image tag, build,
version
- Verified basic examples + checked operator logs everything looks as
expected
- Verified hashes, signatures and source release contains no binaries
- Ran built-in tests, built jars + docker image from source successfully

Best,
Marton

On Mon, Feb 20, 2023 at 2:58 PM Gyula Fóra  wrote:

> +1 (binding)
>
>  - Downloaded archives, validated binary content, checksum signatures,
> NOTICE files
>  - Installed to local k8s cluster from Helm repo, verified docker image,
> logs
>  - Ran some basic examples, ran autoscaler example
>
>  Cheers
> Gyula
>
> On Sun, Feb 19, 2023 at 5:09 AM Peter Huang 
> wrote:
>
> > Thanks for preparing the release!
> >
> > +1 (non-binding)
> >
> > 1. Downloaded the archives, checksums, and signatures
> > 2. Extract and inspect the source code for binaries
> > 3. Compiled and tested the source code via mvn verify
> > 4. Deployed helm chart from flink-kubernetes-operator-1.4.0-helm.tgz to
> > local minikube cluster
> > 5. Ran example jobs with V1_14, V1_15, V1_16, and verified operator logs
> > 6. Built the docker container locally and verified it through repeatting
> > step 5
> >
> > Best Regards
> > Peter Huang
> >
> > On Fri, Feb 17, 2023 at 8:53 PM Jim Busche  wrote:
> >
> > > Thanks Gyula for the release.
> > >
> > > +1 (non-binding)
> > >
> > >
> > > I tested the following:
> > >
> > >   *   Helm repo install from flink-kubernetes-operator-1.4.0-helm.tgz
> > >   *   Podman Dockerfile build from source, looked good.
> > >   *   Twistlock security scans of proposed image looks good.  No
> > currently
> > > known vulnerabilities with the built image or
> > > ghcr.io/apache/flink-kubernetes-operator:7fc23a1
> > >   *   UI, basic sample, basic session jobs look good. Logs look as
> > > expected.
> > >   *   Checksums looked good
> > >   *   Tested OLM build/install on OpenShift 4.12
> > >   *   Verified that sessionjob correctly can write in the operator’s
> > > /opt/flink/artifacts filesystem on OpenShift even in a non-default
> > > namespace.
> > >
> > >
> > >
> > > Thanks, Jim
> > >
> >
>

Re: Proposal to add support for Kafka Headers to KafkaRecordSerializationSchemaWrapper

[Márton Balassi]

Hi Alex,

Please do share, this comes up somewhat frequently.

Marton

On Mon, Feb 13, 2023 at 7:44 PM Őrhidi Mátyás 
wrote:

> Hi Alex,
>
> This is a reasonable request IMO. I've recently bumped into this topic
> myself. This could be handy for supporting schema registries in Kafka to
> Kafka scenarios for example. Looking forward to your proposal.
>
> Cheers,
> Matyas
>
> On Mon, Feb 13, 2023 at 7:08 AM Alex Gout 
> wrote:
>
> > Hi all,
> >
> > I'm currently working on a few pipelines sinking to Kafka. The downstream
> > consumers of the sink topics expect some Kafka headers to be set. However
> > the default org.apache.flink.connector.kafka.sink.KafkaSink does
> > not support adding Kafka record headers.
> >
> > I tracked the code path down to
> >
> org.apache.flink.connector.kafka.sink.KafkaRecordSerializationSchemaWrapper
> > where the RecordProducer is created.
> > It is relatively simple to add support for record headers by adding a
> > "HeaderProducer" next to the key and value serializers and using the
> > appropriate RecordProducer constructor.
> >
> > For the benefit of my own projects, I have implemented this header
> support
> > and would be eager to share my implementation as a proposal if there's a
> > consensus this would indeed be a valuable addition.
> >
> > Please let me know what you think.
> > Thanks,
> > - Alex
> >
>

Re: [VOTE] Apache Flink Kubernetes Operator Release 1.3.1, release candidate #1

[Márton Balassi]

Thank you, Gyula.

+1 (binding)

- Verified Helm repo works as expected, points to correct image tag, build,
version
- Verified basic examples + checked operator logs everything looks as
expected
- Verified hashes, signatures and source release contains no binaries
- Ran built-in tests, built jars + docker image from source successfully

Best,
Marton

On Sun, Jan 1, 2023 at 7:43 PM Gyula Fóra  wrote:

> Hi everyone,
>
> Please review and vote on the release candidate #1 for the version 1.3.1 of
> Apache Flink Kubernetes Operator,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> **Release Overview**
>
> As an overview, the release consists of the following:
> a) Kubernetes Operator canonical source distribution (including the
> Dockerfile), to be deployed to the release repository at dist.apache.org
> b) Kubernetes Operator Helm Chart to be deployed to the release repository
> at dist.apache.org
> c) Maven artifacts to be deployed to the Maven Central Repository
> d) Docker image to be pushed to dockerhub
>
> **Staging Areas to Review**
>
> The staging areas containing the above mentioned artifacts are as follows,
> for your review:
> * All artifacts for a,b) can be found in the corresponding dev repository
> at dist.apache.org [1]
> * All artifacts for c) can be found at the Apache Nexus Repository [2]
> * The docker image for d) is staged on github [3]
>
> All artifacts are signed with the key 21F06303B87DAFF1 [4]
>
> Other links for your review:
> * JIRA release notes [5]
> * source code tag "release-1.3.1-rc1" [6]
> * PR to update the website Downloads page to include Kubernetes Operator
> links [7]
>
> **Vote Duration**
>
> The voting time will run for at least 72 hours.
> It is adopted by majority approval, with at least 3 PMC affirmative votes.
>
> **Note on Verification**
>
> You can follow the basic verification guide here[8].
> Note that you don't need to verify everything yourself, but please make
> note of what you have tested together with your +- vote.
>
> Happy New Year!
> Gyula Fora
>
> [1]
>
> https://dist.apache.org/repos/dist/dev/flink/flink-kubernetes-operator-1.3.1-rc1/
> [2]
> https://repository.apache.org/content/repositories/orgapacheflink-1571/
> [3] ghcr.io/apache/flink-kubernetes-operator:8e60e9b
> [4] https://dist.apache.org/repos/dist/release/flink/KEYS
> [5]
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12352697
> [6]
> https://github.com/apache/flink-kubernetes-operator/tree/release-1.3.1-rc1
> [7] https://github.com/apache/flink-web/pull/599
> [8]
>
> https://cwiki.apache.org/confluence/display/FLINK/Verifying+a+Flink+Kubernetes+Operator+Release
>

Re: Flink Forward Session Question

[Márton Balassi]

Hi Rion,

Unlike the previous Flink Forwards to the best of my knowledge the latest
edition was not uploaded to YouTube. It might make sense to reach out to
the authors directly.

On Sat, Dec 31, 2022 at 5:35 PM Rion Williams  wrote:

> Hey Flinkers,
>
> Firstly, early Happy New Year’s to everyone in the community. I’ve been
> digging a bit into exactly-once processing with Flink and Pinot and I came
> across this session from Flink Foward last year:
>
> -
> https://www.slideshare.net/FlinkForward/exactlyonce-financial-data-processing-at-scale-with-flink-and-pinot
>
> I was curious if anyone knew if this session was recording as the deck
> itself seemed to have quite a bit of value. I figured the mailing list
> would be a reasonable place to ask.
>
> Thanks in advance,
>
> Rion
>

Re: [DISCUSS] Release new FRocksDB

[Márton Balassi]

Hi Yanfei,

Makes sense, arm architecture support would be great. Thanks for looking
into this.

On Mon, Dec 12, 2022 at 3:03 PM Yanfei Lei  wrote:

> Hi devs,
>
> I'd like to bring up a discussion about releasing the new frocksdbjni
> version,  we are planning to adapt frocksdbjni to Apple M1 machines in
> Flink 1.17[1], a new frocksdbjni jar needs to be released.
>
> Thanks to the efforts of the community, FLINK-24932[1] has been merged into
> FRocksDB-6.20.3 branch, and many thanks to *Ververica* for providing a paid
> CircleCI for testing, which ensures the correctness of the code.
>
> The new frocksdbjni version is still based on 6.20.3[2], the code
> interfaces and performance[3] will not change. In addition to
> FLINK-24932[1],  some fixes of vulnerabilities[4] will also be included in
> the new version, see FLINK-30321[5].
>
> Are there any minor fixes you would like to include in this release?
> Welcome to create PR for FrocksDB[6].
> Looking forward to your opinions.
>
> [1] https://issues.apache.org/jira/browse/FLINK-24932
> [2]
>
> https://mvnrepository.com/artifact/com.ververica/frocksdbjni/6.20.3-ververica-1.0
> [3]
>
> https://issues.apache.org/jira/browse/FLINK-24932?focusedCommentId=17569889&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17569889
> [4] https://lists.apache.org/thread/rm40f45qfw6rls70k35o2dt0k4tz9bsr
> [5] https://issues.apache.org/jira/browse/FLINK-30321
> [6] https://github.com/ververica/frocksdb
>
> --
> Best,
> Yanfei
>

Re: [VOTE] Apache Flink Kubernetes Operator Release 1.3.0, release candidate #1

[Márton Balassi]

Thank you, Matyas.

+1 (binding)

- Verified Helm repo works as expected, points to correct image tag, build,
version
- Verified basic examples + checked operator logs everything looks as
expected
- Verified hashes, signatures and source release contains no binaries
- Ran built-in tests, built jars + docker image from source successfully

Best,
Marton

On Thu, Dec 8, 2022 at 2:41 AM Őrhidi Mátyás 
wrote:

> Hi everyone,
> Please review and vote on the release candidate #1 for the version 1.3.0 of
> Apache Flink Kubernetes Operator,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
> **Release Overview**
> As an overview, the release consists of the following:
> a) Kubernetes Operator canonical source distribution (including the
> Dockerfile), to be deployed to the release repository at dist.apache.org
> b) Kubernetes Operator Helm Chart to be deployed to the release repository
> at dist.apache.org
> c) Maven artifacts to be deployed to the Maven Central Repository
> d) Docker image to be pushed to dockerhub
> **Staging Areas to Review**
> The staging areas containing the above mentioned artifacts are as follows,
> for your review:
> * All artifacts for a,b) can be found in the corresponding dev repository
> at dist.apache.org [1]
> * All artifacts for c) can be found at the Apache Nexus Repository [2]
> * The docker image for d) is staged on github [3]
> All artifacts are signed with the key
> ACD1386E2C4D07053D775A0648E78F054AA33CB5 [4]
> Other links for your review:
> * JIRA release notes [5]
> * source code tag "release-1.3.0-rc1" [6]
> * PR to update the website Downloads page to include Kubernetes Operator
> links [7]
> **Vote Duration**
> The voting time will run for at least 72 hours.
> It is adopted by majority approval, with at least 3 PMC affirmative votes.
> **Note on Verification**
> You can follow the basic verification guide here[8].
> Note that you don't need to verify everything yourself, but please make
> note of what you have tested together with your +- vote.
> Thanks,
> Matyas Orhidi
> [1]
>
> https://dist.apache.org/repos/dist/dev/flink/flink-kubernetes-operator-1.3.0-rc1/
> [2]
> https://repository.apache.org/content/repositories/orgapacheflink-1559/
> [3] ghcr.io/apache/flink-kubernetes-operator:b7e40da
> [4] https://dist.apache.org/repos/dist/release/flink/KEYS
> [5]
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12352322
> [6]
> https://github.com/apache/flink-kubernetes-operator/tree/release-1.3.0-rc1
> [7] https://github.com/apache/flink-web/pull/593
> [8]
>
> https://cwiki.apache.org/confluence/display/FLINK/Verifying+a+Flink+Kubernetes+Operator+Release
>

Re: [VOTE] FLIP-271: Autoscaling

[Márton Balassi]

+1 (binding)

On Tue, Nov 29, 2022 at 6:13 AM Chenya Zhang 
wrote:

> +1 (non-binding)
>
> On Sun, Nov 27, 2022 at 5:49 PM Jiangang Liu 
> wrote:
>
> > +1 (non-binding)
> >
> > Best,
> > Jiangang Liu
> >
> > Thomas Weise  于2022年11月28日周一 06:23写道：
> >
> > > +1 (binding)
> > >
> > >
> > > On Sat, Nov 26, 2022 at 8:11 AM Zheng Yu Chen 
> > wrote:
> > >
> > > > +1(no-binding)
> > > >
> > > > Maximilian Michels  于 2022年11月24日周四 上午12:25写道：
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > I'd like to start a vote for FLIP-271 [1] which we previously
> > discussed
> > > > on
> > > > > the dev mailing list [2].
> > > > >
> > > > > I'm planning to keep the vote open for at least until Tuesday, Nov
> > 29.
> > > > >
> > > > > -Max
> > > > >
> > > > > [1]
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-271%3A+Autoscaling
> > > > > [2]
> https://lists.apache.org/thread/pvfb3fw99mj8r1x8zzyxgvk4dcppwssz
> > > > >
> > > >
> > >
> >
>

Re: Kubernetes Operator 1.3.0 release

[Márton Balassi]

Hi all,

Thanks for volunteering, Matyas and +1 for the 1.3 operator release.
The modified timeline suggested by Gyula sounds good to me.

On Tue, Nov 29, 2022 at 7:04 AM Őrhidi Mátyás 
wrote:

> Thanks Gyula! Sounds good.
>
> On Mon, Nov 28, 2022 at 9:59 PM Gyula Fóra  wrote:
>
> > Hi Matyas!
> >
> > +1 for the 1.3.0 release.
> >
> > Thank you for the proposal and for volunteering as a release manager.
> >
> > Given that it's already Nov 29 here, I suggest moving the feature freeze
> > to the end of this week (Dec 2) and release cut to mid next week (Dec 7)
> to
> > give us some flexibility in merging the currently pending PRs.
> >
> > Thanks,
> > Gyula
> >
> > On Tue, Nov 29, 2022 at 6:44 AM Őrhidi Mátyás 
> > wrote:
> >
> >> Hi Devs,
> >>
> >> Our planned milestones for the upcoming operator 1.3.0 release are the
> >> following:
> >> Feature Freeze:   Nov 28
> >> Release Cut:Dec 5
> >> Release Date:  Dec 12
> >>
> >> There are a few JIRAs that still need to be addressed, but most of them
> >> are work in progress, so it seems we can most probably meet these
> deadlines.
> >>
> >> I volunteer as the release manager.
> >>
> >> Cheers,
> >> Matyas
> >>
> >
>

[ANNOUNCE] New Apache Flink Committer - Matyas Orhidi

[Márton Balassi]

Hi everyone,

On behalf of the PMC, I'm very happy to announce Matyas Orhidi as a new
Flink
committer.

Matyas has over a decade of experience of the Big Data ecosystem and has
been working with Flink full time for the past 3 years. In the open source
community he is one of the key driving members of the Kubernetes Operator
subproject. He implemented multiple key features in the operator including
the metrics system and the ability to dynamically configure watched
namespaces. He enjoys spreading the word about Flink and regularly does so
via authoring blogposts and giving talks or interviews representing the
community.

Please join me in congratulating Matyas for becoming a Flink committer!

Best,
Marton

Re: [VOTE] FLIP-272: Generalized delegation token support

[Márton Balassi]

+1 (binding)

On Thu, Nov 17, 2022 at 9:06 AM Gabor Somogyi 
wrote:

> Hi All,
>
> I'm hereby opening a vote for FLIP-272 Generalized delegation token
> support.
> The related documents can be found here:
> - FLIP on wiki: [1]
> - Discussion thread: [2]
>
> Voting will be open for at least 72 hours (since weekend is involved EOB
> Monday is planned).
>
> BR,
> G
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-272%3A+Generalized+delegation+token+support
> [2] https://lists.apache.org/thread/vgg5hbf5jljcxopfhb32w3l0wjoyko4o
>

Re: [DISCUSS] Issue tracking workflow

[Márton Balassi]

Hi Martjin,

Given the situation let us set up the Jira signup mailing list following
the Calcite model. This seems the most sensible to me as of now.

On Fri, Nov 11, 2022 at 7:26 PM Martijn Visser 
wrote:

> Hi everyone,
>
> Unfortunately ASF Infra has already implemented the change and new Jira
> users can't sign up.
>
> I think there is consensus that we shouldn't move from Jira now. My
> proposal would be to setup a separate mailing list to which users can send
> their request for an account, which gets sent to the PMC so they can create
> accounts for them. I don't see any other short term solution.
>
> If agreed, let's open up a vote thread on this.
>
> Thanks, Martijn
>
>
> Op do 3 nov. 2022 om 04:51 schreef Xintong Song 
>
> > Thanks all for the valuable feedback, opinions and suggestions.
> >
> > # Option 1.
> > I know this is the first choice for pretty much everyone. Many people
> from
> > the Flink community (including myself) have shared their opinion with
> > Infra. However, based on the feedback so far, TBH I don't think things
> > would turn out the way we want. I don't see what else we can do. Does
> > anyone have more suggestions on this option? Or we probably have to
> > scratch it out of the list.
> >
> > # Option 4.
> > Seems there are also quite some concerns on using solely GH issues:
> limited
> > features (thus the significant changes to the current issue/release
> > management processes), migration cost, source of truth, etc. I think I'm
> > also convinced that this may not be a good choice.
> >
> > # Option 2 & 3.
> > Between the two options, I'm leaning towards option 2.
> > - IMO, making it as easy as possible for users to report issues should
> be a
> > top priority. Having to wait for a human response for creating an account
> > does not meet that requirement. That makes a strong objection to option 3
> > from my side.
> > - Using GH issues for consumer-facing issues and reflecting the valid
> ones
> > back to Jira (either manually by committers or by bot) sounds good to me.
> > The status (open/closed) and labels should make tracking the issues
> easier
> > compared to in mailing lists / slack, in terms of whether an issue has
> been
> > taken care of / reflected to Jira / closed as invalid. That does not mean
> > we should not reflect things from mailing lists / slack to Jira. Ideally,
> > we leverage every possible channel for collecting user issues / feedback,
> > while guiding / suggesting users to choose GH issues over the others.
> > - For new contributors, they still need to request an account from a PMC
> > member. They can even make that request on GH issues, if they do not mind
> > posting the email address publicly.
> > - I would not be worried very much about the privacy issue, if the Jira
> > account creation is restricted to contributors. Contributors are exposing
> > their email addresses publicly anyway, in dev@ mailing list and commit
> > history. I'm also not strongly against creating a dedicated mailing list
> > though.
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Wed, Nov 2, 2022 at 9:16 PM Chesnay Schepler 
> > wrote:
> >
> > > Calcite just requested a separate mailing list for users to request a
> > > JIRA account.
> > >
> > >
> > > I think I'd try going a similar route. While I prefer the openness of
> > > github issues, they are really limited, and while some things can be
> > > replicated with labels (like fix versions / components), things like
> > > release notes can't.
> > > We'd also lose a central place for collecting issues, since we'd have
> to
> > > (?) scope issues per repo.
> > >
> > > I wouldn't want to import everything into GH issues (it's just a flawed
> > > approach in the long-term imo), but on the other hand I don't know if
> > > the auto linker even works if it has to link to either jira or a GH
> > issue.
> > >
> > > Given that we need to change workflows in any case, I think I'd prefer
> > > sticking to JIRA.
> > > For reported bugs I'd wager that in most cases we can file the tickets
> > > ourselves and communicate with users on slack/MLs to gather all the
> > > information. I'd argue that if we'd had been more pro-active with
> filing
> > > tickets for user issues (instead of relying on them to do it) we
> > > would've addressed several issues way sooner.
> > >
> > > Additionally, since either option would be a sort of experiment, then
> > > JIRA is a safer option. We have to change less and there aren't any
> > > long-term ramifications (like having to re-import GH tickets into
> JIRA).
> > >
> > > On 28/10/2022 16:49, Piotr Nowojski wrote:
> > > > Hi,
> > > >
> > > > I'm afraid of the migration cost to only github issues and lack of
> many
> > > > features that we are currently using. That would be very disruptive
> and
> > > > annoying. For me github issues are far worse compared to using the
> > Jira.
> > > >
> > > > I would strongly prefer Option 1. over the others. Option 4 I would
> > like
> > > > the least. I would 

Re: [DISCUSS] FLIP-271: Autoscaling

[Márton Balassi]

Thanks for preparing the FLIP and kicking off the discussion, Max. Looking
forward to this. :-)

On Sat, Nov 5, 2022 at 9:27 AM Niels Basjes  wrote:

> I'm really looking forward to seeing this in action.
>
> Niels
>
> On Fri, 4 Nov 2022, 19:37 Maximilian Michels,  wrote:
>
>> Hi,
>>
>> I would like to kick off the discussion on implementing autoscaling for
>> Flink as part of the Flink Kubernetes operator. I've outlined an approach
>> here which I find promising:
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-271%3A+Autoscaling
>>
>> I've been discussing this approach with some of the operator contributors:
>> Gyula, Marton, Matyas, and Thomas (all in CC). We started prototyping an
>> implementation based on the current FLIP design. If that goes well, we
>> would like to contribute this to Flink based on the results of the
>> discussion here.
>>
>> I'm curious to hear your thoughts.
>>
>> -Max
>>
>

Re: [ANNOUNCE] Apache Flink 1.16.0 released

[Márton Balassi]

Awesome, thanks team! Thanks Xingbo for managing the release.

On Fri, Oct 28, 2022 at 11:04 AM Biao Liu  wrote:

> Congrats! Glad to hear that.
>
> BTW, I just found the document link of 1.16 from https://flink.apache.org/
> is not correct.
>
> [image: 截屏2022-10-28 17.01.28.png]
>
> Thanks,
> Biao /'bɪ.aʊ/
>
>
>
> On Fri, 28 Oct 2022 at 14:46, Xingbo Huang  wrote:
>
>> The Apache Flink community is very happy to announce the release of Apache
>> Flink 1.16.0, which is the first release for the Apache Flink 1.16 series.
>>
>> Apache Flink® is an open-source stream processing framework for
>> distributed, high-performing, always-available, and accurate data
>> streaming
>> applications.
>>
>> The release is available for download at:
>> https://flink.apache.org/downloads.html
>>
>> Please check out the release blog post for an overview of the
>> improvements for this release:
>> https://flink.apache.org/news/2022/10/28/1.16-announcement.html
>>
>> The full release notes are available in Jira:
>>
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351275
>>
>> We would like to thank all contributors of the Apache Flink community
>> who made this release possible!
>>
>> Regards,
>> Chesnay, Martijn, Godfrey & Xingbo
>>
>

Re: [VOTE] FLIP-265 Deprecate and remove Scala API support

[Márton Balassi]

+1 (binding)

On Mon, Oct 17, 2022 at 3:39 PM Martijn Visser 
wrote:

> Hi everyone,
>
> I'm hereby opening a vote for FLIP-265 Deprecate and remove Scala API
> support. The related discussion can be found here [1].
>
> Voting will be open for at least 72 hours.
>
> Best regards,
>
> Martijn
> https://twitter.com/MartijnVisser82
> https://github.com/MartijnVisser
>
> [1] https://lists.apache.org/thread/d3borhdzj496nnggohq42fyb6zkwob3h
>

Re: [DISCUSS] Changing the minimal supported version of Hadoop to 2.10.2

[Márton Balassi]

Hi Martjin,

+1 for 2.10.2. Do you expect to have bandwidth in the near term to
implement the bump?

On Wed, Oct 5, 2022 at 5:00 PM Gabor Somogyi 
wrote:

> Hi Martin,
>
> Thanks for bringing this up! Lately I was thinking about to bump the hadoop
> version to at least 2.6.1 to clean up issues like this:
>
> https://github.com/apache/flink/blob/8d05393f5bcc0a917b2dab3fe81a58acaccabf13/flink-filesystems/flink-hadoop-fs/src/main/java/org/apache/flink/runtime/util/HadoopUtils.java#L157-L159
>
> All in all +1 from my perspective.
>
> Just a question here. Are we stating the minimum Hadoop version for users
> somewhere in the doc or they need to find it out from source code like
> this?
>
> https://github.com/apache/flink/blob/3a4c11371e6f2aacd641d86c1d5b4fd86435f802/tools/azure-pipelines/build-apache-repo.yml#L113
>
> BR,
> G
>
>
> On Wed, Oct 5, 2022 at 5:02 AM Martijn Visser 
> wrote:
>
> > Hi everyone,
> >
> > Little over a year ago a discussion thread was opened on changing the
> > minimal supported version of Hadoop and bringing that to 2.8.5. [1] In
> this
> > discussion thread, I would like to propose to bring that minimal
> supported
> > version of Hadoop to 2.10.2.
> >
> > Hadoop 2.8.5 is vulnerable for multiple CVEs which are classified as
> > Critical. [2] [3]. While Flink is not directly impacted by those, we do
> see
> > vulnerability scanners flag Flink as being vulnerable. We could easily
> > mitigate that by bumping the minimal supported version of Hadoop to
> 2.10.2.
> >
> > I'm looking forward to your opinions on this topic.
> >
> > Best regards,
> >
> > Martijn
> > https://twitter.com/MartijnVisser82
> > https://github.com/MartijnVisser
> >
> > [1] https://lists.apache.org/thread/81fhnwfxomjhyy59f9bbofk9rxpdxjo5
> > [2] https://nvd.nist.gov/vuln/detail/CVE-2022-25168
> > [3] https://nvd.nist.gov/vuln/detail/CVE-2022-26612
> >
>

Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support

[Márton Balassi]

Hi Martjin,

After some more careful consideration I am in favor of dropping the Scala
API support in with Flink 2.0 given that we add Java 17 support earlier or
latest at the same time.

Best,
Marton

On Thu, Oct 13, 2022 at 12:01 PM Chesnay Schepler 
wrote:

> Support for records has not been investigated yet. We're still at the
> stage of getting things to run at all on Java 17.
>
> It _may_ be possible, it _may_ not be.
>
> On 13/10/2022 07:39, Salva Alcántara wrote:
>
> Hi Martijn,
>
> Maybe a bit of an off-topic, but regarding Java 17 support, will it be
> possible to replace POJOs with Java records in existing applications?
>
> In a project I maintain we use Lombok a lot, but with Java records we
> would probably stop using it (or significantly reduce its usage).
>
> Will there be a way to promote existing POJOs (either written "manually"
> or using Lombok) to Java records without breaking serialization? (assuming
> that those POJOs are used as immutable values, e.g., setters are never
> used).
>
> Regards,
>
> Salva
>
> On Wed, Oct 12, 2022 at 9:11 PM Martijn Visser 
> wrote:
>
>> Hi everyone,
>>
>> Thanks again for all your feedback. It's very much appreciated.
>>
>> My overall feeling is that people are not opposed to the FLIP. There is
>> demand for adding Java 17 support before dropping the Scala APIs. Given
>> that the proposal for actually dropping the Scala APIs would only happen
>> with a Flink 2.0 and Java 17 support would either happen in a new minor
>> version or a new major version (I haven't seen a FLIP or discussion being
>> opened adding Java 17 support, only on deprecating Java 8), Java 17 support
>> would either be there earlier (in a new minor version) or at the same time
>> (with Flink 2.0) when the Scala APIs would be dropped.
>>
>> If there are no more discussion topics, I would move this FLIP to a vote
>> at the beginning of next week.
>>
>> Best regards,
>>
>> Martijn
>>
>> On Sun, Oct 9, 2022 at 10:36 AM guenterh.lists 
>> wrote:
>>
>>> Hi Martijn
>>>
>>> I do not maintain a large production application based on Flink, so it
>>> would not be a problem for me to convert existing implementations to
>>> whatever API.
>>>
>>> I am working in the area of cultural heritage, which is mainly about the
>>> processing of structured (meta)-data (scientific libraries, archives and
>>> museums)
>>> My impression: People without much background/experience with Java
>>> implementations find it easier to get into the functional mindset as
>>> supported in Scala. That's why I think it would be very unfortunate if the
>>> use of Scala in Flink becomes more and more limited or neglected.
>>>
>>> I think using the Java API in Scala is a possible way also in my
>>> environment.
>>>
>>> In the last weeks I tried to port the examples from the "Flink Course"
>>> of Daniel Ciorcilan (https://rockthejvm.com/p/flink - he mainly offers
>>> Scala courses), which are exclusively based on the native Scala API, to the
>>> Java API. This has worked without any problems as far as I can see. So far
>>> I haven't tried any examples based on the Table API or streaming workflows
>>> in batch mode (which would be important for our environment).
>>>
>>> My main trouble: So far I don't know enough about the limitations of
>>> using the Java API in a Scala implementation and what that means. My
>>> current understanding: the limitation is mainly in deriving the type
>>> information in generic APIs with Scala types. For me it would be very
>>> significant and helpful if there would be more information, descriptions
>>> and examples about this topic.
>>>
>>> So far unfortunately I had too little time to deal with a wrapper like
>>> flink-scala-api (https://github.com/findify/flink-scala-api ) and the
>>> current alternative is probably going to be deprecated in the future (
>>> https://github.com/ariskk/flink4s/issues/17#issuecomment-1125806808 )
>>>
>>> Günter
>>>
>>>
>>> On 04.10.22 13:58, Martijn Visser wrote:
>>>
>>> Hi Marton,
>>>
>>> You're making a good point, I originally wanted to include already the
>>> User mailing list to get their feedback but forgot to do so. I'll do some
>>> more outreach via other channels as well.
>>>
>>> @Users of Flink, I've made a proposal to deprecate and remove Scala API
&

Re: Re: Re: [Discuss]- Donate Iceberg Flink Connector

[Márton Balassi]

+1 from me, thanks for the clarification.

On Wed, Oct 12, 2022 at 7:57 AM Péter Váry 
wrote:

> Thanks Abid,
>
> Count me in, and drop a note, if I can help in any way.
>
> Thanks,
> Peter
>
> On Tue, Oct 11, 2022, 20:13  wrote:
>
> > Hi Martijn,
> >
> > Yes catalog integration exists and catalogs can be created using Flink
> > SQL.
> >
> >
> https://iceberg.apache.org/docs/latest/flink/#creating-catalogs-and-using-catalogs
> > has more details.
> > We may need some discussion within Iceberg community but based on the
> > current iceberg-flink code structure we are looking to externalize this
> as
> > well.
> >
> > Thanks
> > Abid
> >
> >
> > On 2022/10/11 08:24:44 Martijn Visser wrote:
> > > Hi Abid,
> > >
> > > Thanks for the FLIP. I have a question about Iceberg's Catalog: has
> that
> > > integration between Flink and Iceberg been created already and are you
> > > looking to externalize that as well?
> > >
> > > Thanks,
> > >
> > > Martijn
> > >
> > > On Tue, Oct 11, 2022 at 12:14 AM  wrote:
> > >
> > > > Hi Marton,
> > > >
> > > > Yes, we are initiating this as part of the Externalize Flink
> Connectors
> > > > effort. Plan is to externalize the existing Flink connector from
> > Iceberg
> > > > repo into a separate repo under the Flink umbrella.
> > > >
> > > > Sorry about the doc permissions! I was able to create a FLIP-267:
> > > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP+267%3A+Iceberg+Connector
> > > > <
> > > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP+267%3A+Iceberg+Connector
> > > > >
> > > > Lets use that to discuss.
> > > >
> > > > Thanks
> > > > Abid
> > > >
> > > > On 2022/10/10 12:57:32 Márton Balassi wrote:
> > > > > Hi Abid,
> > > > >
> > > > > Just to clarify does your suggestion mean that the Iceberg
> community
> > > > would
> > > > > like to remove the iceberg-flink connector from the Iceberg
> codebase
> > and
> > > > > maintain it under Flink instead? A new separate repo under the
> Flink
> > > > > project umbrella given the current existing effort to extract
> > connectors
> > > > to
> > > > > their individual repos (externalize) makes sense to me.
> > > > >
> > > > > [1]
> https://lists.apache.org/thread/mpzzlpob9ymkjfybm96vz2y2m5fjyvfo
> > > > >
> > > > > Best,
> > > > > Marton
> > > > >
> > > > >
> > > > > On Mon, Oct 10, 2022 at 5:31 AM Jingsong Li 
> wrote:
> > > > >
> > > > > > Thanks Abid for driving.
> > > > > >
> > > > > > +1 for this.
> > > > > >
> > > > > > Can you open the permissions for
> > > > > >
> > > > > >
> > > >
> >
> https://docs.google.com/document/d/1WC8xkPiVdwtsKL2VSPAUgzm9EjrPs8ZRjEtcwv93ISI/edit?usp=sharing
> > > > > > ?
> > > > > >
> > > > > > Best,
> > > > > > Jingsong
> > > > > >
> > > > > > On Mon, Oct 10, 2022 at 9:22 AM Abid Mohammed
> > > > > >  wrote:
> > > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I would like to start a discussion about contributing Iceberg
> > Flink
> > > > > > Connector to Flink.
> > > > > > >
> > > > > > > I created a doc <
> > > > > >
> > > >
> >
> https://docs.google.com/document/d/1WC8xkPiVdwtsKL2VSPAUgzm9EjrPs8ZRjEtcwv93ISI/edit?usp=sharing
> > > > >
> > > > > > with all the details following the Flink Connector template as I
> > don’t
> > > > have
> > > > > > permissions to create a FLIP yet.
> > > > > > > High level details are captured below:
> > > > > > >
> > > > > > > Motivation:
> > > > > > >
> > > > > > > This FLIP aims to contribute the existing Apache Iceberg Flink
> > > > Connector
> > > > > > to Flink.
&

Re: [Discuss]- Donate Iceberg Flink Connector

[Márton Balassi]

Hi Abid,

Just to clarify does your suggestion mean that the Iceberg community would
like to remove the iceberg-flink connector from the Iceberg codebase and
maintain it under Flink instead? A new separate repo under the Flink
project umbrella given the current existing effort to extract connectors to
their individual repos (externalize) makes sense to me.

[1] https://lists.apache.org/thread/mpzzlpob9ymkjfybm96vz2y2m5fjyvfo

Best,
Marton


On Mon, Oct 10, 2022 at 5:31 AM Jingsong Li  wrote:

> Thanks Abid for driving.
>
> +1 for this.
>
> Can you open the permissions for
>
> https://docs.google.com/document/d/1WC8xkPiVdwtsKL2VSPAUgzm9EjrPs8ZRjEtcwv93ISI/edit?usp=sharing
> ?
>
> Best,
> Jingsong
>
> On Mon, Oct 10, 2022 at 9:22 AM Abid Mohammed
>  wrote:
> >
> > Hi,
> >
> > I would like to start a discussion about contributing Iceberg Flink
> Connector to Flink.
> >
> > I created a doc <
> https://docs.google.com/document/d/1WC8xkPiVdwtsKL2VSPAUgzm9EjrPs8ZRjEtcwv93ISI/edit?usp=sharing>
> with all the details following the Flink Connector template as I don’t have
> permissions to create a FLIP yet.
> > High level details are captured below:
> >
> > Motivation:
> >
> > This FLIP aims to contribute the existing Apache Iceberg Flink Connector
> to Flink.
> >
> > Apache Iceberg is an open table format for huge analytic datasets.
> Iceberg adds tables to compute engines including Spark, Trino, PrestoDB,
> Flink, Hive and Impala using a high-performance table format that works
> just like a SQL table.
> > Iceberg avoids unpleasant surprises. Schema evolution works and won’t
> inadvertently un-delete data. Users don’t need to know about partitioning
> to get fast queries. Iceberg was designed to solve correctness problems in
> eventually-consistent cloud object stores.
> >
> > Iceberg supports both Flink’s DataStream API and Table API. Based on the
> guideline of the Flink community, only the latest 2 minor versions are
> actively maintained. See the Multi-Engine Support#apache-flink for further
> details.
> >
> >
> > Iceberg connector supports:
> >
> > • Source: detailed Source design <
> https://docs.google.com/document/d/1q6xaBxUPFwYsW9aXWxYUh7die6O7rDeAPFQcTAMQ0GM/edit#>,
> based on FLIP-27
> > • Sink: detailed Sink design and interfaces used <
> https://docs.google.com/document/d/1O-dPaFct59wUWQECXEEYIkl9_MOoG3zTbC2V-fZRwrg/edit#
> >
> > • Usable in both DataStream and Table API/SQL
> > • DataStream read/append/overwrite
> > • SQL create/alter/drop table, select, insert into, insert
> overwrite
> > • Streaming or batch read in Java API
> > • Support for Flink’s Python API
> >
> > See Iceberg Flink  for
> detailed usage instructions.
> >
> > Looking forward to the discussion!
> >
> > Thanks
> > Abid
>

Re: [VOTE] Apache Flink Kubernetes Operator Release 1.2.0, release candidate #2

[Márton Balassi]

+1 (binding)

Verified upgrade process from 1.1 with a running application

On Thu, Oct 6, 2022 at 5:52 PM Biao Geng  wrote:

> +1(non-binding)
> Thanks a lot for the great work.
>
> Successfully verified the following:
> - Checksums and gpg signatures of the tar files.
> - No binaries in source release
> - Build from source, build image from source without errors
> - Helm Repo works, Helm install works
> - Run HA/python example in application mode
> - Check licenses in source code
>
> Best,
> Biao Geng
>
>
> Maximilian Michels  于2022年10月6日周四 20:16写道：
>
> > Turns out the issue with the Helm installation was that I was using
> > cert-manager 1.9.1 instead of the recommended version 1.8.2. The operator
> > now deploys cleanly in my local environment.
> >
> > On Thu, Oct 6, 2022 at 12:34 PM Maximilian Michels 
> wrote:
> >
> > > +1 (binding) because the source release looks good.
> > >
> > > I've verified the following:
> > >
> > > 1. Downloaded, compiled, and verified the signature of the source
> release
> > > staged at
> > >
> >
> https://dist.apache.org/repos/dist/dev/flink/flink-kubernetes-operator-1.2.0-rc2/
> > > 2. Verified licenses (Not a blocker: the LICENSE file does not contain
> a
> > > link to the bundled licenses directory, this should be fixed in future
> > > releases)
> > > 3. Verified the Helm Chart
> > >
> > > The Helm chart installation resulted in the following pod events:
> > >
> > > Events:
> > >   Type Reason   Age   From
>  Message
> > >    --     
>  ---
> > >   Normal   Scheduled5m42s default-scheduler
> > >  Successfully assigned
> default/flink-kubernetes-operator-54fcd9df98-645rf
> > > to docker-desktop
> > >   Warning  FailedMount  3m39s kubeletUnable
> > to
> > > attach or mount volumes: unmounted volumes=[keystore], unattached
> > > volumes=[kube-api-access-pdnzw keystore flink-operator-config-volume]:
> > > timed out waiting for the condition
> > >   Warning  FailedMount  92s (x10 over 5m42s)  kubelet
> > >  MountVolume.SetUp failed for volume "keystore" : secret
> > > "webhook-server-cert" not found
> > >   Warning  FailedMount  84s   kubeletUnable
> > to
> > > attach or mount volumes: unmounted volumes=[keystore], unattached
> > > volumes=[flink-operator-config-volume kube-api-access-pdnzw keystore]:
> > > timed out waiting for the condition
> > >
> > > Do we need to list any additional steps in the docs?
> > >
> >
> https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/try-flink-kubernetes-operator/quick-start/
> > >
> > > -Max
> > >
> > > On Wed, Oct 5, 2022 at 2:16 PM Őrhidi Mátyás 
> > > wrote:
> > >
> > >> +1 (non-binding)
> > >>
> > >> - Verified source distributions (except the licenses and maven
> > artifacts)
> > >> - Verified Helm chart and Docker image
> > >> - Verified basic examples
> > >>
> > >> Everything seems okay to me.
> > >>
> > >> Cheers,
> > >> Matyas
> > >>
> > >> On Tue, Oct 4, 2022 at 10:27 PM Gyula Fóra 
> > wrote:
> > >>
> > >> > +1 (binding)
> > >> >
> > >> > - Verified Helm repo works as expected, points to correct image tag,
> > >> build,
> > >> > version
> > >> > - Verified examples + checked operator logs everything looks as
> > expected
> > >> > - Verified hashes, signatures and source release contains no
> binaries
> > >> > - Ran built-in tests, built jars + docker image from source
> > successfully
> > >> >
> > >> > Cheers,
> > >> > Gyula
> > >> >
> > >> > On Sat, Oct 1, 2022 at 2:27 AM Jim Busche 
> wrote:
> > >> >
> > >> > > +1 (not-binding)
> > >> > >
> > >> > > Thank you Gyula,
> > >> > >
> > >> > >
> > >> > > Helm install from flink-kubernetes-operator-1.2.0-helm.tgz looks
> > good,
> > >> > > logs look normal
> > >> > >
> > >> > > podman Dockerfile build from source looks good.
> > >> > >
> > >> > > twistlock security scans of the proposed image look good:
> > >> > > ghcr.io/apache/flink-kubernetes-operator:95128bf
> > >> > >
> > >> > > UI and basic sample look good.
> > >> > >
> > >> > > Checksums looked good.
> > >> > >
> > >> > > Tested on OpenShift 4.10.25.  Will try additional versions (4.8
> and
> > >> 4.11)
> > >> > > if I get an opportunity, but I don't expect issues.
> > >> > >
> > >> > >
> > >> > >
> > >> > > Thank you,
> > >> > >
> > >> > > James Busche
> > >> > >
> > >> >
> > >>
> > >
> >
>

Re: [DISCUSS] FLIP-265 Deprecate and remove Scala API support

[Márton Balassi]

Hi Martjin,

Thanks for compiling the FLIP. I agree with the sentiment that Scala poses
considerable maintenance overhead and key improvements (like 2.13 or 2.12.8
supports) are hanging stale. With that said before we make this move we
should attempt to understand the userbase affected.
A quick Slack and user mailing list search does return quite a bit of
results for scala (admittedly a cursory look at them suggest that many of
them have to do with missing features in Scala that exist in Java or Scala
versions). I would love to see some polls on this topic, we could also use
the Flink twitter handle to ask the community about this.

I am aware of users having large existing Scala codebases for Flink. This
move would pose a very large effort on them, as they would need to rewrite
much of their existing code. What are the alternatives in your opinion,
Martjin?

On Tue, Oct 4, 2022 at 6:22 AM Martijn Visser 
wrote:

> Hi everyone,
>
> I would like to open a discussion thread on FLIP-265 Deprecate and remove
> Scala API support. Please take a look at
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support
> and provide your feedback.
>
> Best regards,
>
> Martijn
> https://twitter.com/MartijnVisser82
> https://github.com/MartijnVisser
>

Re: Kubernetes Operator 1.2.0 release

[Márton Balassi]

Thanks, +1 for the schedule.

On Tue, Sep 20, 2022 at 6:20 AM Yang Wang  wrote:

> Thanks Gyula for managing the release.
>
> +1 for the time schedule.
>
>
> Best,
> Yang
>
>
>
> Őrhidi Mátyás  于2022年9月19日周一 22:28写道：
>
> > Thanks Gyula!
> >
> > Sounds good! Happy to help as always.
> >
> > Cheers,
> > Matyas
> >
> > On Mon, Sep 19, 2022 at 1:37 PM Gyula Fóra  wrote:
> >
> > > Hi Devs!
> > >
> > > The originally planned (October 1) release date for 1.2.0 is fast
> > > approaching and we are already slightly behind schedule. There are a
> > couple
> > > outstanding bug tickets with 2 known blockers at the moment that should
> > be
> > > fixed in the next few days.
> > >
> > > As we are not aware of any larger critical issues or outstanding
> > features I
> > > propose the following adjusted release schedule:
> > >
> > >
> > > *Feature Freeze: September 23Release Branch Cut & RC1: September 28*
> > >
> > > Hopefully then we can finalize the release somewhere in the first week
> of
> > > October :)
> > >
> > > I volunteer as the release manager.
> > >
> > > Cheers,
> > > Gyula
> > >
> >
>

Re: [DISCUSS] Stale PR action for flink and flink-kubernetes-operator repos

[Márton Balassi]

Hi gents,

Thanks for the stats Martjin, that valuable insight into the situation.
Having a large number of open, stale PRs can also result in a bad
contributor experience down the line, as in my opinion it can reach a point
where it discourages committers reviewing them as it starts to feel like an
effort too large to tackle.

Would you be comfortable if we experimented with the approach G is
suggesting on the kubernetes operator repo to gather some experience and
shared it here?

Cheers,
Marton

On Wed, Sep 14, 2022 at 3:42 PM Chesnay Schepler  wrote:

> We've had discussions about closing stale PRs several times in the past
> and always rejected it.
>
> I see no reason to change this.
> If you want to close a PR, then do so while informing the contributor
> about the reason.
>
> On 14/09/2022 15:36, Martijn Visser wrote:
> > Hi Gabor,
> >
> > I have my doubts: I think the majority of the open PRs are not open
> because
> > of inactivity from the contributor, but I think the majority (at least
> for
> > the flink repository) are open because there are not enough reviews
> > occurring. If we actively mark those as stale and close them, I think it
> > makes a bad impression towards the contributor, since they can't
> influence
> > getting a review in.
> >
> > Some numbers for the Flink repo: there's now 957 open PRs, of which 839
> > haven't been reviewed. That's 88%. 91 PRs have been reviewed and
> > changes have been requested, which is almost 10%.
> >
> > Best regards,
> >
> > Martijn
> >
> > Op wo 14 sep. 2022 om 15:30 schreef Őrhidi Mátyás <
> matyas.orh...@gmail.com>:
> >
> >> Hi Gabor,
> >>
> >> Thanks for bringing this to our attention. I'd be happy to see such
> >> automatism guarding our repos. We could start a trial period on the
> >> operator repo I guess until we have the confidence it's a good thing.
> Are
> >> you aware of this plugin being used at other ASF projects? Any
> pros/cons?
> >>
> >> Cheers,
> >> Matyas
> >>
> >>
> >> On Wed, Sep 14, 2022 at 2:58 PM Gabor Somogyi <
> gabor.g.somo...@gmail.com>
> >> wrote:
> >>
> >>> Hi All,
> >>>
> >>> As I see there is no action for stale PRs for flink and
> >>> flink-kubernetes-operator repos however almost 1k PRs are open.
> >>>
> >>> I would like to suggest to add new stale PR action based on the
> following
> >>> github plugin:
> >>> https://github.com/marketplace/actions/close-stale-issues
> >>>
> >>> I think the default values for the plugin looks sufficient to our
> needs:
> >>> * Add a label "Stale" on pull requests after 60 days of inactivity and
> >>> comment on them
> >>> * Close the stale pull requests after 7 days of inactivity
> >>> * If an update/comment occur on stale issues or pull requests, the
> stale
> >>> label will be removed and the timer will restart
> >>>
> >>> A playground repo is created to test the feature here:
> >>> https://github.com/gaborgsomogyi/stale-test-repo
> >>>
> >>> Waiting on opinions/suggestions to make our PR queue manageable.
> >>>
> >>> Thanks in advance!
> >>>
> >>> BR,
> >>> G
> >>>
>
>

Re: [VOTE] FLIP-250: Support Customized Kubernetes Schedulers Proposal

[Márton Balassi]

+1 (binding)

On Wed, Jul 27, 2022 at 8:22 PM Gyula Fóra  wrote:

> +1 (binding)
>
> Gyula
>
> On Wed, 27 Jul 2022 at 03:52, MrL  wrote:
>
> > +1 (non-binding)
> >
> > > 2022年7月27日 09:18，William Wang  写道：
> > >
> > > +1 (non-binding)
> > >
> > > bo zhaobo  于2022年7月25日周一 09:38写道：
> > >
> > >> Hi all,
> > >>
> > >> Thank you very much for all feedback after the discussion in [2][3].
> > >> Now I'd like to proceed with the vote for FLIP-250 [1], as no more
> > >> objections
> > >> or issues were raised in ML thread [2][3].
> > >>
> > >> The vote will be opened until July 28th earliest(at least 72 hours)
> > unless
> > >> there is an objection or
> > >> insufficient votes.
> > >>
> > >> Thank you all.
> > >>
> > >> BR
> > >>
> > >> Bo Zhao
> > >>
> > >> [1]
> > >>
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-250%3A+Support+Customized+Kubernetes+Schedulers+Proposal
> > >> [2] https://lists.apache.org/thread/pf8dvbvqf845wh0x63z68jmhh4pvsbow
> > >> [3] https://lists.apache.org/thread/zbylkkc6jojrqwds7tt02k2t8nw62h26
> > >>
> >
> >
>

Re: [VOTE] Apache Flink Kubernetes Operator Release 1.1.0, release candidate #1

[Márton Balassi]

+1 (binding)

Successfully verified the following:
- Checksums and gpg signatures
- No binaries in source release
- Build from source, build image from source
- Helm Repo works, Helm install works
- Notice files look good
- Upgraded a cluster from 1.0.0 and run some examples

Specifically for FLINK-28637<
https://issues.apache.org/jira/browse/FLINK-28637>: Thank you for reporting
it, Jim. Fortunately both the Fabric8 and the JOSDK community was very
responsive, this gives a path for fixing this. However given the following:

1. The HTTP client is internal to the operator, this vulnerability is very
unlikely to affect it,
2. We also need to bump the dependency within the Flink native k8s
integration,
3. We need extensive testing to make sure the new dependency version
behaves properly,

My suggestion is to release 1.1.0 with this as a known issue and fix it in
1.1.1. That said we can merge a fix for it to the release-1.1 as soon as
possible, so folks who are prohibited to use the 1.1.0 version can roll
their own image from source.


On Thu, Jul 21, 2022 at 6:33 PM Gyula Fóra  wrote:

> Thank you for flagging this Jim. I looked a little into this and it comes
> from the fabric8 client, so it affects all current operator (and flink)
> versions.
>
> I think it would be a bit risky for us to manually bump this dependency as
> the usage is not controlled by us and it's hard to test for all the
> consequences of this major version change in the http client.
> Also it seems that this vulnerability would require direct user access to
> the http client, which is not the case here.
>
> At this point I think we should not consider this a blocker, I have also
> commented on the jira ticket.
>
> Gyula
>
> On Thu, Jul 21, 2022 at 6:27 PM Jim Busche  wrote:
>
> > Thanks for the release
> >
> > I’m continuing to test and so far it’s looking good, but I found a high
> > security vulnerability in the
> > /flink-kubernetes-operator/flink-kubernetes-operator-1.1.0-shaded.jar
> > file.  I’ve created issue FLINK-28637<
> > https://issues.apache.org/jira/browse/FLINK-28637> and seeing if I can
> > successfully upgrade to the newer okhttp version.
> >
> >
> >
> > Thanks, Jim
> >
>

Re: [VOTE] Creating benchmark channel in Apache Flink slack

[Márton Balassi]

+1 (binding). Thanks!

I can help you with the Slack admin steps if needed.

On Mon, Jul 11, 2022 at 10:55 AM godfrey he  wrote:

> +1, Thanks for driving this!
>
> Best,
> Godfrey
>
> Yuan Mei  于2022年7月11日周一 16:13写道：
> >
> > +1 (binding) & thanks for the efforts!
> >
> > Best
> > Yuan
> >
> >
> >
> > On Mon, Jul 11, 2022 at 2:08 PM Yun Gao 
> > wrote:
> >
> > > +1 (binding)
> > >
> > > Thanks Anton for driving this!
> > >
> > >
> > > Best,
> > > Yun Gao
> > >
> > >
> > > --
> > > From:Anton Kalashnikov 
> > > Send Time:2022 Jul. 8 (Fri.) 22:59
> > > To:undefined 
> > > Subject:[VOTE] Creating benchmark channel in Apache Flink slack
> > >
> > > Hi everyone,
> > >
> > >
> > > I would like to start a vote for creating the new channel in Apache
> > > Flink slack for sending benchamrk's result to it. This should help the
> > > community to notice the performance degradation on time.
> > >
> > > The discussion of this idea can be found here[1]. The ticket for this
> is
> > > here[2].
> > >
> > >
> > > [1] https://www.mail-archive.com/dev@flink.apache.org/msg58666.html
> > >
> > > [2] https://issues.apache.org/jira/browse/FLINK-28468
> > >
> > > --
> > >
> > > Best regards,
> > > Anton Kalashnikov
>

Re: [VOTE] Apache Flink Kubernetes Operator Release 1.0.1, release candidate #1

[Márton Balassi]

+1 (binding)

Successfully verified the following:
- Verify that the checksums and GPG files
- Verify that the source distributions do not contain any binaries
- Build binary and image from release source
- Verify the NOTICE and licenses in source release and the docker image
- Operator functionality manual testing
- Start a session FlinkDeployment and submit a SessionJob CR
- Verify the FlinkUI could be accessed via ingress
- No strange operator logs

Best,
Marton

On Mon, Jun 27, 2022 at 4:44 AM Yang Wang  wrote:

> +1 (binding)
>
> Successfully verified the following:
> - Verify that the checksums and GPG files
> - Verify that the source distributions do not contain any binaries
> - Build binary and image from release source
> - Verify the NOTICE and licenses in source release and the docker image
> - Verify the helm chart values with correct appVersion and image tag
> - The image tag is *b93417f1*, not *b93417f*. But it is trivial because
> they have the same SHA.
> - Operator functionality manual testing
> - Start a session FlinkDeployment and submit a SessionJob CR
> - Start a Flink Application job(both streaming and batch) with 1.15
> - Verify the FlinkUI could be accessed via ingress
> - No strange operator logs
>
>
> Best,
> Yang
>
> Chenya Zhang  于2022年6月24日周五 22:31写道：
>
> > +1 (non-binding)
> >
> > - Verified maven and helm chart versions for built from source
> > - Verified helm chart points to correct docker image and deploys it by
> > default
> > - Verified helm installation and basic checkpointing, stateful examples
> > with upgrades and manual savepoints
> > - Verified online documents including Quick Start etc.
> >
> > Chenya
> >
> > On Fri, Jun 24, 2022 at 4:26 AM Őrhidi Mátyás 
> > wrote:
> >
> > > +1
> > > - Verified hashes/signatures
> > > - Verified Helm chart, helm install
> > > - Ran a few example jobs
> > >
> > > Matyas
> > >
> > >
> > > On Fri, Jun 24, 2022 at 11:30 AM Gyula Fóra  wrote:
> > >
> > > > +1 (binding)
> > > >
> > > >  - Verified hashes/signatures, license headers for the release
> > artifacts
> > > >  - Verified Helm chart, helm install
> > > >  - Ran a few example jobs
> > > >  - Verified release notes
> > > >
> > > > Gyul
> > > >
> > > >
> > > > On Thu, Jun 23, 2022 at 8:34 AM Gyula Fóra 
> wrote:
> > > >
> > > > > Hi Devs,
> > > > >
> > > > > Please review and vote on the release candidate #1 for the version
> > > 1.0.1
> > > > of
> > > > > Apache Flink Kubernetes Operator,
> > > > > as follows:
> > > > > [ ] +1, Approve the release
> > > > > [ ] -1, Do not approve the release (please provide specific
> comments)
> > > > >
> > > > > **Release Overview**
> > > > >
> > > > > As an overview, the release consists of the following:
> > > > > a) Kubernetes Operator canonical source distribution (including the
> > > > > Dockerfile), to be deployed to the release repository at
> > > dist.apache.org
> > > > > b) Kubernetes Operator Helm Chart to be deployed to the release
> > > > repository
> > > > > at dist.apache.org
> > > > > c) Maven artifacts to be deployed to the Maven Central Repository
> > > > > d) Docker image to be pushed to dockerhub
> > > > >
> > > > > **Staging Areas to Review**
> > > > >
> > > > > The staging areas containing the above mentioned artifacts are as
> > > > follows,
> > > > > for your review:
> > > > > * All artifacts for a,b) can be found in the corresponding dev
> > > repository
> > > > > at dist.apache.org [1]
> > > > > * All artifacts for c) can be found at the Apache Nexus Repository
> > [2]
> > > > > * The docker image for d) is staged on github [7]
> > > > >
> > > > > All artifacts are signed with the key
> > > > > 0B4A34ADDFFA2BB54EB720B221F06303B87DAFF1[3]
> > > > >
> > > > > Other links for your review:
> > > > > * JIRA release notes [4]
> > > > > * source code tag "release-1.0.1-rc1" [5]
> > > > > * PR to update the website Downloads page to include Kubernetes
> > > Operator
> > > > > links [6]
> > > > >
> > > > > **Vote Duration**
> > > > >
> > > > > The voting time will run for at least 72 hours.
> > > > > It is adopted by majority approval, with at least 3 PMC affirmative
> > > > votes.
> > > > >
> > > > > **Note on Verification**
> > > > >
> > > > > You can follow the basic verification guide here[8].
> > > > > Note that you don't need to verify everything yourself, but please
> > make
> > > > > note of what you have tested together with your +- vote.
> > > > >
> > > > > Thanks,
> > > > > Gyula
> > > > >
> > > > > [1]
> > > > >
> > > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/flink/flink-kubernetes-operator-1.0.1-rc1/
> > > > > [2]
> > > > >
> > >
> https://repository.apache.org/content/repositories/orgapacheflink-1512/
> > > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > >
> > > > > [4]
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351812
> > > > > <
> > > >
> > >
> >
> https://issues.apache.org/jira/

Re: [DISCUSS] Release Kubernetes operator 1.0.1

[Márton Balassi]

Hi team,

+1 for having a 1.0.1 for the Kubernetes Operator.

On Wed, Jun 22, 2022 at 4:23 PM Gyula Fóra  wrote:

> Hi Devs!
>
> How do you feel about releasing the 1.0.1 patch release for the Kubernetes
> operator?
>
> We have fixed a few annoying issues that many people tend to hit.
>
> Given that we are about halfway until the next minor release based on the
> proposed schedule I think we could prepare a 1.0.1 RC1 in the next 1-2 days
> .
>
> I can volunteer to be the release manager.
>
> What do you think?
>
> Cheers,
> Gyula
>

Re: [DISCUSS] Flink Kubernetes Operator release cadence proposal

[Márton Balassi]

Thanks for the proposal, Matyas.

+1 for 2 month release cycles with the breakdown Gyula suggested.

@Yang: we could start tagging features with 1.1 / 1.2 version then, good
call.

On Wed, Jun 22, 2022 at 4:58 AM Yang Wang  wrote:

> +1 for 2 month release cycles.
>
> Since we have promised the backward compatibility for CRD, I think it is
> also reasonable for us to maintain the latest two minor versions with patch
> releases.
>
> Given that we only have 5~6 weeks for feature development, maybe we need to
> confirm down the features as soon as possible in the release cycle.
> Otherwise, we are at great risk of delay. If we are targeting the 1.1.0
> release to Aug 1. It is the time to determine which features we want to
> include in this release.
>
> I agree with Gyula that we need to continuously improve the test coverage,
> especially the e2e tests. The users are very welcome to share their
> production use cases
> and we could consider whether they could be covered by e2e tests. Benefit
> from this, we could ship a stable release more quickly and easily.
>
>
>
> Best,
> Yang
>
> Gyula Fóra  于2022年6月21日周二 19:57写道：
>
> > Hi Matyas!
> >
> > Thanks for starting the discussion. I think the 2 month release cycle
> > sounds reasonable.
> >
> > I think it's important for users to have frequent operator releases as
> > these affect all Flink jobs running in the environment. We should also
> only
> > adopt a schedule that we can most likely keep.
> > If we want to be successful with the proposed schedule we have to ensure
> > that each release has a relatively small scope of new features and we
> have
> > good test coverage.
> >
> > In addition to your suggestion I would like to add a feature-freeze for
> > bigger new features after 5 weeks (1 week before cutting the release
> > branch).
> >
> > So in practice for 1.1.0 this would look like:
> >
> > - Jun 6 : 1.0.0 was released
> > - July 11: Feature Freeze
> > - July 18: Cut release-1.1 branch
> > - Aug 1: Target 1.1.0 release date
> >
> > Cheers,
> > Gyula
> >
> > On Tue, Jun 21, 2022 at 9:04 AM Őrhidi Mátyás 
> > wrote:
> >
> > > Hi Devs,
> > >
> > > After the successful Kubernetes Operator 1.0.0 release, which is
> > > considered to be the first production grade one, it is probably a good
> > time
> > > now to also agree on a predictable release cadence for the Operator
> too,
> > > similarly to the time-based release plan we have for the Flink core
> > project.
> > >
> > > Given that the Operator itself is not strictly bound to Flink versions
> it
> > > can be upgraded independently from the runtime versions it manages. It
> > > would benefit the community to have frequent minor releases until the
> > > majority of the roadmap items are complete that also encourages users
> to
> > > upgrade regularly in reasonable boundaries. Based on some offline
> > > discussion with Gyula Fora I would like to propose the following
> > > operational model for Operator releases:
> > >
> > > - time-based release cadence with 2 month release cycles ( This would
> > give
> > > us roughly 6 weeks pure dev time and leave 2 weeks for the release
> > process
> > > to finish)
> > > - on-demand patch releases for critical issues only
> > > - support the current and previous minor releases with bug fixes
> > >
> > > I am looking forward to your feedback and suggestions on this topic.
> Once
> > > we have an agreement I will formalize it on a Wiki page.
> > >
> > > Thanks,
> > > Matyas
> > >
> >
>

Re: [RESULT] [VOTE] Apache Flink Kubernetes Operator Release 1.0.0, release candidate #4

[Márton Balassi]

Well done, team. :-) Thanks for managing the release, Yang.

On Fri 3. Jun 2022 at 17:58, Yang Wang  wrote:

> I'm happy to announce that we have unanimously approved this release.
>
> There are 5 approving votes, 3 of which are binding:
>
> * Marton Balassi (binding)
>
> * Gyula Fora (binding)
>
> * Biao Geng (non-binding)
>
> * Jim Busche (non-binding)
>
> * Yang Wang (binding)
>
>
> There are no disapproving votes.
>
> Thank you all for verifying the release candidate. I will proceed to
> finalize
>
> the release on the weekend and announce it once everything is published.
>
>
>
> Best,
>
> Yang
>

Re: [VOTE] Apache Flink Kubernetes Operator Release 1.0.0, release candidate #4

[Márton Balassi]

Hi team,

+1 (binding)

Verified the following:

- NOTICE file looks good :-)
- Signatures, Hashes
- No binaries in source release
- Helm Repo works, Helm install works, docker image matches release commit
tag
- Build from source, build container from source
- Submit example application, session and session job deployments without
errors

Best,
Marton

On Wed, Jun 1, 2022 at 1:31 PM Gyula Fóra  wrote:

> Hi Yang!
>
> +1 (binding)
>
> Thank you for fixing the NOTICE file issue
>
> Successfully verified the following:
> - Signatures, Hashes
> - No binaries in source release
> - Helm Repo works, Helm install works, docker image matches release commit
> tag
> - Build from source, build container from source
> - Submit example application, session and session job deployments without
> errors
> - Reviewed website PR
>
> Cheers,
> Gyula
>
> Thank you!
> Gyula
>
> On Wed, Jun 1, 2022 at 12:16 PM Yang Wang  wrote:
>
> > Hi everyone,
> >
> > Please review and vote on the release candidate #4 for the version 1.0.0
> of
> > Apache Flink Kubernetes Operator,
> > as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> > **Release Overview**
> >
> > As an overview, the release consists of the following:
> > a) Kubernetes Operator canonical source distribution (including the
> > Dockerfile), to be deployed to the release repository at dist.apache.org
> > b) Kubernetes Operator Helm Chart to be deployed to the release
> repository
> > at dist.apache.org
> > c) Maven artifacts to be deployed to the Maven Central Repository
> > d) Docker image to be pushed to dockerhub
> >
> > **Staging Areas to Review**
> >
> > The staging areas containing the above mentioned artifacts are as
> follows,
> > for your review:
> > * All artifacts for a,b) can be found in the corresponding dev repository
> > at dist.apache.org [1]
> > * All artifacts for c) can be found at the Apache Nexus Repository [2]
> > * The docker image for d) is staged on github [7]
> >
> > All artifacts are signed with the key
> > 2FF2977BBBFFDF283C6FE7C6A301006F3591EE2C [3]
> >
> > Other links for your review:
> > * JIRA release notes [4]
> > * source code tag "release-1.0.0-rc4" [5]
> > * PR to update the website Downloads page to include Kubernetes Operator
> > links [6]
> >
> > **Vote Duration**
> >
> > Since there's no functional changes from release candidate #3, the voting
> > time will run for 48 hours.
> > It is adopted by majority approval, with at least 3 PMC affirmative
> votes.
> >
> > **Note on Verification**
> >
> > You can follow the basic verification guide here[8].
> > Note that you don't need to verify everything yourself, but please make
> > note of what you have tested together with your +- vote.
> >
> > Thanks,
> > Yang
> >
> > [1]
> >
> >
> https://dist.apache.org/repos/dist/dev/flink/flink-kubernetes-operator-1.0.0-rc4/
> > [2]
> > https://repository.apache.org/content/repositories/orgapacheflink-1506/
> > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > [4]
> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351500
> > [5]
> >
> https://github.com/apache/flink-kubernetes-operator/tree/release-1.0.0-rc4
> > [6] https://github.com/apache/flink-web/pull/542
> > [7] ghcr.io/apache/flink-kubernetes-operator:fa2cd14
> > [8]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Verifying+a+Flink+Kubernetes+Operator+Release
> >
>

Re: [VOTE] Apache Flink Kubernetes Operator Release 1.0.0, release candidate #3

[Márton Balassi]

Thanks, Yang. Looks good after your change. +1 for the 48 hour voting time.


On Wed, Jun 1, 2022 at 7:59 AM Gyula Fóra  wrote:

> I agree to create a new RC based on this find.
> And 48h voting time seems reasonable as all functional testing will carry
> over to this new candidate.
>
> Gyula
>
> On Wed, Jun 1, 2022 at 7:56 AM Yang Wang  wrote:
>
>> Thanks all for your testing and patience.
>>
>> And sorry for I have to cancel this VOTE since @Márton Balassi
>>  found a license issue. We do not list the
>> CSS/docs dependencies in the NOTICE file of source release[1].
>>
>> I will create another release candidate today including this fix. Given
>> that there's no functional changes, we could set the voting period to 48
>> hours for next RC.
>>
>>
>> [1].
>>
>> https://dist.apache.org/repos/dist/dev/flink/flink-kubernetes-operator-1.0.0-rc3/
>>
>> Best,
>> Yang
>>
>> Jim Busche  于2022年6月1日周三 08:30写道：
>>
>> > Hi Yang,  Thank you for RC3
>> >
>> > +1 (not-binding)
>> >
>> >
>> >   *   Podman builds look great with your .dockerignore file and “COPY .
>> > .”  Thank you for the fix.
>> >   *   Twistlock security scans look clean of both your “
>> > ghcr.io/apache/flink-kubernetes-operator:52b50c2 image” as well as a
>> > locally built image from source.
>> >   *   I’ve tried it on both Openshift 4.8 and 4.10 and the basic test
>> and
>> > UI look good, using helm repo install from the provided helm repo as
>> well
>> > as from source with the local image.
>> >   *   Logs look normal
>> >
>> >
>> >
>> > Thank you,
>> >
>> > Jim
>> >
>> >
>> >
>>
>

Re: [DISCUSS] Bi-Weekly Flink Community Sync Meeting

[Márton Balassi]

Hi Robert,

Thanks for the suggestion +1 from me. You already listed the topic of the
timezone on the wiki that I wanted to bring up.

On Tue, May 31, 2022 at 9:38 AM Robert Metzger  wrote:

> Hi everyone,
>
> We currently have a bi-weekly release sync meeting on Google Meet every
> Tuesday at 9am CEST / 3pm China Standard Time / 7am UTC.
> I would like to propose extending the purpose of the meeting to a general
> "Flink Community Sync" meeting, to discuss current topics in the Flink
> community.
>
> I propose that we just collect agenda items on the mailing list in the two
> weeks prior to the meeting.
> I'm happy to take care of prioritizing agenda items and taking notes in the
> wiki.
> I've created already a page for the next sync:
> https://cwiki.apache.org/confluence/display/FLINK/2022-06-14+Community+Sync
>
> Let me know what you think!
>
> Best,
> Robert
>

Re: [ANNOUNCE] Kubernetes Operator release-1.0 branch cut

[Márton Balassi]

Thanks Gyula and Yang. Awesome!

On Tue, May 17, 2022 at 4:46 PM Gyula Fóra  wrote:

> Hi Flink devs!
>
> The release-1.0 branch has been forked from main and version numbers have
> been upgraded accordingly.
>
> https://github.com/apache/flink-kubernetes-operator/tree/release-1.0
>
> The version on the main branch has been updated to 1.1-SNAPSHOT.
>
> From now on, for PRs that should be presented in 1.0.0, please make sure:
> - Merge the PRs first to main, then backport to release-1.0 branch
> - The JIRA ticket should be closed with the correct fix-versions
>
> There are still a few outstanding tickets, mainly docs/minor fixes but no
> currently known blocker issues.
>
> We are working together with Yang to prepare the first RC by next monday.
>
> Cheers,
> Gyula
>

Re: [VOTE] Creating an Apache Flink slack workspace

[Márton Balassi]

+1 (binding)

On Tue, May 17, 2022 at 11:00 AM Jingsong Li  wrote:

> Thank Xintong for driving this work.
>
> +1
>
> Best,
> Jingsong
>
> On Tue, May 17, 2022 at 4:49 PM Martijn Visser 
> wrote:
>
> > +1 (binding)
> >
> > On Tue, 17 May 2022 at 10:38, Yu Li  wrote:
> >
> > > +1 (binding)
> > >
> > > Thanks Xintong for driving this!
> > >
> > > Best Regards,
> > > Yu
> > >
> > >
> > > On Tue, 17 May 2022 at 16:32, Robert Metzger 
> > wrote:
> > >
> > > > Thanks for starting the VOTE!
> > > >
> > > > +1 (binding)
> > > >
> > > >
> > > >
> > > > On Tue, May 17, 2022 at 10:29 AM Jark Wu  wrote:
> > > >
> > > > > Thank Xintong for driving this work.
> > > > >
> > > > > +1 from my side (binding)
> > > > >
> > > > > Best,
> > > > > Jark
> > > > >
> > > > > On Tue, 17 May 2022 at 16:24, Xintong Song 
> > > > wrote:
> > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > As previously discussed in [1], I would like to open a vote on
> > > creating
> > > > > an
> > > > > > Apache Flink slack workspace channel.
> > > > > >
> > > > > > The proposed actions include:
> > > > > > - Creating a dedicated slack workspace with the name Apache Flink
> > > that
> > > > is
> > > > > > controlled and maintained by the Apache Flink PMC
> > > > > > - Updating the Flink website about rules for using various
> > > > communication
> > > > > > channels
> > > > > > - Setting up an Archive for the Apache Flink slack
> > > > > > - Revisiting this initiative by the end of 2022
> > > > > >
> > > > > > The vote will last for at least 72 hours, and will be accepted
> by a
> > > > > > consensus of active PMC members.
> > > > > >
> > > > > > Best,
> > > > > >
> > > > > > Xintong
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Re: [ANNOUNCE] New Flink PMC member: Yang Wang

[Márton Balassi]

Congrats, Yang. Well deserved :-)

On Tue, May 10, 2022 at 9:16 AM Terry Wang  wrote:

> Congrats Yang!
>
> On Mon, May 9, 2022 at 11:19 AM LuNing Wang  wrote:
>
> > Congrats Yang!
> >
> > Best,
> > LuNing Wang
> >
> > Dian Fu  于2022年5月7日周六 17:21写道：
> >
> > > Congrats Yang!
> > >
> > > Regards,
> > > Dian
> > >
> > > On Sat, May 7, 2022 at 12:51 PM Jacky Lau 
> wrote:
> > >
> > > > Congrats Yang and well Deserved!
> > > >
> > > > Best,
> > > > Jacky Lau
> > > >
> > > > Yun Gao  于2022年5月7日周六 10:44写道：
> > > >
> > > > > Congratulations Yang!
> > > > >
> > > > > Best,
> > > > > Yun Gao
> > > > >
> > > > >
> > > > >
> > > > >  --Original Mail --
> > > > > Sender:David Morávek 
> > > > > Send Date:Sat May 7 01:05:41 2022
> > > > > Recipients:Dev 
> > > > > Subject:Re: [ANNOUNCE] New Flink PMC member: Yang Wang
> > > > > Nice! Congrats Yang, well deserved! ;)
> > > > >
> > > > > On Fri 6. 5. 2022 at 17:53, Peter Huang <
> huangzhenqiu0...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Congrats, Yang!
> > > > > >
> > > > > >
> > > > > >
> > > > > > Best Regards
> > > > > > Peter Huang
> > > > > >
> > > > > > On Fri, May 6, 2022 at 8:46 AM Yu Li  wrote:
> > > > > >
> > > > > > > Congrats and welcome, Yang!
> > > > > > >
> > > > > > > Best Regards,
> > > > > > > Yu
> > > > > > >
> > > > > > >
> > > > > > > On Fri, 6 May 2022 at 14:48, Paul Lam 
> > > wrote:
> > > > > > >
> > > > > > > > Congrats, Yang! Well Deserved!
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Paul Lam
> > > > > > > >
> > > > > > > > > 2022年5月6日 14:38，Yun Tang  写道：
> > > > > > > > >
> > > > > > > > > Congratulations, Yang!
> > > > > > > > >
> > > > > > > > > Best
> > > > > > > > > Yun Tang
> > > > > > > > > 
> > > > > > > > > From: Jing Ge 
> > > > > > > > > Sent: Friday, May 6, 2022 14:24
> > > > > > > > > To: dev 
> > > > > > > > > Subject: Re: [ANNOUNCE] New Flink PMC member: Yang Wang
> > > > > > > > >
> > > > > > > > > Congrats Yang and well Deserved!
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Jing
> > > > > > > > >
> > > > > > > > > On Fri, May 6, 2022 at 7:38 AM Lincoln Lee <
> > > > lincoln.8...@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Congratulations Yang!
> > > > > > > > >>
> > > > > > > > >> Best,
> > > > > > > > >> Lincoln Lee
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> Őrhidi Mátyás  于2022年5月6日周五
> > 12:46写道：
> > > > > > > > >>
> > > > > > > > >>> Congrats Yang! Well deserved!
> > > > > > > > >>> Best,
> > > > > > > > >>> Matyas
> > > > > > > > >>>
> > > > > > > > >>> On Fri, May 6, 2022 at 5:30 AM huweihua <
> > > > huweihua@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >>>
> > > > > > > >  Congratulations Yang!
> > > > > > > > 
> > > > > > > >  Best,
> > > > > > > >  Weihua
> > > > > > > > 
> > > > > > > > 
> > > > > > > > >>>
> > > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
> --
> Best Regards,
> Terry Wang
>

Re: [DISCUSS] DockerHub repository maintainers

[Márton Balassi]

Hi team,

I volunteer for the flink-kubernetes-operator repo.

On Fri, May 6, 2022 at 1:42 PM Xintong Song  wrote:

> @Till,
>
> Thanks for volunteering.
>
> @Konstantin,
>
> From my experience, the effort that requires DockerHub access in the main
> project release process is quite limited. I helped Yun Gao on releasing the
> 1.15.0 images, and what I did was just check out the `flink-docker` repo
> and run the release script, that's it. If all the sub-projects are as easy
> as the main project, then it's probably ok that only a small group of
> people have access. Concerning the redundancy, if a maintainer from a
> sub-project is temporarily unreachable, I believe the other maintainers
> would be glad to help.
>
> It would of course be good to have more seats. I just haven't come up with
> good reasons to persuade the INFRA folks. What's your suggestions?
>
>
> Thank you~
>
> Xintong Song
>
>
>
> On Fri, May 6, 2022 at 6:38 PM Konstantin Knauf  wrote:
>
> > Hi Xintong,
> >
> > it is a pity that we can only have 5 maintainers. Every (patch) release
> of
> > flink, flink-statefun, the flink-kubernetes-operator requires a
> maintainer
> > to publish the image then, if I am not mistaken. As its mostly different
> > groups managing the sub-projects, this is quite the bottleneck. If we
> give
> > one seat to flink-statefun maintainers, one to the
> > flink-kubernetes-operator maintainers, this leaves three seats for Apache
> > Flink core, and there is no redundancy for the other projects. When I
> > managed the last two patch releases, the DockerHub access was also the
> > biggest hurdle. Maybe we can talk to the INFRA people again. We can
> > certainly reduce it, but 5 is very little.
> >
> > Cheers,
> >
> > Konstantin
> >
> >
> >
> >
> >
> >
> > Am Fr., 6. Mai 2022 um 09:00 Uhr schrieb Till Rohrmann <
> > trohrm...@apache.org
> > >:
> >
> > > Hi everyone,
> > >
> > > thanks for starting this discussion Xintong. I would volunteer as a
> > > maintainer of the flink-statefun Docker repository if you need one.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Fri, May 6, 2022 at 6:22 AM Xintong Song 
> > wrote:
> > >
> > >> It seems to me we at least don't have a consensus on dropping the use
> of
> > >> apache namespace, which means we need to decide on a list of
> maintainers
> > >> anyway. So maybe we can get the discussion back to the maintainers. We
> > may
> > >> continue the official-image vs. apache-namespace in a separate thread
> if
> > >> necessary.
> > >>
> > >> As mentioned previously, we need to reduce the number of maintainers
> > from
> > >> 20 to 5, as required by INFRA. Jingsong and I would like to volunteer
> > as 2
> > >> of the 5, and we would like to learn who else wants to join us. Of
> > course
> > >> the list of maintainers can be modified later.
> > >>
> > >> *This also means the current maintainers may be removed from the
> list.*
> > >> Please let us know if you still need that privilege. CC-ed all the
> > current
> > >> maintainers for attention.
> > >>
> > >> Thank you~
> > >>
> > >> Xintong Song
> > >>
> > >>
> > >>
> > >> On Wed, May 4, 2022 at 3:14 PM Chesnay Schepler 
> > >> wrote:
> > >>
> > >> > One advantage is that the images are periodically rebuilt to get
> > >> > security fixes.
> > >> >
> > >> > The operator is a different story anyway because it is AFAIK only
> > >> > supposed to be used via docker
> > >> > (i.e., no standalone mode), which alleviates concerns about keeping
> > the
> > >> > logic within the image
> > >> > to a minimum (which bit us in the past on the flink side).
> > >> >
> > >> > On 03/05/2022 16:09, Yang Wang wrote:
> > >> > > The flink-kubernetes-operator project is only published
> > >> > > via apache/flink-kubernetes-operator on docker hub and github
> > >> packages.
> > >> > > We do not find the obvious advantages by using docker hub official
> > >> > images.
> > >> > >
> > >> > > Best,
> > >> > > Yang
> > >> > >
> > >> > > Xintong Song  于2022年4月28日周四 19:27写道：
> > >> > >
> > >> > >> I agree with you that doing QA for the image after the release
> has
> > >> been
> > >> > >> finalized doesn't feel right. IIUR, that is mostly because
> official
> > >> > image
> > >> > >> PR needs 1) the binary release being deployed and propagated and
> 2)
> > >> the
> > >> > >> corresponding git commit being specified. I'm not completely sure
> > >> about
> > >> > >> this. Maybe we can improve the process by investigating more
> about
> > >> the
> > >> > >> feasibility of pre-verifying an official image PR before
> finalizing
> > >> the
> > >> > >> release. It's definitely a good thing to do if possible.
> > >> > >>
> > >> > >> I also agree that QA from DockerHub folks is valuable to us.
> > >> > >>
> > >> > >> I'm not against publishing official-images, and I'm not against
> > >> working
> > >> > >> closely with the DockerHub folks to improve the process of
> > delivering
> > >> > the
> > >> > >> official image. However, I don't think these should become
> reasons
> >

Re: [RESULT] [VOTE] Apache Flink Kubernetes Operator Release 0.1.0, release candidate #3

[Márton Balassi]

Thank you team, it was a pleasure to witness the community investing into
this much requested topic and coming to an initial release so swiftly.

On Sat, Apr 2, 2022 at 4:45 PM Gyula Fóra  wrote:

> I'm happy to announce that we have unanimously approved this release.
>
> There are 8 approving votes, 4 of which are binding:
> * Marton Balassi (binding)
> * Yang Wang (non-binding)
> * Gyula Fora (binding)
> * Biao Geng (non-binding)
> * Thomas Weise (binding)
> * Xintong Song (binding)
> * Aitozi (non-binding)
> * Nicholas Jiang (non-binding)
>
> There are no disapproving votes.
>
> Thank you all for verifying the release candidate. I will now proceed to
> finalize the release and announce it once everything is published.
>
> Cheers,
> Gyula
>

Re: [VOTE] Apache Flink Kubernetes Operator Release 0.1.0, release candidate #3

[Márton Balassi]

+1 (binding)

Verified the following:

   - shasums
   - gpg signatures
   - source does not contain any binaries
   - built from source
   - deployed via helm after adding the distribution webserver endpoint as
   a helm registry
   - all relevant files have license headers


On Wed, Mar 30, 2022 at 4:39 PM Gyula Fóra  wrote:

> Hi everyone,
>
> Please review and vote on the release candidate #3 for the version 0.1.0 of
> Apache Flink Kubernetes Operator,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
> **Release Overview**
>
> As an overview, the release consists of the following:
> a) Kubernetes Operator canonical source distribution (including the
> Dockerfile), to be deployed to the release repository at dist.apache.org
> b) Kubernetes Operator Helm Chart to be deployed to the release repository
> at dist.apache.org
> c) Maven artifacts to be deployed to the Maven Central Repository
> d) Docker image to be pushed to dockerhub
>
> **Staging Areas to Review**
>
> The staging areas containing the above mentioned artifacts are as follows,
> for your review:
> * All artifacts for a,b) can be found in the corresponding dev repository
> at dist.apache.org [1]
> * All artifacts for c) can be found at the Apache Nexus Repository [2]
> * The docker image is staged on github [7]
>
> All artifacts are signed with the key
> 0B4A34ADDFFA2BB54EB720B221F06303B87DAFF1 [3]
>
> Other links for your review:
> * JIRA release notes [4]
> * source code tag "release-0.1.0-rc3" [5]
> * PR to update the website Downloads page to include Kubernetes Operator
> links [6]
>
> **Vote Duration**
>
> The voting time will run for at least 72 hours.
> It is adopted by majority approval, with at least 3 PMC affirmative votes.
>
> **Note on Verification**
>
> You can follow the basic verification guide here
> <
> https://cwiki.apache.org/confluence/display/FLINK/Verifying+a+Flink+Kubernetes+Operator+Release
> >
> .
> Note that you don't need to verify everything yourself, but please make
> note of what you have tested together with your +- vote.
>
> Thanks,
> Gyula
>
> [1]
>
> https://dist.apache.org/repos/dist/dev/flink/flink-kubernetes-operator-0.1.0-rc3/
> [2]
> https://repository.apache.org/content/repositories/orgapacheflink-1492/
> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> [4]
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12351499
> [5]
> https://github.com/apache/flink-kubernetes-operator/tree/release-0.1.0-rc3
> [6] https://github.com/apache/flink-web/pull/519
> [7] ghcr.io/apache/flink-kubernetes-operator:2c166e3
>

Re: [VOTE] Apache Flink Kubernetes Operator Release 0.1.0, release candidate #2

[Márton Balassi]

Hi team,

I would like to ask for your input in naming the operator docker image and
helm chart. [1]

For the sake of brevity when we started the Kubernetes Operator work we
named the docker image and the helm chart simply flink-operator, while the
git repository is named flink-kubernetes-operator. [2]
Now closing in on our preview release it makes sense to reconsider this, it
might be preferred to name all artifacts flink-kubernetes-operator for the
sake of consistency.
Currently docker images of our builds are available in the GitHub Registry
tagged with the short git commit hash and the last build of select branches
is tagged with the branch name:

ghcr.io/apache/flink-operator:439bd41ghcr.io/apache/flink-operator:main

During the release process we plan to move the docker image to dockerhub
following the process established for Flink.
Currently the helm operator for the release candidate can be installed as
follows:

helm repo add operator-rc2
https://dist.apache.org/repos/dist/dev/flink/flink-kubernetes-operator-0.1.0-rc2
helm install flink-operator operator-rc2/flink-operator

So the helm chart itself is called flink-operator, but to follow the name
of the project it is packaged into flink-kubernetes-operator-.tgz.
Do you prefer flink-operator for brevity or flink-kubernetes-operator for
consistency? If we vote for changing the current setup we could get the
changes in for the next RC.

[1] https://issues.apache.org/jira/browse/FLINK-26924
[2] https://github.com/apache/flink-kubernetes-operator
[3]
https://cwiki.apache.org/confluence/display/FLINK/Creating+a+Flink+Release#CreatingaFlinkRelease-PublishtheDockerfilesforthenewrelease

On Wed, Mar 30, 2022 at 9:17 AM Gyula Fóra  wrote:

> Hi Devs,
>
> I am cancelling the current vote as we have promoted
> https://issues.apache.org/jira/browse/FLINK-26916 as a blocker issue.
>
> Yang has found a fix that we can reasonably do in a short time frame, so we
> will work on that today before creating the next RC (hopefully later today)
>
> Thank you!
> Gyula
>
> On Tue, Mar 29, 2022 at 7:51 PM Gyula Fóra  wrote:
>
> > Hi Devs,
> >
> > I just want to give you heads up regarding a significant issue Matyas
> > found around the last-state upgrade mode:
> > https://issues.apache.org/jira/browse/FLINK-26916
> >
> > Long story short, the last-state mode does not actually upgrade the job,
> > only the job/taskmanagers. The fix is non-trivial to say the least and
> > requires changes to the Flink Kubernetes HA implementations to be
> feasible.
> > Please see the jira ticket for details.
> >
> > I discussed this offline with Thomas Weise and we agreed to clearly
> > highlight this limitation with warnings in the documentation and the
> > release blogpost but not block the release on it as the fix will most
> > likely require Flink 1.15 if we can include the required changes there.
> >
> > Please continue testing the current RC :)
> >
> > Thank you all!
> > Gyula
> >
> >
> > On Tue, Mar 29, 2022 at 5:17 PM Chenya Zhang  wrote:
> >
> >> +1 non-binding, thanks for the great efforts and look forward to the
> >> release!
> >>
> >> Chenya
> >>
> >> On Tue, Mar 29, 2022 at 2:10 AM Gyula Fóra  wrote:
> >>
> >> > Hi everyone,
> >> >
> >> > Please review and vote on the release candidate #2 for the version
> >> 0.1.0 of
> >> > Apache Flink Kubernetes Operator,
> >> > as follows:
> >> > [ ] +1, Approve the release
> >> > [ ] -1, Do not approve the release (please provide specific comments)
> >> >
> >> > **Release Overview**
> >> >
> >> > As an overview, the release consists of the following:
> >> > a) Kubernetes Operator canonical source distribution (including the
> >> > Dockerfile), to be deployed to the release repository at
> >> dist.apache.org
> >> > b) Kubernetes Operator Helm Chart to be deployed to the release
> >> repository
> >> > at dist.apache.org
> >> > c) Maven artifacts to be deployed to the Maven Central Repository
> >> > d) Docker image to be pushed to dockerhub
> >> >
> >> > **Staging Areas to Review**
> >> >
> >> > The staging areas containing the above mentioned artifacts are as
> >> follows,
> >> > for your review:
> >> > * All artifacts for a,b) can be found in the corresponding dev
> >> repository
> >> > at dist.apache.org [1]
> >> > * All artifacts for c) can be found at the Apache Nexus Repository [2]
> >> > * The docker image is staged on github [7]
> >> >
> >> > All artifacts are signed with the
> >> > key 911F218F79ACEA8EB453799EEE325DDEBFED467D [3]
> >> >
> >> > Other links for your review:
> >> > * JIRA release notes [4]
> >> > * source code tag "release-0.1.0-rc2" [5]
> >> > * PR to update the website Downloads page to include Kubernetes
> Operator
> >> > links [6]
> >> >
> >> > **Vote Duration**
> >> >
> >> > The voting time will run for at least 72 hours.
> >> > It is adopted by majority approval, with at least 3 PMC affirmative
> >> votes.
> >> >
> >> > **Note for Functional Verification**
> >> >
> >> > You can use the dev repository as a helm repo

Re: Change of focus

[Márton Balassi]

Thank you, Till. Good luck with the next chapter. :-)

On Mon, Feb 28, 2022 at 1:49 PM Flavio Pompermaier 
wrote:

> Good luck for your new adventure Till!
>
> On Mon, Feb 28, 2022 at 12:00 PM Till Rohrmann 
> wrote:
>
> > Hi everyone,
> >
> > I wanted to let you know that I will be less active in the community
> > because I’ve decided to start a new chapter in my life. Hence, please
> don’t
> > wonder if I might no longer be very responsive on mails and JIRA issues.
> >
> > It is great being part of such a great community with so many amazing
> > people. Over the past 7,5 years, I’ve learned a lot thanks to you and
> > together we have shaped how people think about stream processing
> nowadays.
> > This is something we can be very proud of. I am sure that the community
> > will continue innovating and setting the pace for what is possible with
> > real time processing. I wish you all godspeed!
> >
> > Cheers,
> > Till
> >
>

Re: [DISCUSS] FLIP-212: Introduce Flink Kubernetes Operator

[Márton Balassi]

> > > > > > > > > > Thanks,
> > > > > > > > > > > > Thomas
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Feb 4, 2022 at 5:15 AM Gyula Fóra <
> > > > > > gyula.f...@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi Danny,
> > > > > > > > > > > > >
> > > > > > > > > > > > > So far we have been focusing our dev efforts on the
> > > > initial
> > > > > > > > native
> > > > > > > > > > > > > implementation with the team.
> > > > > > > > > > > > > If the discussion and vote goes well for this FLIP
> we
> > > are
> > > > > > > looking
> > > > > > > > > > > forward
> > > > > > > > > > > > > to contributing the initial version sometime next
> > week
> > > > > > (fingers
> > > > > > > > > > > crossed).
> > > > > > > > > > > > >
> > > > > > > > > > > > > At that point I think we can already start the dev
> > work
> > > > to
> > > > > > > > support
> > > > > > > > > > the
> > > > > > > > > > > > > standalone mode as well, especially if you can
> > dedicate
> > > > > some
> > > > > > > > effort
> > > > > > > > > > to
> > > > > > > > > > > > > pushing that side.
> > > > > > > > > > > > > Working together on this sounds like a great idea
> and
> > > we
> > > > > > should
> > > > > > > > > start
> > > > > > > > > > > as
> > > > > > > > > > > > > soon as possible! :)
> > > > > > > > > > > > >
> > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > Gyula
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Fri, Feb 4, 2022 at 2:07 PM Danny Cranmer <
> > > > > > > > > > dannycran...@apache.org>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > I have been discussing this one with my team. We
> > are
> > > > > > > interested
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > > Standalone mode, and are willing to contribute
> > > towards
> > > > > the
> > > > > > > > > > > > implementation.
> > > > > > > > > > > > > > Potentially we can work together to support both
> > > modes
> > > > in
> > > > > > > > > parallel?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Wed, Feb 2, 2022 at 4:02 PM Gyula Fóra <
> > > > > > > > gyula.f...@gmail.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Danny!
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for the feedback :)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Versioning:
> > > > > > > > > > > > > > > Versioning will be independent from Flink and
> the
> > > > > >

Re: [VOTE] FLIP-212: Introduce Flink Kubernetes Operator

[Márton Balassi]

+1 (binding)

On Sat, Feb 5, 2022 at 5:35 PM Israel Ekpo  wrote:

> I am very excited to see this.
>
> Thanks for driving the effort
>
> +1 (non-binding)
>
>
> On Sat, Feb 5, 2022 at 10:53 AM Shqiprim Bunjaku <
> shqiprimbunj...@gmail.com>
> wrote:
>
> > +1 (non-binding)
> >
> >
> >
> > On Sat, Feb 5, 2022 at 4:39 PM Chenya Zhang  >
> > wrote:
> >
> > > +1 (non-binding)
> > >
> > > Thanks folks for leading this effort and making it happen so fast!
> > >
> > > Best,
> > > Chenya
> > >
> > > On Sat, Feb 5, 2022 at 12:02 AM Gyula Fóra  wrote:
> > >
> > > > Hi Thomas!
> > > >
> > > > +1 (binding) from my side
> > > >
> > > > Happy to see this effort getting some traction!
> > > >
> > > > Cheers,
> > > > Gyula
> > > >
> > > > On Sat, Feb 5, 2022 at 3:00 AM Thomas Weise  wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > I'd like to start a vote on FLIP-212: Introduce Flink Kubernetes
> > > > > Operator [1] which has been discussed in [2].
> > > > >
> > > > > The vote will be open for at least 72 hours unless there is an
> > > > > objection or not enough votes.
> > > > >
> > > > > [1]
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-212%3A+Introduce+Flink+Kubernetes+Operator
> > > > > [2]
> https://lists.apache.org/thread/1z78t6rf70h45v7fbd2m93rm2y1bvh0z
> > > > >
> > > > > Thanks!
> > > > > Thomas
> > > > >
> > > >
> > >
> >
> --
> Israel Ekpo
> Lead Instructor, IzzyAcademy.com
> https://www.youtube.com/c/izzyacademy
> https://izzyacademy.com/
>

Re: [DISCUSS] FLIP-212: Introduce Flink Kubernetes Operator

[Márton Balassi]

Hi team,

Thank you for the great feedback, Thomas has updated the FLIP page
accordingly. If you are comfortable with the currently existing design and
depth in the FLIP [1] I suggest moving forward to the voting stage - once
that reaches a positive conclusion it lets us create the separate code
repository under the flink project for the operator.

I encourage everyone to keep improving the details in the meantime, however
I believe given the existing design and the general sentiment on this
thread that the most efficient path from here is starting the
implementation so that we can collectively iterate over it.

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-212%3A+Introduce+Flink+Kubernetes+Operator

On Mon, Jan 31, 2022 at 10:15 PM Thomas Weise  wrote:

> HI Xintong,
>
> Thanks for the feedback and please see responses below -->
>
> On Fri, Jan 28, 2022 at 12:21 AM Xintong Song 
> wrote:
>
> > Thanks Thomas for drafting this FLIP, and everyone for the discussion.
> >
> > I also have a few questions and comments.
> >
> > ## Job Submission
> > Deploying a Flink session cluster via kubectl & CR and then submitting
> jobs
> > to the cluster via Flink cli / REST is probably the approach that
> requires
> > the least effort. However, I'd like to point out 2 weaknesses.
> > 1. A lot of users use Flink in perjob/application modes. For these users,
> > having to run the job in two steps (deploy the cluster, and submit the
> job)
> > is not that convenient.
> > 2. One of our motivations is being able to manage Flink applications'
> > lifecycles with kubectl. Submitting jobs from cli sounds not aligned with
> > this motivation.
> > I think it's probably worth it to support submitting jobs via kubectl &
> CR
> > in the first version, both together with deploying the cluster like in
> > perjob/application mode and after deploying the cluster like in session
> > mode.
> >
>
> The intention is to support application management through operator and CR,
> which means there won't be any 2 step submission process, which as you
> allude to would defeat the purpose of this project. The CR example shows
> the application part. Please note that the bare cluster support is an
> *additional* feature for scenarios that require external job management. Is
> there anything on the FLIP page that creates a different impression?
>
>
> >
> > ## Versioning
> > Which Flink versions does the operator plan to support?
> > 1. Native K8s deployment was firstly introduced in Flink 1.10
> > 2. Native K8s HA was introduced in Flink 1.12
> > 3. The Pod template support was introduced in Flink 1.13
> > 4. There was some changes to the Flink docker image entrypoint script in,
> > IIRC, Flink 1.13
> >
>
> Great, thanks for providing this. It is important for the compatibility
> going forward also. We are targeting Flink 1.14.x upwards. Before the
> operator is ready there will be another Flink release. Let's see if anyone
> is interested in earlier versions?
>
>
> >
> > ## Compatibility
> > What kind of API compatibility we can commit to? It's probably fine to
> have
> > alpha / beta version APIs that allow incompatible future changes for the
> > first version. But eventually we would need to guarantee backwards
> > compatibility, so that an early version CR can work with a new version
> > operator.
> >
>
> Another great point and please let me include that on the FLIP page. ;-)
>
> I think we should allow incompatible changes for the first one or two
> versions, similar to how other major features have evolved recently, such
> as FLIP-27.
>
> Would be great to get broader feedback on this one.
>
> Cheers,
> Thomas
>
>
>
> >
> > Thank you~
> >
> > Xintong Song
> >
> >
> >
> > On Fri, Jan 28, 2022 at 1:18 PM Thomas Weise  wrote:
> >
> > > Thanks for the feedback!
> > >
> > > >
> > > > # 1 Flink Native vs Standalone integration
> > > > Maybe we should make this more clear in the FLIP but we agreed to do
> > the
> > > > first version of the operator based on the native integration.
> > > > While this clearly does not cover all use-cases and requirements, it
> > > seems
> > > > this would lead to a much smaller initial effort and a nicer first
> > > version.
> > > >
> > >
> > > I'm also leaning towards the native integration, as long as it reduces
> > the
> > > MVP effort. Ultimately the operator will need to also support the
> > > standalone mode. I would like to gain more confidence that native
> > > integration reduces the effort. While it cuts the effort to handle the
> TM
> > > pod creation, some mapping code from the CR to the native integration
> > > client and config needs to be created. As mentioned in the FLIP, native
> > > integration requires the Flink job manager to have access to the k8s
> API
> > to
> > > create pods, which in some scenarios may be seen as unfavorable.
> > >
> > >  > > > # Pod Template
> > > > > > Is the pod template in CR same with what Flink has already
> > > > supported[4]?
> > > > > > Then I am afraid not

Re: [VOTE] FLIP-211: Kerberos delegation token framework

[Márton Balassi]

+1 (binding)

Given [1] I consider the issue David raised resolved. Thanks David and
please confirm here.

[1]  https://lists.apache.org/thread/cvwknd5fhohj0wfv8mfwn70jwpjvxrjj

On Mon, Jan 24, 2022 at 11:07 AM David Morávek 
wrote:

> Hi Gabor,
>
> Thanks for driving this. This is headed in a right direction, but I feel
> that the FLIP still might need bit more work.
>
> -1 (non-binding) until the discussion thread is resolved [1].
>
> [1] https://lists.apache.org/thread/cvwknd5fhohj0wfv8mfwn70jwpjvxrjj
>
> Best,
> D.
>
>
>
> On Mon, Jan 24, 2022 at 10:47 AM Gyula Fóra  wrote:
>
> > Hi Gabor,
> >
> > +1 (binding) from me
> >
> > This is a great effort and significant improvement to the Kerberos
> security
> > story .
> >
> > Cheers
> > Gyula
> >
> > On Fri, 21 Jan 2022 at 15:58, Gabor Somogyi 
> > wrote:
> >
> > > Hi devs,
> > >
> > > I would like to start the vote for FLIP-211 [1], which was discussed
> and
> > > reached a consensus in the discussion thread [2].
> > >
> > > The vote will be open for at least 72h, unless there is an objection or
> > not
> > > enough votes.
> > >
> > > BR,
> > > G
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-211%3A+Kerberos+delegation+token+framework
> > >
> > > [2] https://lists.apache.org/thread/cvwknd5fhohj0wfv8mfwn70jwpjvxrjj
> > >
> >
>

Re: Flink native k8s integration vs. operator

[Márton Balassi]

Hi All,

I am pleased to see the level of enthusiasm and technical consideration
already emerging in this thread. I wholeheartedly support building an
operator and endorsing it via placing it under the Apache Flink umbrella
(as a separate repository) as the current lack of it is clearly becoming an
adoption bottleneck for large scale Flink users. The next logical step is
to write a FLIP to agree on the technical details, so that we can put
forward the proposal to the Flink PMC for creating a new repository with a
clear purpose in mind. I volunteer to work with Thomas and Gyula on the
initial wording on the proposal which we will put up for public discussion
in the coming weeks.

Best,
Marton

On Thu, Jan 13, 2022 at 9:22 AM Konstantin Knauf  wrote:

> Hi Thomas,
>
> Yes, I was referring to a separate repository under Apache Flink.
>
> Cheers,
>
> Konstantin
>
> On Thu, Jan 13, 2022 at 6:19 AM Thomas Weise  wrote:
>
>> Hi everyone,
>>
>> Thanks for the feedback and discussion. A few additional thoughts:
>>
>> [Konstantin] > With respect to common lifecycle management operations:
>> these features are
>> > not available (within Apache Flink) for any of the other resource
>> providers
>> > (YARN, Standalone) either. From this perspective, I wouldn't consider
>> this
>> > a shortcoming of the Kubernetes integration.
>>
>> I think time and evolution of the ecosystem are factors to consider as
>> well. The state and usage of Flink was much different when YARN
>> integration was novel. Expectations are different today and the
>> lifecycle functionality provided by an operator may as well be
>> considered essential to support the concept of a Flink application on
>> k8s. After few years learning from operator experience outside of
>> Flink it might be a good time to fill the gap.
>>
>> [Konstantin] > I still believe that we should keep this focus on low
>> > level composable building blocks (like Jobs and Snapshots) in Apache
>> Flink
>> > to make it easy for everyone to build fitting higher level abstractions
>> > like a FlinkApplication Custom Resource on top of it.
>>
>> I completely agree that it is important that the basic functions of
>> Flink are solid and continued focus is necessary. Thanks for sharing
>> the pointers, these are great improvements. At the same time,
>> ecosystem, contributor base and user spectrum are growing. There have
>> been significant additions in many areas of Flink including connectors
>> and higher level abstractions like statefun, SQL and Python. It's also
>> evident from additional repositories/subprojects that we have in Flink
>> today.
>>
>> [Konstantin] > Having said this, if others in the community have the
>> capacity to push and
>> > *maintain* a somewhat minimal "reference" Kubernetes Operator for Apache
>> > Flink, I don't see any blockers. If or when this happens, I'd see some
>> > clear benefits of using a separate repository (easier independent
>> > versioning and releases, different build system & tooling (go, I
>> assume)).
>>
>> Naturally different contributors to the project have different focus.
>> Let's find out if there is strong enough interest to take this on and
>> strong enough commitment to maintain. As I see it, there is a
>> tremendous amount of internal investment going into operationalizing
>> Flink within many companies. Improvements to the operational side of
>> Flink like the operator would complement Flink nicely. I assume that
>> you are referring to a separate repository within Apache Flink, which
>> would give it the chance to achieve better sustainability than the
>> existing external operator efforts. There is also the fact that some
>> organizations which are heavily invested in operationalizing Flink are
>> allowing contributing to Apache Flink itself but less so to arbitrary
>> github projects. Regarding the tooling, it could well turn out that
>> Java is a good alternative given the ecosystem focus and that there is
>> an opportunity for reuse in certain aspects (metrics, logging etc.).
>>
>> [Yang] > I think Xintong has given a strong point why we introduced
>> the native K8s integration, which is active resource management.
>> > I have a concrete example for this in the production. When a K8s node
>> is down, the standalone K8s deployment will take longer
>> > recovery time based on the K8s eviction time(IIRC, default is 5
>> minutes). For the native K8s integration, Flink RM could be aware of the
>> > TM heartbeat lost and allocate a new one timely.
>>
>> Thanks for sharing this, we should evaluate it as part of a proposal.
>> If we can optimize recovery or scaling with active resource management
>> then perhaps it is worth to support it through the operator.
>> Previously mentioned operators all rely on the standalone model.
>>
>> Cheers,
>> Thomas
>>
>> On Wed, Jan 12, 2022 at 3:21 AM Konstantin Knauf 
>> wrote:
>> >
>> > cc dev@
>> >
>> > Hi Thomas, Hi everyone,
>> >
>> > Thank you for starting this discussion and sorry for chiming 

Re: [DISCUSS] FLIP-211: Kerberos delegation token framework

[Márton Balassi]

Hi G,

Thanks for taking this challenge on. Scalable Kerberos authentication
support is important for Flink, delegation tokens is a great mechanism to
future-proof this. I second your assessment that the existing
implementation could use some improvement too and like the approach you
have outlined. It is crucial that the changes are self-contained and will
not affect users that do not use Kerberos, while are minimal for the ones
who do (configuration values change, but the defaults just keep working in
most cases).

Thanks,
Marton

On Tue, Jan 11, 2022 at 2:59 PM Gabor Somogyi 
wrote:

> Hi All,
>
> Hope all of you have enjoyed the holiday season.
>
> I would like to start the discussion on FLIP-211
> <
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-211%3A+Kerberos+delegation+token+framework
> >
> which
> aims to provide a
> Kerberos delegation token framework that /obtains/renews/distributes tokens
> out-of-the-box.
>
> Please be aware that the FLIP wiki area is not fully done since the
> discussion may
> change the feature in major ways. The proposal can be found in a google doc
> here
> <
> https://docs.google.com/document/d/1JzMbQ1pCJsLVz8yHrCxroYMRP2GwGwvacLrGyaIx5Yc/edit?fbclid=IwAR0vfeJvAbEUSzHQAAJfnWTaX46L6o7LyXhMfBUCcPrNi-uXNgoOaI8PMDQ
> >
> .
> As the community agrees on the approach the content will be moved to the
> wiki page.
>
> Feel free to add your thoughts to make this feature better!
>
> BR,
> G
>

Re: Looking for Maintainers for Flink on YARN

[Márton Balassi]

Hi Konstantin,

Thank you for raising this topic, our development team at Cloudera would be
happy to step up to address this responsibility.

Best,
Marton

On Thu, Jul 29, 2021 at 10:15 AM Konstantin Knauf  wrote:

> Dear community,
>
> We are looking for community members, who would like to maintain Flink's
> YARN support going forward. So far, this has been handled by teams at
> Ververica & Alibaba. The focus of these teams has shifted over the past
> months so that we only have little time left for this topic. Still, we
> think, it is important to maintain high quality support for Flink on YARN.
>
> What does "Maintaining Flink on YARN" mean? There are no known bigger
> efforts outstanding. We are mainly talking about addressing
> "test-stability" issues, bugs, version upgrades, community contributions &
> smaller feature requests. The prioritization of these would be up to the
> future maintainers, except "test-stability" issues which are important to
> address for overall productivity.
>
> If a group of community members forms itself, we are happy to give an
> introduction to relevant pieces of the code base, principles, assumptions,
> ... and hand over open threads.
>
> If you would like to take on this responsibility or can join this effort in
> a supporting role, please reach out!
>
> Cheers,
>
> Konstantin
> for the Deployment & Coordination Team at Ververica
>
> --
>
> Konstantin Knauf
>
> https://twitter.com/snntrable
>
> https://github.com/knaufk
>

Re: [VOTE] FLIP-181: Custom netty HTTP request inbound/outbound handlers

[Márton Balassi]

Thanks for your input, team. Good catch, Chesney.

Till, we will address said comments.

All in all now we can close the vote successfully with the binding +1 votes
of Gyula, Konstantin and Till.

On Mon, Jul 12, 2021 at 10:42 AM Till Rohrmann  wrote:

> Thanks for starting the vote Marton.
>
> I have two comments:
>
> * I would suggest that the interfaces return Optional or
> at least have a @Nullable annotation in order to make the contract explicit.
> * The test plan should contain tests for the general infrastructure which
> should live in Flink. We should test that factories are loaded and that the
> handlers are set up in the correct order.
>
> I would consider these two changes to the original FLIP small. I give
> my +1 (binding) conditionally under the assumption that the comments will
> be addressed.
>
> Cheers,
> Till
>
> On Mon, Jul 12, 2021 at 10:15 AM Konstantin Knauf 
> wrote:
>
>> +1 (binding)
>>
>> Assuming that we continue to vote in this thread for now.
>>
>> Thank you for your patience!
>>
>> On Mon, Jul 12, 2021 at 8:56 AM Chesnay Schepler 
>> wrote:
>>
>> > The vote has not reached the required number of votes to be considered
>> > successful.
>> >
>> > As outlined in the bylaws
>> > <
>> >
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120731026#FlinkBylaws-Actions
>> >
>> >
>> > FLIP votes require 3 binding +1 votes (i.e., from committers).
>> >
>> > On 10/07/2021 16:13, Márton Balassi wrote:
>> > > Hi team,
>> > >
>> > > Thank you for your input, I am closing this vote as successful.
>> > > Austin: thank you, I have added the experimental annotation
>> explicitly to
>> > > the FLIP.
>> > >
>> > > On Tue, Jul 6, 2021 at 5:17 PM Gabor Somogyi <
>> gabor.g.somo...@gmail.com>
>> > > wrote:
>> > >
>> > >> +1 (non-binding)
>> > >> The @Experimental annotation is really missing, Marton could you add
>> it
>> > >> please?
>> > >>
>> > >>
>> > >> On Tue, Jul 6, 2021 at 5:04 PM Austin Cawley-Edwards <
>> > >> austin.caw...@gmail.com> wrote:
>> > >>
>> > >>> Hi Márton,
>> > >>>
>> > >>> The FLIP looks generally good to me, though could we add the
>> > >>> `@Experimental` annotation to the proposed interfaces so it is in
>> sync
>> > >> with
>> > >>> what was agreed in the discussion thread?
>> > >>>
>> > >>> Thanks,
>> > >>> Austin
>> > >>>
>> > >>> On Tue, Jul 6, 2021 at 9:40 AM Gyula Fóra 
>> wrote:
>> > >>>
>> > >>>> +1 from my side
>> > >>>>
>> > >>>> This is a good addition that will open many possibilities in the
>> > future
>> > >>> and
>> > >>>> solve some immediate issues with the current Kerberos integration.
>> > >>>>
>> > >>>> Gyula
>> > >>>>
>> > >>>> On Tue, Jul 6, 2021 at 2:50 PM Márton Balassi <
>> > >> balassi.mar...@gmail.com>
>> > >>>> wrote:
>> > >>>>
>> > >>>>> Hi everyone, I would like to start a vote on FLIP-181 [1] which
>> was
>> > >>>>> discussed in this thread [2]. The vote will be open for at least
>> 72
>> > >>> hours
>> > >>>>> until July 9th unless there is an objection or not enough votes.
>> > >>>>>
>> > >>>>> [1] https://cwiki.apache.org/confluence/x/CAUBCw
>> > >>>>> [2]
>> > >>>>>
>> > >>>>>
>> > >>
>> >
>> https://lists.apache.org/thread.html/r53b6b8931b6248a849855dad27b1a431e55cdd48ca055910e8f015a8%40%3Cdev.flink.apache.org%3E
>> >
>> >
>> >
>>
>> --
>>
>> Konstantin Knauf
>>
>> https://twitter.com/snntrable
>>
>> https://github.com/knaufk
>>
>

Re: Introduction email

[Márton Balassi]

Hi Srini,

Welcome to the community. Awesome to hear that LinkedIn decided to embrace
Flink, excited to see where this joint path leads us.

On Mon, Jul 19, 2021 at 11:02 AM Timo Walther  wrote:

> Hi Srini,
>
> welcome aboard! Great to see more adoption in the SQL space. Looking
> forward to collaboration.
>
> Regards,
> Timo
>
> On 19.07.21 10:58, Till Rohrmann wrote:
> > Hi Srini,
> >
> > Welcome to the Flink community :-) Great to hear what you are planning to
> > do with Flink at LinkedIn. I think sharing this is very motivational for
> > the community and also gives context for what you are focusing on.
> Looking
> > forward to working with you and improving Flink.
> >
> > Cheers,
> > Till
> >
> > On Fri, Jul 16, 2021 at 8:36 PM Srinivasulu Punuru <
> srinipunur...@gmail.com>
> > wrote:
> >
> >> Hi Flink Devs,
> >>
> >> I am Srini, I work for stream processing team at LinkedIn. LinkedIn is
> >> taking a big bet on Apache Flink and migrating all the existing
> streaming
> >> SQL apps to Flink. You might have seen mails from some of our team
> members
> >> past few months. Thanks a lot for your support!
> >>
> >> I just wanted to Say Hi to everyone before I take up some of the starter
> >> Jiras and start contributing.
> >>
> >> Thanks Again! Looking forward to collaboration :)
> >>
> >> Here are some of the quick notes about our Flink scenarios.
> >>
> >> 1. We will be using Flink SQL just for stream processing
> applications.
> >> 2. Most of our current SQL apps are stateless, But stateful SQL
> >> capabilities is one of the reasons we are migrating to Flink. SQL
> state
> >> management is an area of interest.
> >> 3. We also have customers asking for batch and streaming
> convergence, So
> >> SQL based batch <-> streaming convergence or engine portability of
> SQL
> >> apps
> >> is an area of interest.
> >> 4. We are initially on prem. But LinkedIn as a whole is betting on
> >> Cloud. So taking advantage some of the cloud capabilities like
> Storage
> >> compute disaggregation, Elastic compute (for auto-scaling) for Flink
> >> would
> >> be interesting.
> >> 5. We also provide a managed streaming SQL service i.e. We manage
> the
> >> SQL jobs for our developers. So reliability, operability and quick
> >> recovery
> >> is critical as well :).
> >>
> >> Thanks,
> >> Srini.
> >>
> >
>
>

Re: [VOTE] FLIP-181: Custom netty HTTP request inbound/outbound handlers

[Márton Balassi]

Hi team,

Thank you for your input, I am closing this vote as successful.
Austin: thank you, I have added the experimental annotation explicitly to
the FLIP.

On Tue, Jul 6, 2021 at 5:17 PM Gabor Somogyi 
wrote:

> +1 (non-binding)
> The @Experimental annotation is really missing, Marton could you add it
> please?
>
>
> On Tue, Jul 6, 2021 at 5:04 PM Austin Cawley-Edwards <
> austin.caw...@gmail.com> wrote:
>
> > Hi Márton,
> >
> > The FLIP looks generally good to me, though could we add the
> > `@Experimental` annotation to the proposed interfaces so it is in sync
> with
> > what was agreed in the discussion thread?
> >
> > Thanks,
> > Austin
> >
> > On Tue, Jul 6, 2021 at 9:40 AM Gyula Fóra  wrote:
> >
> > > +1 from my side
> > >
> > > This is a good addition that will open many possibilities in the future
> > and
> > > solve some immediate issues with the current Kerberos integration.
> > >
> > > Gyula
> > >
> > > On Tue, Jul 6, 2021 at 2:50 PM Márton Balassi <
> balassi.mar...@gmail.com>
> > > wrote:
> > >
> > > > Hi everyone, I would like to start a vote on FLIP-181 [1] which was
> > > > discussed in this thread [2]. The vote will be open for at least 72
> > hours
> > > > until July 9th unless there is an objection or not enough votes.
> > > >
> > > > [1] https://cwiki.apache.org/confluence/x/CAUBCw
> > > > [2]
> > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/r53b6b8931b6248a849855dad27b1a431e55cdd48ca055910e8f015a8%40%3Cdev.flink.apache.org%3E
> > > >
> > >
> >
>

[VOTE] FLIP-181: Custom netty HTTP request inbound/outbound handlers

[Márton Balassi]

Hi everyone, I would like to start a vote on FLIP-181 [1] which was
discussed in this thread [2]. The vote will be open for at least 72 hours
until July 9th unless there is an objection or not enough votes.

[1] https://cwiki.apache.org/confluence/x/CAUBCw
[2]
https://lists.apache.org/thread.html/r53b6b8931b6248a849855dad27b1a431e55cdd48ca055910e8f015a8%40%3Cdev.flink.apache.org%3E

Re: [DISCUSS] Dashboard/HistoryServer authentication

[Márton Balassi]

omething that we do
> not
> >> have control over. It also locks Flink into the current major version of
> >> Netty (and the Netty framework itself) for the foreseeable future.
> >>
> >> I am saying we should not do this, perhaps this is the best solution to
> >> finding a good compromise here, but I am trying to discover +
> acknowledge
> >> the full implications of this proposal so they can be discussed.
> >>
> >> What do you think about marking this API as @Experimental and not
> >> guaranteeing stability between versions? Then, if we do decide we need
> to
> >> upgrade Netty (or move away from it), we can do so.
> >>
> >> * I share Till's concern about multiple factories – other HTTP
> middleware
> >>>> frameworks commonly support chaining middlewares. Since the proposed
> API
> >>>> does not include these features/guarantee ordering, do you see any
> reason
> >>>> to allow more than one factory?
> >>>>
> >>> I personally can't come up with a use-case where ordering is a must.
> I'm
> >>> not telling that this is not a valid use-case but adding a feature w/o
> >>> business rationale would include the maintenance cost (though I'm open
> to
> >>> add).
> >>> As I've seen Till also can't give example for that (please see the doc
> >>> comments). If you have anything in mind please share it and we can add
> >>> priority to the API.
> >>> There is another option too, namely we can be defensive and we can add
> >>> the priority right now. I would do this only if everybody states in
> mail
> >>> that it would be the best option,
> >>> otherwise I would stick to the original plan.
> >>>
> >>
> >> Let me try to come up with a use case:
> >> * Someone creates an authentication module for integrating with Google's
> >> OAuth and publishes it to flink-packages
> >> * Another person in another org wants to use Google OAuth and then add
> >> internal authorization based on the user
> >> * In this scenario, *Google OAuth must come before the internal
> >> authorization*
> >> * They place their module and the Google OAuth module to be picked up by
> >> the service loader
> >> * What happens?
> >>
> >> I do not think that the current proposal has a way to handle this,
> >> besides having the implementor of the internal authorization module
> bundle
> >> everything into one, as you have suggested. Since this is the only way
> to
> >> achieve order, why not restrict the service loader to only allow one?
> This
> >> way the API is explicit in what it supports.
> >>
> >>
> >> Let me know what you think,
> >> Austin
> >>
> >>
> >> On Wed, Jun 30, 2021 at 5:24 AM Gabor Somogyi <
> gabor.g.somo...@gmail.com>
> >> wrote:
> >>
> >>> Hi Austin,
> >>>
> >>> Please see my answers embedded down below.
> >>>
> >>> BR,
> >>> G
> >>>
> >>>
> >>>
> >>> On Tue, Jun 29, 2021 at 9:59 PM Austin Cawley-Edwards <
> >>> austin.caw...@gmail.com> wrote:
> >>>
> >>>> Hi all,
> >>>>
> >>>> Thanks for the updated proposal. I have a few questions about the API,
> >>>> please see below.
> >>>>
> >>>> * What stability semantics do you envision for this API?
> >>>>
> >>> As I foresee the API will be as stable as Netty API. Since there is
> >>> guarantee on no breaking changes between minor versions we can give the
> >>> same guarantee.
> >>> If for whatever reason we need to break it we can do it in major
> version
> >>> like every other open source project does.
> >>>
> >>>
> >>>> * Does Flink expose dependencies’ APIs in other places? Since this
> >>>> exposes the Netty API, will this make it difficult to upgrade Netty?
> >>>>
> >>> I don't expect breaking changes between minor versions so such cases
> >>> there will be no issues. If there is a breaking change in major version
> >>> we need to wait Flink major version too.
> >>>
> >>>
> >>>> * I share Till's concern about multiple factories – other HTTP
> >>>> middleware frameworks commonly support chaining middlewa

Re: [DISCUSS] Dashboard/HistoryServer authentication

[Márton Balassi]

Hi all,

I commend Konstantin and Till when it comes to standing up for the
community values.

Based on your feedback we are withdrawing the original proposal and
attaching a more general custom netty handler API proposal [1] written by
G. The change necessary to the Flink repository is approximately 500 lines
of code. [2]

Please let us focus on discussing the details of this API and whether it
covers the necessary use cases.

[1]
https://docs.google.com/document/d/1Idnw8YauMK1x_14iv0rVF0Hqm58J6Dg-hi-hEuL6hwM/edit#heading=h.ijcbce3c5gip
[2]
https://github.com/gaborgsomogyi/flink/commit/942f23679ac21428bb87fc85557b9b443fcaf310

Thanks,
Marton

On Wed, Jun 23, 2021 at 9:36 PM Austin Cawley-Edwards <
austin.caw...@gmail.com> wrote:

> Hi all,
>
> Thanks, Konstantin and Till, for guiding the discussion.
>
> I was not aware of the results of the call with Konstantin and was
> attempting to resolve the unanswered questions before more, potentially
> fruitless, work was done.
>
> I am also looking forward to the coming proposal, as well as increasing my
> understanding of this specific use case + its limitations!
>
> Best,
> Austin
>
> On Tue, Jun 22, 2021 at 6:32 AM Till Rohrmann 
> wrote:
>
> > Hi everyone,
> >
> > I do like the idea of keeping the actual change outside of Flink but to
> > enable Flink to support such a use case (different authentication
> > mechanisms). I think this is a good compromise for the community that
> > combines long-term maintainability with support for new use-cases. I am
> > looking forward to your proposal.
> >
> > I also want to second Konstantin here that the tone of your last email,
> > Marton, does not reflect the values and manners of the Flink community
> and
> > is not representative of how we conduct discussions. Especially, the more
> > senior community members should know this and act accordingly in order to
> > be good role models for others in the community. Technical discussions
> > should not be decided by who wields presumably the greatest authority but
> > by the soundness of arguments and by what is the best solution for a
> > problem.
> >
> > Let us now try to find the best solution for the problem at hand!
> >
> > Cheers,
> > Till
> >
> > On Tue, Jun 22, 2021 at 11:24 AM Konstantin Knauf 
> > wrote:
> >
> > > Hi everyone,
> > >
> > > First, Marton and I had a brief conversation yesterday offline and
> > > discussed exploring the approach of exposing the authentication
> > > functionality via an API. So, I am looking forward to your proposal in
> > that
> > > direction. The benefit of such a solution would be that it is
> extensible
> > > for others and it does add a smaller maintenance (in particular
> testing)
> > > footprint to Apache Flink itself. If we end up going down this route,
> > > flink-packages.org would be a great way to promote these third party
> > > "authentication modules".
> > >
> > > Second, Marton, I understand your frustration about the long discussion
> > on
> > > this "simple matter", but the condescending tone of your last mail
> feels
> > > uncalled for to me. Austin expressed a valid opinion on the topic,
> which
> > is
> > > based on his experience from other Open Source frameworks (CNCF
> mostly).
> > I
> > > am sure you agree that it is important for Apache Flink to stay open
> and
> > to
> > > consider different approaches and ideas and I don't think it helps the
> > > culture of discussion to shoot it down like this ("This is where this
> > > discussion stops.").
> > >
> > > Let's continue to move this discussion forward and I am sure we'll
> find a
> > > consensus based on product and technological considerations.
> > >
> > > Thanks,
> > >
> > > Konstantin
> > >
> > > On Tue, Jun 22, 2021 at 9:31 AM Márton Balassi <
> balassi.mar...@gmail.com
> > >
> > > wrote:
> > >
> > > > Hi Austin,
> > > >
> > > > Thank you for your thoughts. This is where this discussion stops.
> This
> > > > email thread already contains more characters than the implementation
> > and
> > > > what is needed for the next 20 years of maintenance.
> > > >
> > > > It is great that you have a view on modern solutions and thank you
> for
> > > > offering your help with brainstorming solutions. I am responsible for
> > > Flink
> > > > at Cloudera and we d

Re: [DISCUSS] Dashboard/HistoryServer authentication

[Márton Balassi]

Hi Austin,

Thank you for your thoughts. This is where this discussion stops. This
email thread already contains more characters than the implementation and
what is needed for the next 20 years of maintenance.

It is great that you have a view on modern solutions and thank you for
offering your help with brainstorming solutions. I am responsible for Flink
at Cloudera and we do need an implementation like this and it is in fact
already in production at dozens of customers. We are open to adapting that
to expose a more generic API (and keeping Kerberos to our fork), to
contribute this to the community as others have asked for it and to protect
ourselves from occasionally having to update this critical implementation
path based on changes in the Apache codebase. I have worked with close to a
hundred Big Data customers as a consultant and an engineering manager and
committed hundreds of changes to Apache Flink over the past decade, please
trust my judgement on a simple matter like this.

Please forgive me for referencing authority, this discussion was getting
out of hand. Please keep vigilant.

Best,
Marton

On Mon, Jun 21, 2021 at 10:50 PM Austin Cawley-Edwards <
austin.caw...@gmail.com> wrote:

> Hi Gabor + Marton,
>
> I don't believe that the issue with this proposal is the specific mechanism
> proposed (Kerberos), but rather that it is not the level to implement it at
> (Flink). I'm just one voice, so please take this with a grain of salt.
>
> In the other solutions previously noted there is no need to instrument
> Flink which, in addition to reducing the maintenance burden, provides a
> better, decoupled end result.
>
> IMO we should not add any new API in Flink for this use case. I think it is
> unfortunate and sympathize with the work that has already been done on this
> feature – perhaps we could brainstorm ways to run this alongside Flink in
> your setup. Again, I don't think the proposed solution of an agnostic API
> would not work, nor is it a bad idea, but is not one that will make Flink
> more compatible with the modern solutions to this problem.
>
> Best,
> Austin
>
> On Mon, Jun 21, 2021 at 2:18 PM Márton Balassi 
> wrote:
>
> > Hi team,
> >
> > Thank you for your input. Based on this discussion I agree with G that
> > selecting and standardizing on a specific strong authentication mechanism
> > is more challenging than the whole rest of the scope of this
> authentication
> > story. :-) I suggest that G and I go back to the drawing board and come
> up
> > with an API that can support multiple authentication mechanisms, and we
> > would only merge said API to Flink. Specific implementations of it can be
> > maintained outside of the project. This way we tackle the main challenge
> in
> > a truly minimal way.
> >
> > Best,
> > Marton
> >
> > On Mon, Jun 21, 2021 at 4:18 PM Gabor Somogyi  >
> > wrote:
> >
> > > Hi All,
> > >
> > > We see that adding any kind of specific authentication raises more
> > > questions than answers.
> > > What would be if a generic API would be added without any real
> > > authentication logic?
> > > That way every provider can add its own protocol implementation as
> > > additional jar.
> > >
> > > BR,
> > > G
> > >
> > >
> > > On Thu, Jun 17, 2021 at 7:53 PM Austin Cawley-Edwards <
> > > austin.caw...@gmail.com> wrote:
> > >
> > >> Hi all,
> > >>
> > >> Sorry to be joining the conversation late. I'm also on the side of
> > >> Konstantin, generally, in that this seems to not be a core goal of
> Flink
> > >> as
> > >> a project and adds a maintenance burden.
> > >>
> > >> Would another con of Kerberos be that is likely a fading project in
> > terms
> > >> of network security? (serious question, please correct me if there is
> > >> reason to believe it is gaining adoption)
> > >>
> > >> The point about Kerberos being independent of infrastructure is a good
> > one
> > >> but is something that is also solved by modern sidecar proxies +
> service
> > >> meshes that can run across Kubernetes and bare-metal. These solutions
> > also
> > >> handle certificate provisioning, rotation, etc. in addition to
> > >> higher-level
> > >> authorization policies. Some examples of projects with this "universal
> > >> infrastructure support" are Kuma[1] (CNCF Sandbox, I'm a maintainer)
> and
> > >> Istio[2] (Google).
> > >>
> > >> Wondering out lo

Re: [DISCUSS] Dashboard/HistoryServer authentication

[Márton Balassi]

t;>> >> >> G
>> > >>> >> >>
>> > >>> >> >>
>> > >>> >> >> On Fri, Jun 4, 2021 at 3:31 PM Till Rohrmann <
>> > trohrm...@apache.org
>> > >>> >
>> > >>> >> >> wrote:
>> > >>> >> >>
>> > >>> >> >>> As I've said I am not a security expert and that's why I
>> have to
>> > >>> ask
>> > >>> >> for
>> > >>> >> >>> clarification, Gabor. You are saying that if we configure a
>> > >>> >> truststore for
>> > >>> >> >>> the REST endpoint with a single trusted certificate which has
>> > been
>> > >>> >> >>> generated by the operator of the Flink cluster, then the
>> > attacker
>> > >>> can
>> > >>> >> >>> generate a new certificate, sign it and then talk to the
>> Flink
>> > >>> >> cluster if
>> > >>> >> >>> he has access to the node on which the REST endpoint runs? My
>> > >>> >> understanding
>> > >>> >> >>> was that you need the corresponding private key which in my
>> > >>> proposed
>> > >>> >> setup
>> > >>> >> >>> would be under the control of the operator as well (e.g.
>> stored
>> > >>> in a
>> > >>> >> >>> keystore on the same machine but guarded by some secret).
>> That
>> > way
>> > >>> >> (if I am
>> > >>> >> >>> not mistaken), only the entity which has access to the
>> keystore
>> > is
>> > >>> >> able to
>> > >>> >> >>> talk to the Flink cluster.
>> > >>> >> >>>
>> > >>> >> >>> Maybe we are also getting our wires crossed here and are
>> talking
>> > >>> about
>> > >>> >> >>> different things.
>> > >>> >> >>>
>> > >>> >> >>> Thanks for listing the pros and cons of Kerberos. Concerning
>> > what
>> > >>> >> other
>> > >>> >> >>> authentication mechanisms are used in the industry, I am not
>> > 100%
>> > >>> >> sure.
>> > >>> >> >>>
>> > >>> >> >>> Cheers,
>> > >>> >> >>> Till
>> > >>> >> >>>
>> > >>> >> >>> On Fri, Jun 4, 2021 at 11:09 AM Gabor Somogyi <
>> > >>> >> gabor.g.somo...@gmail.com>
>> > >>> >> >>> wrote:
>> > >>> >> >>>
>> > >>> >> >>>> > I did not mean for the user to sign its own certificates
>> but
>> > >>> for
>> > >>> >> the
>> > >>> >> >>>> operator of the cluster. Once the user request hits the
>> proxy,
>> > it
>> > >>> >> should no
>> > >>> >> >>>> longer be under his control. I think I do not fully
>> understand
>> > >>> yet
>> > >>> >> why this
>> > >>> >> >>>> would not work.
>> > >>> >> >>>> I said it's not solving the authentication problem over any
>> > >>> proxy.
>> > >>> >> Even
>> > >>> >> >>>> if the operator is signing the certificate one can have
>> access
>> > >>> to an
>> > >>> >> >>>> internal node.
>> > >>> >> >>>> Such case anybody can craft certificates which is accepted
>> by
>> > the
>> > >>> >> >>>> server. When it's accepted a bad guy can cancel jobs causing
>> > huge
>> > >>> >> impacts.
>> > >>> >> >>>>
>> > >>> >> >>>> > Also, I am missing a bit the comparison of Kerberos to
>> other
>> > >>> >> >>>> authentication mechanisms and why they were rejected in
>> favour
&

Re: [DISCUSS] Dashboard/HistoryServer authentication

[Márton Balassi]

ncluding k8s.
>>>> The main intention is to use kerberos in k8s deployments too since we're
>>>> going this direction as well.
>>>> Please see how Spark does this:
>>>>
>>>> https://spark.apache.org/docs/latest/security.html#secure-interaction-with-kubernetes
>>>>
>>>> Last but not least the most important reason to add at least one strong
>>>> authentication is that we have users who has
>>>> hard requirements on this. They're doing security audits and if they
>>>> fail
>>>> then it's deal breaking.
>>>> That is why we have added kerberos at the first place. Unfortunately we
>>>> can't name them in this public list, however
>>>> the customers who specifically asked for this were mainly in the banking
>>>> and telco sector.
>>>>
>>>> BR,
>>>> G
>>>>
>>>>
>>>> On Thu, Jun 3, 2021 at 9:20 AM Till Rohrmann 
>>>> wrote:
>>>>
>>>> > Thanks for updating the document Márton. Why is it that banks will
>>>> > consider it more secure if Flink comes with Kerberos authentication
>>>> > (assuming a properly secured setup)? I mean if an attacker can get
>>>> access
>>>> > to one of the machines, then it should also be possible to obtain the
>>>> right
>>>> > Kerberos token.
>>>> >
>>>> > I am not an authentication expert and that's why I wanted to ask what
>>>> are
>>>> > other authentication protocols other than Kerberos? Why did we select
>>>> > Kerberos and not any other authentication protocol? Maybe you can
>>>> list the
>>>> > pros and cons for the different protocols. Is Kerberos also the
>>>> standard
>>>> > authentication protocol for Kubernetes deployments? If not, what
>>>> would be
>>>> > the answer when deploying on K8s?
>>>> >
>>>> > Cheers,
>>>> > Till
>>>> >
>>>> > On Wed, Jun 2, 2021 at 12:07 PM Gabor Somogyi <
>>>> gabor.g.somo...@gmail.com>
>>>> > wrote:
>>>> >
>>>> >> Hi team,
>>>> >>
>>>> >> Happy to be here and hope I can provide quality additions in the
>>>> future.
>>>> >>
>>>> >> Thank you all for helpful the suggestions!
>>>> >> Considering them the FLIP has been modified and the work continues
>>>> on the
>>>> >> already existing Jira.
>>>> >>
>>>> >> BR,
>>>> >> G
>>>> >>
>>>> >>
>>>> >> On Wed, Jun 2, 2021 at 11:23 AM Márton Balassi <
>>>> balassi.mar...@gmail.com>
>>>> >> wrote:
>>>> >>
>>>> >>> Thanks, Chesney - I totally missed that. Answered on the ticket
>>>> too, let
>>>> >>> us continue there then.
>>>> >>>
>>>> >>> Till, I agree that we should keep this codepath as slim as
>>>> possible. It
>>>> >>> is an important design decision that we aim to keep the list of
>>>> >>> authentication protocols to a minimum. We believe that this should
>>>> not be a
>>>> >>> primary concern of Flink and a trusted proxy service (for example
>>>> Apache
>>>> >>> Knox) should be used to enable a multitude of enduser authentication
>>>> >>> mechanisms. The bare minimum of authentication mechanisms to support
>>>> >>> consequently consist of a single strong authentication protocol for
>>>> which
>>>> >>> Kerberos is the enterprise solution and HTTP Basic primary for
>>>> development
>>>> >>> and light-weight scenarios.
>>>> >>>
>>>> >>> Added the above wording to G's doc.
>>>> >>>
>>>> >>>
>>>> https://docs.google.com/document/d/1NMPeJ9H0G49TGy3AzTVVJVKmYC0okwOtqLTSPnGqzHw/edit
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> On Tue, Jun 1, 2021 at 11:47 AM Chesnay Schepler <
>>>> ches...@apache.org>
>>>> >>> wrote:
>>>> >>>
>>>> >>>> There's a related effort:
&g

Re: [DISCUSS] Dashboard/HistoryServer authentication

[Márton Balassi]

Thanks, Chesney - I totally missed that. Answered on the ticket too, let us
continue there then.

Till, I agree that we should keep this codepath as slim as possible. It is
an important design decision that we aim to keep the list of authentication
protocols to a minimum. We believe that this should not be a primary
concern of Flink and a trusted proxy service (for example Apache Knox)
should be used to enable a multitude of enduser authentication mechanisms.
The bare minimum of authentication mechanisms to support consequently
consist of a single strong authentication protocol for which Kerberos is
the enterprise solution and HTTP Basic primary for development and
light-weight scenarios.

Added the above wording to G's doc.
https://docs.google.com/document/d/1NMPeJ9H0G49TGy3AzTVVJVKmYC0okwOtqLTSPnGqzHw/edit



On Tue, Jun 1, 2021 at 11:47 AM Chesnay Schepler  wrote:

> There's a related effort:
> https://issues.apache.org/jira/browse/FLINK-21108
>
> On 6/1/2021 10:14 AM, Till Rohrmann wrote:
> > Hi Gabor, welcome to the Flink community!
> >
> > Thanks for sharing this proposal with the community Márton. In general, I
> > agree that authentication is missing and that this is required for using
> > Flink within an enterprise. The thing I am wondering is whether this
> > feature strictly needs to be implemented inside of Flink or whether a
> proxy
> > setup could do the job? Have you considered this option? If yes, then it
> > would be good to list it under the point of rejected alternatives.
> >
> > I do see the benefit of implementing this feature inside of Flink if many
> > users need it. If not, then it might be easier for the project to not
> > increase the surface area since it makes the overall maintenance harder.
> >
> > Cheers,
> > Till
> >
> > On Mon, May 31, 2021 at 4:57 PM Márton Balassi 
> wrote:
> >
> >> Hi team,
> >>
> >> Firstly I would like to introduce Gabor or G [1] for short to the
> >> community, he is a Spark committer who has recently transitioned to the
> >> Flink Engineering team at Cloudera and is looking forward to
> contributing
> >> to Apache Flink. Previously G primarily focused on Spark Streaming and
> >> security.
> >>
> >> Based on requests from our customers G has implemented Kerberos and HTTP
> >> Basic Authentication for the Flink Dashboard and HistoryServer.
> Previously
> >> lacked an authentication story.
> >>
> >> We are looking to contribute this functionality back to the community,
> we
> >> believe that given Flink's maturity there should be a common code
> solution
> >> for this general pattern.
> >>
> >> We are looking forward to your feedback on G's design. [2]
> >>
> >> [1] http://gaborsomogyi.com/
> >> [2]
> >>
> >>
> https://docs.google.com/document/d/1NMPeJ9H0G49TGy3AzTVVJVKmYC0okwOtqLTSPnGqzHw/edit
> >>
>
>

[DISCUSS] Dashboard/HistoryServer authentication

[Márton Balassi]

Hi team,

Firstly I would like to introduce Gabor or G [1] for short to the
community, he is a Spark committer who has recently transitioned to the
Flink Engineering team at Cloudera and is looking forward to contributing
to Apache Flink. Previously G primarily focused on Spark Streaming and
security.

Based on requests from our customers G has implemented Kerberos and HTTP
Basic Authentication for the Flink Dashboard and HistoryServer. Previously
lacked an authentication story.

We are looking to contribute this functionality back to the community, we
believe that given Flink's maturity there should be a common code solution
for this general pattern.

We are looking forward to your feedback on G's design. [2]

[1] http://gaborsomogyi.com/
[2]
https://docs.google.com/document/d/1NMPeJ9H0G49TGy3AzTVVJVKmYC0okwOtqLTSPnGqzHw/edit

Re: [DISCUSS] Upgrade HBase connector to 2.2.x

[Márton Balassi]

Hi All,

I am also fairly torn on this one, however unless we are vigilant in
keeping the flink repository relatively lean the number of modules will
just keep increasing and pose an increasingly greater maintainability
challenge.
Less frequently used connectors are a strong candidate to be maintained in
bahir-flink and/or via flink-packages.org (I do not support creating a
third option in apache/flink-connectors). If the testing infrastructure of
bahir-flink is a concern, then we should invest into improving that, so
that it can serve as a reasonable alternative.

I prefer the option of HBase 2.x in Flink and 1.x in Bahir, with a
community commitment of improving the Bahir testing infra. If taking this
step immediately is deemed too risky I can accept having the two version
side-by-side in Flink for the time being, but without refactoring them to
use a common base module (like flink-kafka-connector-base) as we expect to
move 1.x to Bahir when the infra is satisfactory.

My position is not against HBase by any means, it is for a more
maintainable Flink repository. I have assigned [1] to Miklos, he aims at
opening a PR in the coming days - which we might modify based on the
outcome of this discussion.

[1] https://issues.apache.org/jira/browse/FLINK-18795

On Mon, Aug 10, 2020 at 4:16 PM Robert Metzger  wrote:

> @Jark: Thanks for bringing up these concerns.
> All the problems you've mentioned are "solvable":
> - uber jar: Bahir could provide a hbase1 uber jar (we could theoretically
> also add a dependency from flink to bahir and provide the uber jar from
> Flink)
> - e2e tests: we know that the connector is stable, as long as we are not
> adding major changes (or we are moving the respective e2e tests to bahir).
>
> On the other hand, I agree with you that supporting multiple versions of a
> connector is pretty common (see Kafka or elasticsearch), so why can't we
> allow it for Hbase now?
>
> I'm really torn on this and would like to hear more opinions on this.
>
>
> On Fri, Aug 7, 2020 at 11:24 PM Felipe Lolas  wrote:
>
>> Hi all!
>>
>> Im new here; I have been using the flink connector for hbase 1.2, but
>> recently opt to upgrading to hbase 2.1(basically because was bundled in
>> CDH6)
>>
>> it would be nice to add support for hbase 2.x!
>> I found that supporting hbase 1.4.3 and 2.1 needs minimal changes and
>> keeping that in mind last week I sent a PR with a solution supporting
>> 1.4.3/2.1.0 hbase (maybe not the best, im sorry if i break some rules
>> sending the PR).
>>
>> i would be happy to help if needed!
>>
>>
>>
>> Felipe.
>>
>> El 07-08-2020, a la(s) 10:53, Jark Wu  escribió:
>>
>> 
>> I'm +1 to add HBase 2.x
>>
>> However, I have some concerns about moving HBase 1.x to Bahir:
>> 1) As discussed above, there are still lots of people using HBase 1.x.
>> 2) Bahir doesn't have the infrastructure to run the existing HBase E2E
>> tests.
>> 3) We also paid lots of effort to provide an uber connector jar for HBase
>> (not yet released), it is helpful to improve the out-of-box experience.
>>
>> My thought is that adding HBase 2.x doesn't have to remove HBase 1.x. It
>> doesn't add too much work to maintain a new version.
>> Keeping the old version can also help us to develop the new one. I would
>> suggest to keep HBase 1.x in the repository for at least one more release.
>> Another idea is that maybe it's a good time to have a
>> "apache/flink-connectors" repository, and move both HBase 1.x and 2.x to
>> it.
>> It would also be a good place to accept the contribution of pulsar
>> connector and other connectors.
>>
>> Best,
>> Jark
>>
>>
>> On Fri, 7 Aug 2020 at 17:54, Robert Metzger  wrote:
>>
>>> Hi,
>>>
>>> Thank you for picking this up so quickly. I have no objections regarding
>>> all the proposed items.
>>> @Gyula: Once the bahir contribution is properly reviewed, ping me if you
>>> need somebody to merge it.
>>>
>>>
>>> On Fri, Aug 7, 2020 at 10:43 AM Márton Balassi >> >
>>> wrote:
>>>
>>> > Hi Robert and Gyula,
>>> >
>>> > Thanks for reviving this thread. We have the implementation (currently
>>> for
>>> > 2.2.3) and it is straightforward to contribute it back. Miklos (ccd)
>>> has
>>> > recently written a readme for said version, he would be interested in
>>> > contributing the upgraded connector back. The latest HBase version is
>>> > 2.3.0, if we are touching the codebase anyway I would propose 

Re: [DISCUSS] Upgrade HBase connector to 2.2.x

[Márton Balassi]

Hi Robert and Gyula,

Thanks for reviving this thread. We have the implementation (currently for
2.2.3) and it is straightforward to contribute it back. Miklos (ccd) has
recently written a readme for said version, he would be interested in
contributing the upgraded connector back. The latest HBase version is
2.3.0, if we are touching the codebase anyway I would propose to have that.

If everyone is comfortable with it I would assign [1] to Miklos with double
checking the all functionality that Felipe has proposed is included.
[1] https://issues.apache.org/jira/browse/FLINK-18795
[2] https://hbase.apache.org/downloads.html

On Fri, Aug 7, 2020 at 10:13 AM Gyula Fóra  wrote:

> Hi Robert,
>
> I completely agree with you on the Bahir based approach.
>
> I am happy to help with the contribution on the bahir side, with thorough
>  review and testing.
>
> Cheers,
> Gyula
>
> On Fri, 7 Aug 2020 at 09:30, Robert Metzger  wrote:
>
>> It seems that this thead is not on dev@ anymore. Adding it back ...
>>
>> On Fri, Aug 7, 2020 at 9:23 AM Robert Metzger 
>> wrote:
>>
>>> I would like to revive this discussion. There's a new JIRA[1] + PR[2]
>>> for adding HBase 2 support.
>>>
>>> it seems that there is demand for a HBase 2 connector, and consensus to
>>> do it.
>>>
>>> The remaining question in this thread seems to be the "how". I would
>>> propose to go the other way around as Gyula suggested: We move the legacy
>>> connector (1.4x) to bahir and add the new (2.x.x) to Flink.
>>> Why? In the Flink repo, we have a pretty solid testing infra, where we
>>> also run Hbase end to end tests. This will help us to stabilize the new
>>> connector and ensure a good quality.
>>> It also, the perception of what goes into Flink, and what into Bahir is
>>> a bit clearer if we put the stable, up to date stuff into Flink, and
>>> legacy, experimental or unstable connectors into Bahir.
>>>
>>>
>>> Who can take care of this effort? (Decide which Hbase 2 PR to take,
>>> review and contribution to Bahir)
>>>
>>>
>>> [1] https://issues.apache.org/jira/browse/FLINK-18795
>>> [2] https://github.com/apache/flink/pull/13047
>>>
>>> On Mon, Jun 22, 2020 at 3:32 PM Gyula Fóra  wrote:
>>>
 If we were to go the bahir route, I don't see the point in migrating
 the 1.4.x version there since that's already available in Flink. To me that
 is almost the same as dropping explicit support for 1.4 and telling users
 to use older connector versions if they wish to keep using it.

 If we want to keep 1.4 around for legacy users and slowly deprecate
 that, we can do that inside Flink and only push the 2.4.x version to bahir.

 What do you think?

 Gyula

 On Mon, Jun 22, 2020 at 3:16 PM Arvid Heise 
 wrote:

> If we support both HBase 1 and 2, maybe it's a good time to pull them
> out to Bahir and list them in flink-packages to avoid adding even more
> modules to Flink core?
>
> On Mon, Jun 22, 2020 at 4:05 AM OpenInx  wrote:
>
>> Hi
>>
>> According to my observation in the hbase community, there are still
>> lots of hbase users running their production cluster with version 1.x 
>> (1.4x
>> or 1.5.x). so I'd like to suggest that
>> supporting both hbase1.x & hbase2.x connector.
>>
>> Thanks.
>>
>> On Sat, Jun 20, 2020 at 2:41 PM Ming Li  wrote:
>>
>>> +1 to support both HBase 2.x and Hbase 1.4.x,  just as what we are
>>> doing for Kafka.
>>>
>>> On Fri, Jun 19, 2020 at 4:02 PM Yu Li  wrote:
>>>
 One supplement:

 I noticed that there are discussions in HBase ML this March about
 removing stable-1 pointer and got consensus [1], and will follow up in
 HBase community about why we didn't take real action. However, this 
 doesn't
 change my previous statement / stand due to the number of 1.x usages in
 production.

 Best Regards,
 Yu

 [1]
 http://mail-archives.apache.org/mod_mbox/hbase-dev/202003.mbox/%3c30180be2-bd93-d414-a158-16c9c8d01...@apache.org%3E

 On Fri, 19 Jun 2020 at 15:54, Yu Li  wrote:

> +1 on upgrading the HBase version of the connector, and 1.4.3 is
> indeed an old version.
>
> OTOH, AFAIK there're still quite some 1.x HBase clusters in
> production. We could also see that the HBase community is still 
> maintaining
> 1.x release lines (with "stable-1 release" point to 1.4.13) [1]
>
> Please also notice that HBase follows semantic versioning [2] [3]
> thus don't promise any kind of compatibility (source/binary/wire, 
> etc.)
> between major versions. So if we only maintain 2.x connector, it 
> would not
> be able to work with 1.x HBase clusters.
>
> I totally understand the additional efforts of maintaining two
> modules, but s

Re: [DISCUSS] FLIP-131: Consolidate the user-facing Dataflow SDKs/APIs (and deprecate the DataSet API)

[Márton Balassi]

Hi All,

Thanks for the write up and starting the discussion. I am in favor of
unifying the APIs the way described in the FLIP and deprecating the DataSet
API. I am looking forward to the detailed discussion of the changes
necessary.

Best,
Marton

On Wed, Jul 29, 2020 at 12:46 PM Aljoscha Krettek 
wrote:

> Hi Everyone,
>
> my colleagues (in cc) and I would like to propose this FLIP for
> discussion. In short, we want to reduce the number of APIs that we have
> by deprecating the DataSet API. This is a big step for Flink, that's why
> I'm also cross-posting this to the User Mailing List.
>
> FLIP-131: http://s.apache.org/FLIP-131
>
> I'm posting the introduction of the FLIP below but please refer to the
> document linked above for the full details:
>
> --
> Flink provides three main SDKs/APIs for writing Dataflow Programs: Table
> API/SQL, the DataStream API, and the DataSet API. We believe that this
> is one API too many and propose to deprecate the DataSet API in favor of
> the Table API/SQL and the DataStream API. Of course, this is easier said
> than done, so in the following, we will outline why we think that having
> too many APIs is detrimental to the project and community. We will then
> describe how we can enhance the Table API/SQL and the DataStream API to
> subsume the DataSet API's functionality.
>
> In this FLIP, we will not describe all the technical details of how the
> Table API/SQL and DataStream will be enhanced. The goal is to achieve
> consensus on the idea of deprecating the DataSet API. There will have to
> be follow-up FLIPs that describe the necessary changes for the APIs that
> we maintain.
> --
>
> Please let us know if you have any concerns or comments. Also, please
> keep discussion to this ML thread instead of commenting in the Wiki so
> that we can have a consistent view of the discussion.
>
> Best,
> Aljoscha
>

Re: [Discussion] Job generation / submission hooks & Atlas integration

[Márton Balassi]

Hi Jack,

Yes, we know how to do it and even have the implementation ready and being
reviewed by the Atlas community at the moment. :-)
Would you be interested in having a look?

On Thu, Mar 19, 2020 at 12:56 PM jackylau  wrote:

> Hi:
>   i think flink integrate atlas also need add catalog information such as
> spark atlas project
> .https://github.com/hortonworks-spark/spark-atlas-connector
>  when user use catalog such as JDBCCatalog/HiveCatalog, flink atlas project
> will sync this information to atlas.
>   But i don't find any Event Interface for flink to implement it as
> spark-atlas-connector does. Does anyone know how to do it
>
>
>
> --
> Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/
>