Re: [VOTE] Release 1.8.3, release candidate #3

[Fabian Paul]

Hi Hequn,

+1 (non-binding)

- verified checksums and hashes
- built from sources (Scala 2.11, Scala 2.12)
- build a custom docker image and run several test jobs on Kubernetes

Best,
Fabian Paul



--
Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/

[DISCUSS] Release Flink 1.15.3

[Fabian Paul]

Hi all,

I want to start the discussion of creating a new 1.15 patch release
(1.15.3). The last 1.15 release is almost two months old, and since then,
~60 tickets have been closed, targeting 1.15.3. It includes critical
changes to the sink architecture, including:

- Reverting the sink metric naming [1]
- Recovery problems for sinks using the GlobalCommitter [2][3][4]

If the community agrees to create a new patch release, I could volunteer as
the release manager.

Best,
Fabian

[1] https://issues.apache.org/jira/browse/FLINK-29567
[2] https://issues.apache.org/jira/browse/FLINK-29509
[3] https://issues.apache.org/jira/browse/FLINK-29512
[4] https://issues.apache.org/jira/browse/FLINK-29627

Re: [DISCUSS] Release Flink 1.15.3

[Fabian Paul]

Thanks for all the replies. @xintong I'll definitely come back to your
offer when facing steps that require PMC rights for the release.

I checked the JIRA and found four blocking/critical issues affecting 1.15.2

- FLINK-29830 <https://issues.apache.org/jira/browse/FLINK-29830>
- FLINK-29492 <https://issues.apache.org/jira/browse/FLINK-29492>
- FLINK-29315 <https://issues.apache.org/jira/browse/FLINK-29315>
- FLINK-29234 <https://issues.apache.org/jira/browse/FLINK-29234>

I'll reach out to the ticket owners to get their opinion about the current
status. In case, someone knows of some pending fixes that I haven't
mentioned please let me know.

Best,
Fabian

On Wed, Oct 26, 2022 at 2:01 PM Konstantin Knauf  wrote:

> +1, thanks Fabian.
>
> Am Mi., 26. Okt. 2022 um 08:26 Uhr schrieb Danny Cranmer <
> dannycran...@apache.org>:
>
> > +1, thanks for driving this Fabian.
> >
> > Danny,
> >
> > On Wed, Oct 26, 2022 at 2:22 AM yuxia 
> wrote:
> >
> > > Thanks for driving this.
> > > +1 for release 1.15.3
> > >
> > > Best regards,
> > > Yuxia
> > >
> > > - 原始邮件 -
> > > 发件人: "Leonard Xu" 
> > > 收件人: "dev" 
> > > 发送时间: 星期二, 2022年 10 月 25日 下午 10:00:47
> > > 主题: Re: [DISCUSS] Release Flink 1.15.3
> > >
> > > Thanks Fabian for driving this.
> > >
> > > +1 to release 1.15.3.
> > >
> > > The bug tickets FLINK-26394 and FLINK-27148 should be fixed as well,
> I’ll
> > > help to address them soon.
> > >
> > > Best,
> > > Leonard Xu
> > >
> > >
> > >
> > > > 2022年10月25日 下午8:28，Jing Ge  写道：
> > > >
> > > > +1 The timing is good to have 1.15.3 release. Thanks Fabian for
> > bringing
> > > > this to our attention.
> > > >
> > > > I just checked PRs and didn't find the 1.15 backport of FLINK-29567
> > > > <https://issues.apache.org/jira/browse/FLINK-29567>. Please be aware
> > of
> > > it.
> > > > Thanks!
> > > >
> > > > Best regards,
> > > > Jing
> > > >
> > > > On Tue, Oct 25, 2022 at 11:44 AM Xintong Song  >
> > > wrote:
> > > >
> > > >> Thanks for bringing this up, Fabian.
> > > >>
> > > >> +1 for creating a 1.15.3 release. I've also seen users requiring
> this
> > > >> version [1].
> > > >>
> > > >> I can help with actions that require a PMC role, if needed.
> > > >>
> > > >> Best,
> > > >>
> > > >> Xintong
> > > >>
> > > >>
> > > >> [1]
> https://lists.apache.org/thread/501q4l1c6gs8hwh433bw3v1y8fs9cw2n
> > > >>
> > > >>
> > > >>
> > > >> On Tue, Oct 25, 2022 at 5:11 PM Fabian Paul 
> wrote:
> > > >>
> > > >>> Hi all,
> > > >>>
> > > >>> I want to start the discussion of creating a new 1.15 patch release
> > > >>> (1.15.3). The last 1.15 release is almost two months old, and since
> > > then,
> > > >>> ~60 tickets have been closed, targeting 1.15.3. It includes
> critical
> > > >>> changes to the sink architecture, including:
> > > >>>
> > > >>> - Reverting the sink metric naming [1]
> > > >>> - Recovery problems for sinks using the GlobalCommitter [2][3][4]
> > > >>>
> > > >>> If the community agrees to create a new patch release, I could
> > > volunteer
> > > >> as
> > > >>> the release manager.
> > > >>>
> > > >>> Best,
> > > >>> Fabian
> > > >>>
> > > >>> [1] https://issues.apache.org/jira/browse/FLINK-29567
> > > >>> [2] https://issues.apache.org/jira/browse/FLINK-29509
> > > >>> [3] https://issues.apache.org/jira/browse/FLINK-29512
> > > >>> [4] https://issues.apache.org/jira/browse/FLINK-29627
> > > >>>
> > > >>
> > >
> >
>
>
> --
> https://twitter.com/snntrable
> https://github.com/knaufk
>

Re: [DISCUSS] Release Flink 1.15.3

[Fabian Paul]

I conclude that the community has accepted another release, and I will open
the voting thread shortly. Can someone with PMC rights add 1.15.4 as a new
release version in JIRA [1] so that I can update the still open tickets?

Best,
Fabian

[1]
https://issues.apache.org/jira/plugins/servlet/project-config/FLINK/versions

On Wed, Nov 2, 2022 at 2:07 PM Fabian Paul  wrote:

> Thanks for all the replies. @xintong I'll definitely come back to your
> offer when facing steps that require PMC rights for the release.
>
> I checked the JIRA and found four blocking/critical issues affecting 1.15.2
>
> - FLINK-29830 <https://issues.apache.org/jira/browse/FLINK-29830>
> - FLINK-29492 <https://issues.apache.org/jira/browse/FLINK-29492>
> - FLINK-29315 <https://issues.apache.org/jira/browse/FLINK-29315>
> - FLINK-29234 <https://issues.apache.org/jira/browse/FLINK-29234>
>
> I'll reach out to the ticket owners to get their opinion about the current
> status. In case, someone knows of some pending fixes that I haven't
> mentioned please let me know.
>
> Best,
> Fabian
>
> On Wed, Oct 26, 2022 at 2:01 PM Konstantin Knauf 
> wrote:
>
>> +1, thanks Fabian.
>>
>> Am Mi., 26. Okt. 2022 um 08:26 Uhr schrieb Danny Cranmer <
>> dannycran...@apache.org>:
>>
>> > +1, thanks for driving this Fabian.
>> >
>> > Danny,
>> >
>> > On Wed, Oct 26, 2022 at 2:22 AM yuxia 
>> wrote:
>> >
>> > > Thanks for driving this.
>> > > +1 for release 1.15.3
>> > >
>> > > Best regards,
>> > > Yuxia
>> > >
>> > > - 原始邮件 -
>> > > 发件人: "Leonard Xu" 
>> > > 收件人: "dev" 
>> > > 发送时间: 星期二, 2022年 10 月 25日 下午 10:00:47
>> > > 主题: Re: [DISCUSS] Release Flink 1.15.3
>> > >
>> > > Thanks Fabian for driving this.
>> > >
>> > > +1 to release 1.15.3.
>> > >
>> > > The bug tickets FLINK-26394 and FLINK-27148 should be fixed as well,
>> I’ll
>> > > help to address them soon.
>> > >
>> > > Best,
>> > > Leonard Xu
>> > >
>> > >
>> > >
>> > > > 2022年10月25日 下午8:28，Jing Ge  写道：
>> > > >
>> > > > +1 The timing is good to have 1.15.3 release. Thanks Fabian for
>> > bringing
>> > > > this to our attention.
>> > > >
>> > > > I just checked PRs and didn't find the 1.15 backport of FLINK-29567
>> > > > <https://issues.apache.org/jira/browse/FLINK-29567>. Please be
>> aware
>> > of
>> > > it.
>> > > > Thanks!
>> > > >
>> > > > Best regards,
>> > > > Jing
>> > > >
>> > > > On Tue, Oct 25, 2022 at 11:44 AM Xintong Song <
>> tonysong...@gmail.com>
>> > > wrote:
>> > > >
>> > > >> Thanks for bringing this up, Fabian.
>> > > >>
>> > > >> +1 for creating a 1.15.3 release. I've also seen users requiring
>> this
>> > > >> version [1].
>> > > >>
>> > > >> I can help with actions that require a PMC role, if needed.
>> > > >>
>> > > >> Best,
>> > > >>
>> > > >> Xintong
>> > > >>
>> > > >>
>> > > >> [1]
>> https://lists.apache.org/thread/501q4l1c6gs8hwh433bw3v1y8fs9cw2n
>> > > >>
>> > > >>
>> > > >>
>> > > >> On Tue, Oct 25, 2022 at 5:11 PM Fabian Paul 
>> wrote:
>> > > >>
>> > > >>> Hi all,
>> > > >>>
>> > > >>> I want to start the discussion of creating a new 1.15 patch
>> release
>> > > >>> (1.15.3). The last 1.15 release is almost two months old, and
>> since
>> > > then,
>> > > >>> ~60 tickets have been closed, targeting 1.15.3. It includes
>> critical
>> > > >>> changes to the sink architecture, including:
>> > > >>>
>> > > >>> - Reverting the sink metric naming [1]
>> > > >>> - Recovery problems for sinks using the GlobalCommitter [2][3][4]
>> > > >>>
>> > > >>> If the community agrees to create a new patch release, I could
>> > > volunteer
>> > > >> as
>> > > >>> the release manager.
>> > > >>>
>> > > >>> Best,
>> > > >>> Fabian
>> > > >>>
>> > > >>> [1] https://issues.apache.org/jira/browse/FLINK-29567
>> > > >>> [2] https://issues.apache.org/jira/browse/FLINK-29509
>> > > >>> [3] https://issues.apache.org/jira/browse/FLINK-29512
>> > > >>> [4] https://issues.apache.org/jira/browse/FLINK-29627
>> > > >>>
>> > > >>
>> > >
>> >
>>
>>
>> --
>> https://twitter.com/snntrable
>> https://github.com/knaufk
>>
>

[VOTE] Release 1.15.3, release candidate #1

[Fabian Paul]

Hi everyone, Please review and vote on the release candidate #1 for the
version 1.15.3, as follows: [ ] +1, Approve the release [ ] -1, Do not
approve the release (please provide specific comments) The complete staging
area is available for your review, which includes: - JIRA release notes
[1], - the official Apache source release and binary convenience releases
to be deployed to dist.apache.org [2], which are signed with the key with
fingerprint 90755B0A184BD9FFD22B6BE19D4F76C84EC11E37 [3], - all artifacts
to be deployed to the Maven Central Repository [4], - source code tag
"release-1.15.3-rc1" [5], - website pull request listing the new release
and adding announcement blog post [6]. The vote will be open for at least
72 hours. It is adopted by majority approval, with at least 3 PMC
affirmative votes.

Best, Fabian
[1]
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12352210

[2] https://dist.apache.org/repos/dist/dev/flink/flink-1.15.3-r
c1
[3] https://dist.apache.org/repos/dist/release/flink/KEYS [4]
https://repository.apache.org/content/repositories/orgapacheflink-1548

[5] https://github.com/apache/flink/tree/release-1.15.3-rc
1 [6]
https://github.com/apache/flink-web/pull/581

Re: [VOTE] Release 1.15.3, release candidate #1

[Fabian Paul]

Hi everyone,

I am still looking for volunteers to validate the release. I'll extend
the voting period by another 48hours, please try to give it some time.

Best,
Fabian


On Thu, Nov 10, 2022 at 5:18 PM Fabian Paul  wrote:
>
> Hi everyone, Please review and vote on the release candidate #1 for the 
> version 1.15.3, as follows: [ ] +1, Approve the release [ ] -1, Do not 
> approve the release (please provide specific comments) The complete staging 
> area is available for your review, which includes: - JIRA release notes [1], 
> - the official Apache source release and binary convenience releases to be 
> deployed to dist.apache.org [2], which are signed with the key with 
> fingerprint 90755B0A184BD9FFD22B6BE19D4F76C84EC11E37 [3], - all artifacts to 
> be deployed to the Maven Central Repository [4], - source code tag 
> "release-1.15.3-rc1" [5], - website pull request listing the new release and 
> adding announcement blog post [6]. The vote will be open for at least 72 
> hours. It is adopted by majority approval, with at least 3 PMC affirmative 
> votes.
>
> Best, Fabian
> [1] 
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12352210
>  [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.15.3-rc1
> [3] https://dist.apache.org/repos/dist/release/flink/KEYS [4] 
> https://repository.apache.org/content/repositories/orgapacheflink-1548 [5] 
> https://github.com/apache/flink/tree/release-1.15.3-rc1 [6] 
> https://github.com/apache/flink-web/pull/581

Re: [VOTE] Release 1.15.3, release candidate #1

[Fabian Paul]

Hi again,

Unfortunately, in the initial email, the links are not correctly
displayed, thus
please use the information below for testing.

Please review and vote on the release candidate #1 for the version 1.15.3,
as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)

The complete staging area is available for your review, which includes:

- JIRA release notes [1],
- the official Apache source release and binary convenience releases to
be deployed to dist.apache.org [2], which are signed with the key with
fingerprint 90755B0A184BD9FFD22B6BE19D4F76C84EC11E37 [3],
- all artifacts to be deployed to the Maven Central Repository [4],
- source code tag "release-1.15.3-rc1" [5],
- website pull request listing the new release and adding announcement
blog post [6].

Best,
Fabian

[1] 
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12352210
[2] https://dist.apache.org/repos/dist/dev/flink/flink-1.15.3-rc1
[3] https://dist.apache.org/repos/dist/release/flink/KEYS
[4] https://repository.apache.org/content/repositories/orgapacheflink-1548
[5] https://github.com/apache/flink/tree/release-1.15.3-rc1
[6] https://github.com/apache/flink-web/pull/581

On Mon, Nov 14, 2022 at 8:45 AM Fabian Paul  wrote:
>
> Hi everyone,
>
> I am still looking for volunteers to validate the release. I'll extend
> the voting period by another 48hours, please try to give it some time.
>
> Best,
> Fabian
>
>
> On Thu, Nov 10, 2022 at 5:18 PM Fabian Paul  wrote:
> >
> > Hi everyone, Please review and vote on the release candidate #1 for the 
> > version 1.15.3, as follows: [ ] +1, Approve the release [ ] -1, Do not 
> > approve the release (please provide specific comments) The complete staging 
> > area is available for your review, which includes: - JIRA release notes 
> > [1], - the official Apache source release and binary convenience releases 
> > to be deployed to dist.apache.org [2], which are signed with the key with 
> > fingerprint 90755B0A184BD9FFD22B6BE19D4F76C84EC11E37 [3], - all artifacts 
> > to be deployed to the Maven Central Repository [4], - source code tag 
> > "release-1.15.3-rc1" [5], - website pull request listing the new release 
> > and adding announcement blog post [6]. The vote will be open for at least 
> > 72 hours. It is adopted by majority approval, with at least 3 PMC 
> > affirmative votes.
> >
> > Best, Fabian
> > [1] 
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12352210
> >  [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.15.3-rc1
> > [3] https://dist.apache.org/repos/dist/release/flink/KEYS [4] 
> > https://repository.apache.org/content/repositories/orgapacheflink-1548 [5] 
> > https://github.com/apache/flink/tree/release-1.15.3-rc1 [6] 
> > https://github.com/apache/flink-web/pull/581

Re: [DISCUSS] Release Flink 1.15.3

[Fabian Paul]

Hi all,

The release vote for 1.15.3-rc1 is open [1]. Unfortunately, I am still
missing some votes
and would kindly ask for your help to make this release happen :)

Best,
Fabian

[1] https://lists.apache.org/thread/73l524189mpyrjokzxwb5smt80582pw1

On Thu, Nov 10, 2022 at 7:28 PM Martijn Visser  wrote:
>
> Hi Fabian,
>
> I've added 1.15.4 as a new release version.
>
> Thanks, Martijn
>
> On Thu, Nov 10, 2022 at 5:18 PM Fabian Paul 
>  wrote:
>>
>> I conclude that the community has accepted another release, and I will open
>> the voting thread shortly. Can someone with PMC rights add 1.15.4 as a new
>> release version in JIRA [1] so that I can update the still open tickets?
>>
>> Best,
>> Fabian
>>
>> [1]
>> https://issues.apache.org/jira/plugins/servlet/project-config/FLINK/versions
>>
>> On Wed, Nov 2, 2022 at 2:07 PM Fabian Paul  wrote:
>>
>> > Thanks for all the replies. @xintong I'll definitely come back to your
>> > offer when facing steps that require PMC rights for the release.
>> >
>> > I checked the JIRA and found four blocking/critical issues affecting 1.15.2
>> >
>> > - FLINK-29830 <https://issues.apache.org/jira/browse/FLINK-29830>
>> > - FLINK-29492 <https://issues.apache.org/jira/browse/FLINK-29492>
>> > - FLINK-29315 <https://issues.apache.org/jira/browse/FLINK-29315>
>> > - FLINK-29234 <https://issues.apache.org/jira/browse/FLINK-29234>
>> >
>> > I'll reach out to the ticket owners to get their opinion about the current
>> > status. In case, someone knows of some pending fixes that I haven't
>> > mentioned please let me know.
>> >
>> > Best,
>> > Fabian
>> >
>> > On Wed, Oct 26, 2022 at 2:01 PM Konstantin Knauf 
>> > wrote:
>> >
>> >> +1, thanks Fabian.
>> >>
>> >> Am Mi., 26. Okt. 2022 um 08:26 Uhr schrieb Danny Cranmer <
>> >> dannycran...@apache.org>:
>> >>
>> >> > +1, thanks for driving this Fabian.
>> >> >
>> >> > Danny,
>> >> >
>> >> > On Wed, Oct 26, 2022 at 2:22 AM yuxia 
>> >> wrote:
>> >> >
>> >> > > Thanks for driving this.
>> >> > > +1 for release 1.15.3
>> >> > >
>> >> > > Best regards,
>> >> > > Yuxia
>> >> > >
>> >> > > - 原始邮件 -
>> >> > > 发件人: "Leonard Xu" 
>> >> > > 收件人: "dev" 
>> >> > > 发送时间: 星期二, 2022年 10 月 25日 下午 10:00:47
>> >> > > 主题: Re: [DISCUSS] Release Flink 1.15.3
>> >> > >
>> >> > > Thanks Fabian for driving this.
>> >> > >
>> >> > > +1 to release 1.15.3.
>> >> > >
>> >> > > The bug tickets FLINK-26394 and FLINK-27148 should be fixed as well,
>> >> I’ll
>> >> > > help to address them soon.
>> >> > >
>> >> > > Best,
>> >> > > Leonard Xu
>> >> > >
>> >> > >
>> >> > >
>> >> > > > 2022年10月25日 下午8:28，Jing Ge  写道：
>> >> > > >
>> >> > > > +1 The timing is good to have 1.15.3 release. Thanks Fabian for
>> >> > bringing
>> >> > > > this to our attention.
>> >> > > >
>> >> > > > I just checked PRs and didn't find the 1.15 backport of FLINK-29567
>> >> > > > <https://issues.apache.org/jira/browse/FLINK-29567>. Please be
>> >> aware
>> >> > of
>> >> > > it.
>> >> > > > Thanks!
>> >> > > >
>> >> > > > Best regards,
>> >> > > > Jing
>> >> > > >
>> >> > > > On Tue, Oct 25, 2022 at 11:44 AM Xintong Song <
>> >> tonysong...@gmail.com>
>> >> > > wrote:
>> >> > > >
>> >> > > >> Thanks for bringing this up, Fabian.
>> >> > > >>
>> >> > > >> +1 for creating a 1.15.3 release. I've also seen users requiring
>> >> this
>> >> > > >> version [1].
>> >> > > >>
>> >> > > >> I can help with actions that require a PMC role, if needed.
>> >> > > >>
>> >> > > >> Best,
>> >> > > >>
>> >> > > >> Xintong
>> >> > > >>
>> >> > > >>
>> >> > > >> [1]
>> >> https://lists.apache.org/thread/501q4l1c6gs8hwh433bw3v1y8fs9cw2n
>> >> > > >>
>> >> > > >>
>> >> > > >>
>> >> > > >> On Tue, Oct 25, 2022 at 5:11 PM Fabian Paul 
>> >> wrote:
>> >> > > >>
>> >> > > >>> Hi all,
>> >> > > >>>
>> >> > > >>> I want to start the discussion of creating a new 1.15 patch
>> >> release
>> >> > > >>> (1.15.3). The last 1.15 release is almost two months old, and
>> >> since
>> >> > > then,
>> >> > > >>> ~60 tickets have been closed, targeting 1.15.3. It includes
>> >> critical
>> >> > > >>> changes to the sink architecture, including:
>> >> > > >>>
>> >> > > >>> - Reverting the sink metric naming [1]
>> >> > > >>> - Recovery problems for sinks using the GlobalCommitter [2][3][4]
>> >> > > >>>
>> >> > > >>> If the community agrees to create a new patch release, I could
>> >> > > volunteer
>> >> > > >> as
>> >> > > >>> the release manager.
>> >> > > >>>
>> >> > > >>> Best,
>> >> > > >>> Fabian
>> >> > > >>>
>> >> > > >>> [1] https://issues.apache.org/jira/browse/FLINK-29567
>> >> > > >>> [2] https://issues.apache.org/jira/browse/FLINK-29509
>> >> > > >>> [3] https://issues.apache.org/jira/browse/FLINK-29512
>> >> > > >>> [4] https://issues.apache.org/jira/browse/FLINK-29627
>> >> > > >>>
>> >> > > >>
>> >> > >
>> >> >
>> >>
>> >>
>> >> --
>> >> https://twitter.com/snntrable
>> >> https://github.com/knaufk
>> >>
>> >

Re: [DISCUSS] Release Flink 1.15.3

[Fabian Paul]

Thanks, everyone, for offering your help.

Best,
Fabian

On Wed, Nov 16, 2022 at 12:01 AM Danny Cranmer  wrote:
>
> Hey Fabian,
>
> I am out this week, I can take a look Monday if still required.
>
> Thanks,
> Danny
>
> On Tue, Nov 15, 2022 at 8:37 PM Martijn Visser 
> wrote:
>
> > Hi Fabian,
> >
> > I'll try to have a look tomorrow.
> >
> > Cheers, Martijn
> >
> > On Tue, Nov 15, 2022 at 6:44 PM Fabian Paul  wrote:
> >
> > > Hi all,
> > >
> > > The release vote for 1.15.3-rc1 is open [1]. Unfortunately, I am still
> > > missing some votes
> > > and would kindly ask for your help to make this release happen :)
> > >
> > > Best,
> > > Fabian
> > >
> > > [1] https://lists.apache.org/thread/73l524189mpyrjokzxwb5smt80582pw1
> > >
> > > On Thu, Nov 10, 2022 at 7:28 PM Martijn Visser  > >
> > > wrote:
> > > >
> > > > Hi Fabian,
> > > >
> > > > I've added 1.15.4 as a new release version.
> > > >
> > > > Thanks, Martijn
> > > >
> > > > On Thu, Nov 10, 2022 at 5:18 PM Fabian Paul <
> > fabian.p...@databricks.com.invalid>
> > > wrote:
> > > >>
> > > >> I conclude that the community has accepted another release, and I will
> > > open
> > > >> the voting thread shortly. Can someone with PMC rights add 1.15.4 as a
> > > new
> > > >> release version in JIRA [1] so that I can update the still open
> > tickets?
> > > >>
> > > >> Best,
> > > >> Fabian
> > > >>
> > > >> [1]
> > > >>
> > >
> > https://issues.apache.org/jira/plugins/servlet/project-config/FLINK/versions
> > > >>
> > > >> On Wed, Nov 2, 2022 at 2:07 PM Fabian Paul  wrote:
> > > >>
> > > >> > Thanks for all the replies. @xintong I'll definitely come back to
> > your
> > > >> > offer when facing steps that require PMC rights for the release.
> > > >> >
> > > >> > I checked the JIRA and found four blocking/critical issues affecting
> > > 1.15.2
> > > >> >
> > > >> > - FLINK-29830 <https://issues.apache.org/jira/browse/FLINK-29830>
> > > >> > - FLINK-29492 <https://issues.apache.org/jira/browse/FLINK-29492>
> > > >> > - FLINK-29315 <https://issues.apache.org/jira/browse/FLINK-29315>
> > > >> > - FLINK-29234 <https://issues.apache.org/jira/browse/FLINK-29234>
> > > >> >
> > > >> > I'll reach out to the ticket owners to get their opinion about the
> > > current
> > > >> > status. In case, someone knows of some pending fixes that I haven't
> > > >> > mentioned please let me know.
> > > >> >
> > > >> > Best,
> > > >> > Fabian
> > > >> >
> > > >> > On Wed, Oct 26, 2022 at 2:01 PM Konstantin Knauf  > >
> > > >> > wrote:
> > > >> >
> > > >> >> +1, thanks Fabian.
> > > >> >>
> > > >> >> Am Mi., 26. Okt. 2022 um 08:26 Uhr schrieb Danny Cranmer <
> > > >> >> dannycran...@apache.org>:
> > > >> >>
> > > >> >> > +1, thanks for driving this Fabian.
> > > >> >> >
> > > >> >> > Danny,
> > > >> >> >
> > > >> >> > On Wed, Oct 26, 2022 at 2:22 AM yuxia <
> > luoyu...@alumni.sjtu.edu.cn
> > > >
> > > >> >> wrote:
> > > >> >> >
> > > >> >> > > Thanks for driving this.
> > > >> >> > > +1 for release 1.15.3
> > > >> >> > >
> > > >> >> > > Best regards,
> > > >> >> > > Yuxia
> > > >> >> > >
> > > >> >> > > - 原始邮件 -
> > > >> >> > > 发件人: "Leonard Xu" 
> > > >> >> > > 收件人: "dev" 
> > > >> >> > > 发送时间: 星期二, 2022年 10 月 25日 下午 10:00:47
> > > >> >> > > 主题: Re: [DISCUSS] Release Flink 1.15.3
> > > >> >> > >
> > >

Re: [VOTE] Release 1.15.3, release candidate #1

[Fabian Paul]

Hi Sergey,

Sorry for the confusion, but the links in the initial mail are,
unfortunately, broken. I already sent out a new mail in this thread
with the new links. I've shared below the correct links again.

In general, we are still looking for PMC members who verify the
release. Please try to find some time.

Best,
Fabian

[1] 
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12352210
[2] https://dist.apache.org/repos/dist/dev/flink/flink-1.15.3-rc1
[3] https://dist.apache.org/repos/dist/release/flink/KEYS
[4] https://repository.apache.org/content/repositories/orgapacheflink-1548
[5] https://github.com/apache/flink/tree/release-1.15.3-rc1
[6] https://github.com/apache/flink-web/pull/581


On Mon, Nov 21, 2022 at 11:32 AM Sergey Nuyanzin  wrote:
>
> Maybe I'm doing something wrong...
> however when I'm trying to go to links mentioned in the first email I
> receive 404
> for these links
> https://dist.apache.org/repos/dist/dev/flink/flink-1.15.3-r
> <https://dist.apache.org/repos/dist/dev/flink/flink-1.15.2-rc2>c1
> <https://repository.apache.org/content/repositories/orgapacheflink-1524>
> https://github.com/apache/flink/tree/release-1.15.3-rc
>
> On Thu, Nov 17, 2022 at 5:26 PM Martijn Visser 
> wrote:
>
> > Hi Fabian,
> >
> > +1 (binding)
> >
> > - Validated hashes
> > - Verified signature
> > - Verified that no binaries exist in the source archive
> > - Build the source with Maven
> > - Verified licenses
> > - Verified web PR
> > - Started a cluster and the Flink SQL client, ran multiple jobs/statements
> >
> > On Wed, Nov 16, 2022 at 10:24 AM Dong Lin  wrote:
> >
> > > Thank you Fabian for the release.
> > >
> > > +1 (non-binding)
> > >
> > > - Verified that the source release can be built successfully.
> > > - Verified that the checksum and gpg files match the corresponding source
> > > release files, binary release files, and maven artifacts.
> > > - Verified that the source archives do not contain any binary file.
> > > - Verified that all POM files point to the same version.
> > > - Checked that the README.md file does not have anything unexpected.
> > > - Checked that the source code tag looks good.
> > >
> > > On Wed, Nov 16, 2022 at 1:40 AM Fabian Paul  wrote:
> > >
> > > > Hi again,
> > > >
> > > > Unfortunately, in the initial email, the links are not correctly
> > > > displayed, thus
> > > > please use the information below for testing.
> > > >
> > > > Please review and vote on the release candidate #1 for the version
> > > 1.15.3,
> > > > as follows:
> > > > [ ] +1, Approve the release
> > > > [ ] -1, Do not approve the release (please provide specific comments)
> > > >
> > > > The complete staging area is available for your review, which includes:
> > > >
> > > > - JIRA release notes [1],
> > > > - the official Apache source release and binary convenience releases to
> > > > be deployed to dist.apache.org [2], which are signed with the key with
> > > > fingerprint 90755B0A184BD9FFD22B6BE19D4F76C84EC11E37 [3],
> > > > - all artifacts to be deployed to the Maven Central Repository [4],
> > > > - source code tag "release-1.15.3-rc1" [5],
> > > > - website pull request listing the new release and adding announcement
> > > > blog post [6].
> > > >
> > > > Best,
> > > > Fabian
> > > >
> > > > [1]
> > > >
> > >
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12352210
> > > > [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.15.3-rc1
> > > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > [4]
> > > https://repository.apache.org/content/repositories/orgapacheflink-1548
> > > > [5] https://github.com/apache/flink/tree/release-1.15.3-rc1
> > > > [6] https://github.com/apache/flink-web/pull/581
> > > >
> > > > On Mon, Nov 14, 2022 at 8:45 AM Fabian Paul  wrote:
> > > > >
> > > > > Hi everyone,
> > > > >
> > > > > I am still looking for volunteers to validate the release. I'll
> > extend
> > > > > the voting period by another 48hours, please try to give it some
> > time.
> > > > >
> > > > > Best,
> > > > > Fabian
> > > > >
> > >

[RESULT][VOTE] Release 1.15.3, release candidate #1

[Fabian Paul]

Hi all,

@Yun sorry I forgot to post the result of the vote before continuing.
I only created the tag in the repository after I received the third
binding vote.

I'm happy to announce that we have unanimously approved this release.

There are 6 approving votes, 3 of which are binding:
* Danny
* Martijn
* Xintong

There are no disapproving votes.

Thanks everyone!

[ANNOUNCE] Apache Flink 1.15.3 released

[Fabian Paul]

The Apache Flink community is very happy to announce the release of
Apache Flink 1.15.3, which is the third bugfix release for the Apache
Flink 1.15 series.

Apache Flink® is an open-source stream processing framework for
distributed, high-performing, always-available, and accurate data
streaming applications.

The release is available for download at:
https://flink.apache.org/downloads.html

The full release notes are available in Jira:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12352210

We would like to thank all contributors of the Apache Flink community
who made this release possible!

Feel free to reach out to the release managers (or respond to this
thread) with feedback on the release process. Our goal is to
constantly improve the release process. Feedback on what could be
improved or things that didn't go so well are appreciated.

Regards,
Release Manager

Re: [jira] [Created] (FLINK-30238) Unified Sink committer does not clean up state on final savepoint

[Fabian Paul]

Hi folks,

I did some initial investigation, and the problem seems twofold.

If no post-commit topology is used, we do not run into a problem where
we could lose data but since we do not clean up the state correctly,
we will hit this [1] when trying to stop the pipeline with a savepoint
after we have started it from a savepoint.
AFAICT all two-phase commit sinks are affected Kafka, File etc.

For sinks using the post-commit topology, the same applies.
Additionally, we might never do the commit from the post-commit
topology resulting in lost data.

Best,
Fabian

[1] 
https://github.com/apache/flink/blob/ed46cb2fd64f1cb306ae5b7654d2b4d64ab69f22/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CheckpointCommittableManagerImpl.java#L83

Re: [jira] [Created] (FLINK-30238) Unified Sink committer does not clean up state on final savepoint

[Fabian Paul]

Yes, the StreamingFileSink is not affected.

Best,
Fabian

Re: [VOTE] Release flink-connector-kafka v3.2.0, release candidate #1

[Fabian Paul]

+1 (non-binding)

- Verified signature
- Verified checksum
- Built release tag from source with JDK 11
- approved docs PR

Best,
Fabian

On Wed, May 22, 2024 at 2:22 PM Leonard Xu  wrote:

> +1 (binding)
>
> - verified signatures
> - verified hashsums
> - built from source code with java 1.8 succeeded
> - checked Github release tag
> - reviewed the web PR
> - checked the CI result,
>   minor: the link [7] you post should be [1]
> - checked release notes,
>   minor: the issue FLINK-34961[2] should move to next version
>
>
> Best,
> Leonard
>
> [1]
> https://github.com/apache/flink-connector-kafka/actions/runs/8785158288
> [2] https://issues.apache.org/jira/browse/FLINK-34961
>
>
> > 2024年4月29日 上午12:34，Aleksandr Pilipenko  写道：
> >
> > +1 (non-binding)
> >
> > - Validated checksum
> > - Verified signature
> > - Checked that no binaries exist in the source archive
> > - Build source
> > - Verified web PR
> >
> > Thanks,
> > Aleksandr
> >
> > On Sun, 28 Apr 2024 at 11:35, Hang Ruan  wrote:
> >
> >> +1 (non-binding)
> >>
> >> - Validated checksum hash
> >> - Verified signature
> >> - Verified that no binaries exist in the source archive
> >> - Build the source with Maven and jdk8
> >> - Verified web PR
> >> - Check that the jar is built by jdk8
> >>
> >> Best,
> >> Hang
> >>
> >> Ahmed Hamdy  于2024年4月24日周三 17:21写道：
> >>
> >>> Thanks Danny,
> >>> +1 (non-binding)
> >>>
> >>> - Verified Checksums and hashes
> >>> - Verified Signatures
> >>> - Reviewed web PR
> >>> - github tag exists
> >>> - Build source
> >>>
> >>>
> >>> Best Regards
> >>> Ahmed Hamdy
> >>>
> >>>
> >>> On Tue, 23 Apr 2024 at 03:47, Muhammet Orazov
> >>> 
> >>> wrote:
> >>>
>  Thanks Danny, +1 (non-binding)
> 
>  - Checked 512 hash
>  - Checked gpg signature
>  - Reviewed pr
>  - Built the source with JDK 11 & 8
> 
>  Best,
>  Muhammet
> 
>  On 2024-04-22 13:55, Danny Cranmer wrote:
> > Hi everyone,
> >
> > Please review and vote on release candidate #1 for
> > flink-connector-kafka
> > v3.2.0, as follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> > This release supports Flink 1.18 and 1.19.
> >
> > The complete staging area is available for your review, which
> >> includes:
> > * JIRA release notes [1],
> > * the official Apache source release to be deployed to
> >> dist.apache.org
> > [2],
> > which are signed with the key with fingerprint 125FD8DB [3],
> > * all artifacts to be deployed to the Maven Central Repository [4],
> > * source code tag v3.2.0-rc1 [5],
> > * website pull request listing the new release [6].
> > * CI build of the tag [7].
> >
> > The vote will be open for at least 72 hours. It is adopted by
> >> majority
> > approval, with at least 3 PMC affirmative votes.
> >
> > Thanks,
> > Danny
> >
> > [1]
> >
> 
> >>>
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12354209
> > [2]
> >
> 
> >>>
> >>
> https://dist.apache.org/repos/dist/dev/flink/flink-connector-kafka-3.2.0-rc1
> > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > [4]
> >
> >> https://repository.apache.org/content/repositories/orgapacheflink-1723
> > [5]
> >
> >>>
> https://github.com/apache/flink-connector-kafka/releases/tag/v3.2.0-rc1
> > [6] https://github.com/apache/flink-web/pull/738
> > [7] https://github.com/apache/flink-connector-kafka
> 
> >>>
> >>
>
>

Re: [DISCUSSION] FLIP-456: CompiledPlan support for Batch Execution Mode

[Fabian Paul]

Thanks, Alexey, for the proposal. I think this is a nice addition that
finally fixes the gap in the CompiledPlan. +1

Best,
Fabian

On Tue, May 14, 2024 at 1:19 AM Alexey Leonov-Vendrovskiy <
vendrov...@gmail.com> wrote:

> Thanks Jim.
>
>
>
> > 1. For the testing, I'd call the tests "execution" tests rather than
> > "restore" tests.  For streaming execution, restore tests have the
> compiled
> > plan and intermediate state; the tests verify that those can work
> together
> > and continue processing.
>
>
> Agree that we don't need to store and restore the intermediate state. So
> the most critical part is that the CompiledPlan for batch can be executed.
>
> 2. The FLIP implicitly suggests "completeness tests" (to use FLIP-190's
> > words).  Do we need "change detection tests"?  I'm a little unsure if
> that
> > is presently happening in an automatic way for streaming operators.
>
>
>  We might need to elaborate more on this, but the idea is that  we need to
> make sure that compiled plans created by an older version of SQL Planner
> are executable on newer runtimes.
>
> 3.  Can we remove old versions of batch operators eventually?  Or do we
> > need to keep them forever like we would for streaming operators?
> >
>
> We could have deprecation paths for old operator nodes in some cases. It is
> a matter of the time window: what could be practical the "time distance"
> between query planner and flink runtime against which the query query can
> be resubmitted.
> Note, here we don't have continuous queries, so there is always an option
> to "re-plan" the original SQL query text into a newer version of the
> CompiledPlan.
> With this in mind, a time window of 1yr+ would allow deprecation of older
> batch exec nodes, though I don't see this as a frequent event.
>
> -Alexey
>
>
>
> On Mon, May 13, 2024 at 1:52 PM Jim Hughes 
> wrote:
>
> > Hi Alexey,
> >
> > After some thought, I have a question about deprecations:
> >
> > 3.  Can we remove old versions of batch operators eventually?  Or do we
> > need to keep them forever like we would for streaming operators?
> >
> > Cheers,
> >
> > Jim
> >
> > On Thu, May 9, 2024 at 11:29 AM Jim Hughes  wrote:
> >
> > > Hi Alexey,
> > >
> > > Overall, the FLIP looks good and makes sense to me.
> > >
> > > 1. For the testing, I'd call the tests "execution" tests rather than
> > > "restore" tests.  For streaming execution, restore tests have the
> > compiled
> > > plan and intermediate state; the tests verify that those can work
> > together
> > > and continue processing.
> > >
> > > For batch execution, I think we just want that all existing compiled
> > plans
> > > can be executed in future versions.
> > >
> > > 2. The FLIP implicitly suggests "completeness tests" (to use FLIP-190's
> > > words).  Do we need "change detection tests"?  I'm a little unsure if
> > that
> > > is presently happening in an automatic way for streaming operators.
> > >
> > > In RestoreTestBase, generateTestSetupFiles is disabled and has to be
> run
> > > manually when tests are being written.
> > >
> > > Cheers,
> > >
> > > Jim
> > >
> > > On Tue, May 7, 2024 at 5:11 AM Paul Lam  wrote:
> > >
> > >> Hi Alexey,
> > >>
> > >> Thanks a lot for bringing up the discussion. I’m big +1 for the FLIP.
> > >>
> > >> I suppose the goal doesn’t involve the interchangeability of json
> plans
> > >> between batch mode and streaming mode, right?
> > >> In other words, a json plan compiled in a batch program can’t be run
> in
> > >> streaming mode without a migration (which is not yet supported).
> > >>
> > >> Best,
> > >> Paul Lam
> > >>
> > >> > 2024年5月7日 14:38，Alexey Leonov-Vendrovskiy 
> 写道：
> > >> >
> > >> > Hi everyone,
> > >> >
> > >> > PTAL at the proposed FLIP-456: CompiledPlan support for Batch
> > Execution
> > >> > Mode. It is pretty self-describing.
> > >> >
> > >> > Any thoughts are welcome!
> > >> >
> > >> > Thanks,
> > >> > Alexey
> > >> >
> > >> > [1]
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-456%3A+CompiledPlan+support+for+Batch+Execution+Mode
> > >> > .
> > >>
> > >>
> >
>

Re: Question about v2 committer guarantees

[Fabian Paul]

Hi Scott,

It's great to see further adoption of the Sink V2 architecture.
Happy to answer your questions.

1. The sink architecture should ensure that Committer:commit is always
called with all committables from a subtask for a given subtaskId.There is
an open issue where users have reported a problem with that assumption [1]
but we haven't been able to track the problem down. Behind the scenes, if
you use a SinkWriter->Committer topology, the Committables are transferred
via the checkpoint barrier channel to the committer (not transferring the
committables results in failing the checkpoint) and checkpointed by the
committer before Committer:commit is called. This means when
Committer:commit is called, it reads the comittables from the local state.

2. Committer::commit is called on what we call in Flink
notifyCheckpointComplete which is based on a RPC call that the Jobmanager
makes to all Taskmanagers when a checkpoint is finished. There is no
guarantee when or if this will be called, but eventually. If some of the
RPCs are delayed or do not reach the manager, the Committer will accumulate
committables from multiple checkpoints.

3. I am not sure I fully understand that point. I see two different
requirements. First, you could skip a committable if you do not want to
commit it, which you could do with calling
CommitRequest::signalAlreadyCommitted [2]. It's not the primary purpose of
the method, but it should suffice. The second point is having a
communication mechanism between SinkWriter and Committer, which at the
moment does not exist. I would love to hear more details about why the
rewrite is necessary maybe we can model the sink differently to achieve
that requirement.

Can you explain more about the relation between your questions and using
the SupportsPreCommitTopology::addPreCommitTopology? It sounds like you are
not planning to use the Committer but a custom globalCommitter.

Best,
Fabian

[1] https://issues.apache.org/jira/browse/FLINK-25920
[2]
https://github.com/apache/flink/blob/7fc3aac774f5deb9b48727ba5f916c78085b49b9/flink-core/src/main/java/org/apache/flink/api/connector/sink2/Committer.java#L100

Re: Potential Kafka Connector FLIP: Large Message Handling

[Fabian Paul]

Hi Kevin,

I worked on a project [1] in the past that had a similar purpose. You
should be able to use a similar approach with the existing KafkaSource by
implementing your own KafkaRecordDeserializationSchema that hides the logic
of pulling the records from blob storage from the connector. You can even
use the linked project directly with the KafkaSource using [2] and [3].

I agree there is room for improvements, like propagating Flink's Filesystem
credentials to the custom deserializer, but the overall idea seems to
require only very few changes to Flink.

Best,
Fabian

[1] https://github.com/bakdata/kafka-large-message-serde
[2]
https://github.com/apache/flink-connector-kafka/blob/15d3fbd4e65dae6c334e2386dd337d2bf423c216/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/reader/deserializer/KafkaRecordDeserializationSchema.java#L107
[3]
https://github.com/bakdata/kafka-large-message-serde/blob/09eae933afaf8a1970b1b1bebcdffe934c368cb9/large-message-serde/src/main/java/com/bakdata/kafka/LargeMessageDeserializer.java#L50

On Mon, Jul 8, 2024 at 3:49 PM Kevin Lam 
wrote:

> Hi all,
>
> Thanks for the responses.
>
> Grace those are indeed both challenges, thanks for flagging them. Regarding
> expiry, we could consider having a Mark and Sweep garbage collection
> system. A service can consume the topics with large messages, and track
> references. When there are no references left for large messages, they can
> be removed.
>
> Martjin, I will take a look at if there's any prior discussions in the
> Kafka community and send the proposal to the Kafka Dev mailing list if it
> makes sense :). It'd be much preferred if this was natively supported by
> Kafka, since it's not currently I was also exploring making this work in
> Flink.
>
>
>
> On Mon, Jul 8, 2024 at 3:23 AM Martijn Visser 
> wrote:
>
> > Hi Kevin,
> >
> > I just want to double check, were you planning to send this proposal to
> the
> > Kafka Dev mailing list? Because I don't see directly how this affects
> Flink
> > :)
> >
> > Best regards,
> >
> > Martijn
> >
> > On Mon, Jul 8, 2024 at 8:05 AM Grace Grimwood 
> wrote:
> >
> > > Hi Kevin,
> > >
> > > Thanks for starting this thread.
> > >
> > > This idea is something that was discussed in Kroxylicious (an open
> source
> > > Kafka proxy, I'm a maintainer there). In that discussion [1] we came to
> > the
> > > conclusion that there are a couple of issues with implementing this:
> > > 1. Doesn't scale - very large messages (>1GiB) or large batch sizes
> could
> > > cause extreme memory bloat in clients, as the entire thing would need
> to
> > be
> > > fed into the producer which could very quickly fill its buffers.
> > Depending
> > > on how the subsequent deserialization and payload fetch is handled at
> the
> > > consumer end, it's likely that the same behaviour would also be seen
> > there.
> > > 2. Difficult to sync expiry - when Kafka deletes messages due to
> > retention
> > > (or topic compaction), it does so without notifying clients. There is
> no
> > > (easy) way to ensure the associated payload is deleted from object
> > storage
> > > at the same time.
> > >
> > > It's not totally clear how Conduktor solved these issues, but IMO they
> > are
> > > worth keeping in mind. For Kroxylicious we decided these problems meant
> > it
> > > wasn't practical for us to implement this, but I'd be curious to know
> if
> > > you've got any ideas :)
> > >
> > > Regards,
> > > Grace
> > >
> > > [1] https://github.com/kroxylicious/kroxylicious/discussions/1244
> > >
> > > On Sat, Jul 6, 2024 at 8:21 AM Kevin Lam  >
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > Writing to see if the community would be open to exploring a FLIP for
> > the
> > > > Kafka Table Connectors. The FLIP would allow for storing Kafka
> Messages
> > > > beyond a Kafka cluster's message limit (1 MB by default) out of band
> in
> > > > cloud object storage or another backend.
> > > >
> > > > During serialization the message would be replaced with a reference,
> > and
> > > > during deserialization the reference would be used to fetch the large
> > > > message and pass it to Flink. Something like Option 1 in this blog
> post
> > > > <
> > > >
> > >
> >
> https://www.conduktor.io/kafka/how-to-send-large-messages-in-apache-kafka/#Option-1:-using-an-external-store-(GB-size-messages)-0
> > > > >
> > > > .
> > > >
> > > > What do you think?
> > > >
> > > > We can make it generic by allowing users to implement their own
> > > > LargeMessageSerializer/Deserializer interface for serializing and
> > > > deserializing and handling interactions with object storage or some
> > other
> > > > backend.
> > > >
> > > > The Kafka Connectors can be extended to support ConfigOptions to
> > > > specify the class to load, as well as some user-specified properties.
> > For
> > > > example: `large-record-handling.class` and `
> > > > large-record-handling.properties.*` (where the user can specify any
> > > > properties similar to how 

Re: [DISCUSS] FLIP-468: Introducing StreamGraph-Based Job Submission.

[Fabian Paul]

Thanks for drafting this FLIP. I really like the idea of introducing a
concept in Flink that is close to a logical plan submission.

I have a few questions about the proposal and its future evolvability.

- What is the future plan for job submissions in Flink? With the current
proposal, Flink will support JobGraph/StreamGraph/compiled plan
submissions? It might be confusing for users and complicate the existing
job submission logic significantly.
- The FLIP mentions multiple areas of optimization, first operator chaining
and second dynamic switches between join strategies. I think from a Flink
perspective, these are, at the moment, separate concerns.  For operator
chaining, I agree with the current proposal, which is a concept that
applies generally to Flink's runtime. For join strategies, they are only
applicable when using an optimizer (that's currently not part of Flink's
runtime) with the Table API or Flink SQL. How do we plan to connect the
optimizer with Flink's runtime?
- With table/SQL API we already expose a compiled plan to support stable
version upgrades. It would be great to explore a joined plan to also offer
stable version upgrades with a potentially persistent streamgraph.

Best,
Fabian

[DISCUSS] Retrieve savepoint location after suspension of jobclusters

[Fabian Paul]

Hi all,

Due to recent changes in the shutdown mechanism of Flink [1] it is not 
conveniently possible anymore to suspend a job running on a jobcluster 
with a savepoint and retrieve the savepoint location via the Flink API 
programmatically.

With the introduced changes the rest endpoint shutdowns immediately 
and rejects new request which makes the information inaccessible.

Before the changes it was possible to stop the job and query the savepoint 
info endpoint until the location was shown.
Admittedly, this was never a safe solution because it expected that the 
rest endpoint stays alive long enough.

I would like to see what the community thinks about this and whether it is 
worth to implement a different solution to retrieve those information.

Best,
Fabian
[1] https://issues.apache.org/jira/browse/FLINK-18663

Re: [DISCUSS] Retrieve savepoint location after suspension of jobclusters

[Fabian Paul]

Hi Till,

The problem is reproducible with a basic shell script doing the following 
operations.

1. Post request to /jobs/${JOB_ID}/savepoints with the payload
 {"cancel-job": true,"target-directory": $(LOCATION)}
and store the trigger ID

2. Sleep 10 seconds

3. Get jobs/${JOB_ID}/savepoints/$(TRIGGER_ID)
results in a connect exception because rest endpoint is shutdown.

Sorry, if I misunderstood you previous answer but I would expect that stopping 
the job 
with a savepoint is an asynchronous operation and should block the shutdown 
until 
the result is served.
I also can confirm that the cluster is not shutdown but the rest endpoint is 
which makes 
it impossible to serve the asynchronous result.

Best,
Fabian

Re: [DISCUSS] Retrieve savepoint location after suspension of jobclusters

[Fabian Paul]

I attached the last log lines[1] of the jobmanager after triggering the 
savepoint. I just
saw the release for 1.10.2 is started so it would probably be great if we 
determine 
whether it is a bug to postpone the release if necessary.
What do you think?

Best,
Fabian

[1] https://pastebin.com/eWXN5fzS
 

Re: [VOTE] Release 1.11.2, release candidate #1

[Fabian Paul]

+1 (non-binding)

Checks:

- Verified signature
- Built from source (Java8)
- Ran custom jobs on Kubernetes

Regards,
Fabian

Re: [VOTE] Adopt jemalloc as default memory allocator in docker image

[Fabian Paul]

+1 (non-binding)

Thanks, Yun for the efforts to bring this topic to a vote.

Best,
Fabian

[DISCUSS] Programmatically submit Flink job jar to session cluster

[Fabian Paul]

Hi all,

Currently, the most convenient way of programmatically submitting a job to a 
running session cluster is using Flink’s RestClusterClient.
Unfortunately, it is only supported, as of now, to submit a job graph.[1] To 
construct a job graph from a jar file, additional Flink dependencies are 
required, which is not ideal.

It is also possible to directly use the Flink rest API and first upload the jar 
file via /jars/upload[2] and then run it via /jar/:jarId/run[3]. It has the 
downside that it is impossible to set a Flink execution configuration, and it 
will always take the underlying session cluster configuration. 

I know changing the ClusterClient has already been discussed. It would involve 
a lot of effort, so what do you think of making the jar execution more 
prominent via the rest endpoint by adding the option to pass an execution 
configuration?

Best,
Fabian

[1] 
https://github.com/apache/flink/blob/65cd385d7de504a946b17193aceea73b3c8e78a8/flink-clients/src/main/java/org/apache/flink/client/program/ClusterClient.java#L95
[2] 
https://github.com/apache/flink/blob/c2972b6e336cc3b3a6cbd22c69a6710dab5246e6/flink-container/src/main/java/org/apache/flink/container/entrypoint/StandaloneApplicationClusterConfigurationParserFactory.java#L56
 

[3] 
https://github.com/apache/flink/blob/c2972b6e336cc3b3a6cbd22c69a6710dab5246e6/flink-container/src/main/java/org/apache/flink/container/entrypoint/StandaloneApplicationClusterConfigurationParserFactory.java#L56

Re: [DISCUSS] Drop Mesos in 1.14

[Fabian Paul]

+ 1 for dropping mesos. Most of the PMCs have already left the project [1] and 
a move to attic
was barely avoided. Overall kubernetes has taken its place and it is unlikely 
that we will see a 
surge in Mesos very soon.

Best,
Fabian


[1] 
https://lists.apache.org/thread.html/rab2a820507f7c846e54a847398ab20f47698ec5bce0c8e182bfe51ba%40%3Cdev.mesos.apache.org%3E

Re: [Flink blogs]

[Fabian Paul]

Hi Etienne,

Thanks for reaching out I think your list already looks very appealing.

> * - metrics (https://github.com/apache/flink/pull/14510): it was
>   dealing with delimiters. I think it is a bit low level for a blog post ?
> *

I am also unsure whether this a good fit to present. I can only imagine showing 
what kind of use-case it supports.


> 
> * - migration of pipelines from DataSet API to DataStream API: it is
>   already discussed in the flink website
> *

This is definitely something I’d like to see in my opinion it can also become a 
series because the topic has a lot of aspects. If you want to write a 
post about it it would be great to show the migration of a more complex 
pipeline (i.e. old formats, incompatible types ….). Many users will 
eventually face this so it has a big impact. FYI probably only Flink 1.13 is 
the latest version with full DataSet support.

> 
> * - accumulators (https://github.com/apache/flink/pull/14558): it was
>   about an asynchronous get, once again a bit too low level for a blog
>   post ?
> *

To me accumulator are a kind of internal concept but maybe you can provide the 
use-case which drove this change? Probably explaining the 
semantics of them is already complicated.


> 
> * - FileInputFormat mainly parquet improvements and fixes
>   (https://github.com/apache/flink/pull/15725,
>   https://github.com/apache/flink/pull/15172,
>   https://github.com/apache/flink/pull/15156): interesting but as this
>   API is being decommissioned, it might not be a good subject ?
> *

You have already summarized it: it is being deprecated and a much more 
interesting topic is the migration from DataSet to the DataStream API in 
case these old formats are used.


> 
> * - doing a manual join in DataStream API in batch mode with
>   
> /KeyedCoProcessFunction///(https://issues.apache.org/jira/browse/FLINK-22587).
>   As the target is more Flink table/SQL for these kind of things, the
>   same deprecation comment as above applies.
> *
> 

I tend to not show this topic because my recommendation would be to use the 
Table API directly and not build your own join in the DataStream API ;)

> => maybe a blog post on back pressure in checkpointing 
> (https://github.com/apache/flink/pull/13040). WDYT ?
> 

This is also an interesting topic but we constantly work on improving the 
situation and I am unsure if the blogpost is already not up-to-date anymore 
when it is released. 


Please let me know what you think I am also happy to give more feedback for one 
of the topics in more detail if you need it.

Best,
Fabian

[DISCUSS] FLIP-191: Extend unified Sink interface to support small file compaction

[Fabian Paul]

Hi all,

More and more data lake sinks rely on columnar formats which benefit from few 
larger files than a lot of small files (read amplification). 
Our current FileSink cannot ensure a certain size when writing to an external 
filesystem which I call the small file compaction 
problem. Unfortunately, there is no good way with the current unified Sink 
operator topology to support this use case.

I would like to propose to extend the unified Sink interface which we proposed 
in FLIP-143 to resolve the small file compaction problem.
Therefore I have created FLIP-191 [1] to outline three different options how 
the problem could be addressed.

1. Global Sink Coordinator
2. Committable Aggregator Operator
3. Custom sink topology

Further information about the alternatives can be found in the document and I 
would appreciate your feedback to decide on which way to go to 
finally resolve this problem.

Best,
Fabian

[1] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-191%3A+Extend+unified+Sink+interface+to+support+small+file+compaction

Re: [DISCUSS] FLIP-191: Extend unified Sink interface to support small file compaction

[Fabian Paul]

Hi David and Till,

Thanks for your great feedback. One definitely confusing point in the FLIP is 
who is doing the actual compaction. The compaction will not be done 
by the CommittableAggregator operator but the committers so it should also not 
affect the checkpointing duration or have a significant performance 
bottleneck because the committers are executed in parallel (also in batch mode 
[1]). 

I will update the FLIP to clarify it.

> From your description I would be in favour of option 2 for the following
> reasons: Assuming that option 2 solves all our current problems, it seems
> like the least invasive change and smallest in scope. Your main concern is
> that it might not cover future use cases. Do you have some specific use
> cases in mind?

No, I do not have anything specific in mind I just wanted to raise the point 
that adding more and more operators to the sink might complicate the 
development in the future that they can all be used together.

> What I am missing a bit
> from the description is how option 2 will behave wrt checkpoints and the
> batch execution mode.

My idea was to always invoke CommittableAggregate#aggregate on a checkpoint and 
endOfInput. In the batch case the aggregation is only done 
once on all committables.


> Few thoughts on the option 2)
> 
> The file compaction is by definition quite costly IO bound operation. If I
> understand the proposal correctly, the aggregation itself would run during
> operator (aggregator) checkpoint. Would this significantly increase the
> checkpoint duration?
> 
> Compaction between different sub-tasks incur additional network IO (to
> fetch the raw non-compacted files from the remote filesystem), so this
> could quickly become a bottleneck. Basically we're decreasing the sink
> parallelism (possible throughput) to parallelism of the aggregator.

Hopefully these concerns are covered by the explanation at the beginning.

> To be really effective here, compaction would ideally be able to compact
> files from multiple checkpoints. However there is a huge tradeoff between
> latency a efficiency (especially with exactly once). Is this something
> worth exploring?

I agree with you by enabling the compaction across checkpoint the latency can 
increase because files might be committed several checkpoints 
later. I guess the best we can do is to let the user configure the behaviour. 
By configuring the checkpointing interval and the wanted file size the 
user can already affect the latency.
Is this answering you questions? I am not fully sure what you are referring to 
with efficiency. @dvmk

> I hope that with option 2, we can support both use cases: single task
compaction as well as cross task compaction if needed. Similarly for single
checkpoint compaction as well as cross checkpoint compaction.

Compaction across subtasks should be controllable by the parallelism of the 
commttableAggregator operator i.e. a parallelism of 2 can reduce 
the computational complexity but might not compute the best compaction.

Best,
Fabian

[1] https://github.com/apache/flink/pull/17536 
)

Re: [DISCUSS] FLIP-191: Extend unified Sink interface to support small file compaction

[Fabian Paul]

Hi all,

Thanks for the lively discussions. I am really excited to see so many people
participating in this thread. It also underlines the need that many people would
like to see a solution soon.

I have updated the FLIP and removed the parallelism configuration because it is
unnecessary since users can configure a constant exchange key to send all
committables to only one committable aggregator.


1. Burden for developers w.r.t batch stream unification.

@yun @guowei, from a theoretical point you are right about exposing the 
DataStream
API in the sink users have the full power to write correct batch and streaming
sinks. I think in reality a lot of users still struggle to build pipelines with
i.e. the operator pipeline which works correct in streaming and batch mode.
Another problem I see is by exposing more deeper concepts is that we cannot do
any optimization because we cannot reason about how sinks are built in the
future.

We should also try to steer users towards using only `Functions` to give us more
flexibility to swap the internal operator representation. I agree with @yun we
should try to make the `ProcessFunction` more versatile to work on that goal but
I see this as unrelated to the FLIP.


2. Regarding Commit / Global commit

I envision the global committer to be specific depending on the data lake
solution you want to write to. However, it is entirely orthogonal to the 
compaction.
Currently, I do not expect any changes w.r.t the Global commit introduces by
this FLIP.


3. Regarding the case of trans-checkpoints merging

@yun, as user, I would expect that if the committer receives in a checkpoint 
files
to merge/commit that these are also finished when the checkpoint finishes.
I think all sinks rely on this principle currently i.e., KafkaSink needs to
commit all open transactions until the next checkpoint can happen.

Maybe in the future, we can somehow move the Committer#commit call to an
asynchronous execution, but we should discuss it as a separate thread.

> We probably should first describe the different causes of small files and
> what problems was this proposal trying to solve. I wrote a data shuffling
> proposal [1] for Flink Iceberg sink (shared with Iceberg community [2]). It
> can address small files problems due to skewed data distribution across
> Iceberg table partitions. Streaming shuffling before writers (to files) is
> typically more efficient than post-write file compaction (which involves
> read-merge-write). It is usually cheaper to prevent a problem (small files)
> than fixing it.


@steven you are raising a good point, although I think only using a customizable
shuffle won't address the generation of small files. One assumption is that
at least the sink generates one file per subtask, which can already be too many.
Another problem is that with low checkpointing intervals, the files do not meet
the required size. The latter point is probably addressable by changing the
checkpoint interval, which might be inconvenient for some users.

> The sink coordinator checkpoint problem (mentioned in option 1) would be
> great if Flink can address it. In the spirit of source (enumerator-reader)
> and sink (writer-coordinator) duality, sink coordinator checkpoint should
> happen after the writer operator. This would be a natural fit to support
> global committer in FLIP-143. It is probably an orthogonal matter to this
> proposal.


To me the question here is what are the benefits of having a coordinator in
comparison to a global committer/aggregator operator.

> Personally, I am usually in favor of keeping streaming ingestion (to data
> lake) relatively simple and stable. Also sometimes compaction and sorting
> are performed together in data rewrite maintenance jobs to improve read
> performance. In that case, the value of compacting (in Flink streaming
> ingestion) diminishes.


I agree it is always possible to have scheduled maintenance jobs keeping care of
your data i.e., doing compaction. Unfortunately, the downside is that you
have to your data after it is already available for other downstream consumers.
I guess this can lead to all kinds of visibility problems. I am also surprised 
that
you personally are a fan of this approach and, on the other hand, are developing
the Iceberg sink, which goes somewhat against your mentioned principle of 
keeping
the sink simple.

> Currently, it is unclear from the doc and this thread where the compaction
> is actually happening. Jingsong's reply described one model
> writer (parallel) -> aggregator (single-parallelism compaction planner) ->
> compactor (parallel) -> global committer (single-parallelism)


My idea of the topology is very similar to the one outlined by Jinsong. The
compaction will happen in the committer operator.

> 
> In the Iceberg community, the following model has been discussed. It is
> better for Iceberg because it won't delay the data availability.
> writer (parallel) -> global committer for append (single parallelism)

Re: [NOTICE] Please keep flink-examples up to date

[Fabian Paul]

Hi Seth,

Thanks for brining up this topic.

Huge appreciations that you take this over initially and we should definitely
take care as a community to what we show beginner users.

We can also take the examples as show cases about things we have developed and
are proud of.

Best,
Fabian

Re: [DISCUSS] Update Policy for old releases

[Fabian Paul]

Thanks for bringing up this topic Piotr.
I also think we should try to decouple our release cycles from our support
plans. Currently we are very limited by the approach because faster release
cycles result in also faster deprecation of versions.


Therefore I am also favoring version 2 where we can align the next LTS version
with our development speed. Option 1 I think can easily lead to confusion when
the number of supported releases constantly changes.

Best,
Fabian

Re: [ANNOUNCE] New Apache Flink Committer - Fabian Paul

[Fabian Paul]

Thanks for the warm welcome, I am looking forward to continuing
working with you all.

Best,
Fabian

Re: [DISCUSS] Definition of Done for Apache Flink

[Fabian Paul]

Hi all,

Maybe I am the devil's advocate but I see the stability of master and
the definition of done as disjunct properties. I think it is more a
question of prioritization that test instabilities are treated as
critical tickets and have to be addressed before continuing any other
work. It will always happen that we merge code that is not 100%
stable; that is probably the nature of software development. I agree
when it comes to documentation that PRs are only mergeable if the
documentation has also been updated.

On the other hand I am a silent fan of the current PR template because
it also provides a summary of the PR to make it easier for committers
to determine the impacts. It also reminds the contributors of our
principles i.e. how do you verify the change should probably not be
answered with "test were not possible".

I agree with @Martijn Visser that we can improve the CI i.e.
performance regression test, execute s3 test but these things should
be addressed in another discussion.

So I would prefer to keep the current PR template.

Best,
Fabian

On Tue, Nov 16, 2021 at 10:17 AM Martijn Visser  wrote:
>
> Hi all,
>
> Thanks for bringing this up for this discussion, because I think it's an
> important aspect.
>
> From my perspective, a 'definition of done' serves two purposes:
> 1. It informs the contributor on what's expected when making a contribution
> in the form of a PR
> 2. It instructs the committer on what to check before accepting/merging a PR
>
> I would use a Github template primarily to deal with the first purpose. I
> think that should be short and easily understandable, preferably with as
> many automated checks as possible.
>
> I would propose something like this to condense information.
>
> 1. It is following the code contribution process, including code style and
> quality guide https://flink.apache.org/contributing/contribute-code.html
> 2. It is covered by tests and all tests have passed
> 3. If it has user facing changes the documentation has been updated
> according to the documentation style guide
>
> These 3 DoD can probably be broken down into multiple automation tests:
>
> * Run a spotless check
> * Run a license check
> * Compile application
> * Run tests
> * Run E2E tests
> * Build documentation
> * Check if JIRA has been mentioned and exists in the PR title and commit
> message
> etc.
>
> Best regards,
>
> Martijn
>
> On Tue, 16 Nov 2021 at 09:08, Francesco Guardiani 
> wrote:
>
> > +1 with Ingo proposal, the goal of the template should be to help developer
> > to do a self check of his/her PR quality, not to define whether something
> > is done or not. It's up to the committer to check that the "definition of
> > done" is fulfilled.
> >
> > > The Definition of Done as suggested:
> >
> > This checklist makes sense to me, although it seems to me we already have
> > these bullet points defined here:
> > https://flink.apache.org/contributing/contribute-code.html
> >
> > On Tue, Nov 16, 2021 at 8:16 AM Ingo Bürk  wrote:
> >
> > > Hi Joe,
> > >
> > > thank you for starting this discussion. Having a common agreement on what
> > > to expect from a PR for it to be merged is very much a worthwhile goal.
> > >
> > > I'm slightly worried about the addition to the PR template. We shouldn't
> > > make opening PRs even more difficult (unless it adds sufficient benefit).
> > >
> > > There are two main benefits to have from using templates: requiring
> > > information from authors to automate certain feedback, and serving as a
> > > self-control checklist for contributors.
> > >
> > > As it stands, a large number of PRs don't fill out the template, and I
> > > haven't yet seen anyone not merge a PR over that, so de-facto we are not
> > > using it for the former.
> > >
> > > For the latter purpose of contributors having a checklist for
> > themselves, I
> > > think the current template is too long already and contains the wrong
> > > content. Being short here is key if we want anyone to read it, and
> > > personally I would cut it down significantly to a description and a
> > couple
> > > of checkboxes.
> > >
> > > This isn't exactly the scope of your proposal, but personally I wouldn't
> > > like to add even more questions that need to be filled out, especially
> > > since they don't actually need to be filled out. It just creates an
> > > annoying burden for contributors and is ignored by those who might
> > benefit
> > > most from reading it anyway.
> > >
> > >
> > > Ingo
> > >
> > >
> > > On Mon, Nov 15, 2021, 22:36 Johannes Moser  wrote:
> > >
> > > > Dear Flink Community,
> > > >
> > > > We as the release managers of the 1.15 release are suggesting to
> > > introduce
> > > > a “Definition of Done".
> > > >
> > > > Let me elaborate a bit on the reasons:
> > > > * During the release process for 1.14 the stability of master was
> > > > sometimes in a state that made contributing to Apache Flink a bad
> > > > experience.
> > > > * Some of the changes that have been contributed seem to be 

Re: [DISCUSS] Releasing Flink 1.14.1

[Fabian Paul]

Hi Martijn,

Thanks for bringing up this topic. I think it would be great to release a
patch version of 1.14 before the end of the year.

Currently, FLINK-24596  is
in progress and I would block the release until it is merged because it
unblocks several use cases by our users. I think the ticket is done by the
end of this week.

Best,
Fabian


On Thu, Nov 25, 2021 at 3:31 AM Yingjie Cao  wrote:

> Hi Martijn,
>
> I moved the fix version of "FLINK-21788
>  - Throw
> PartitionNotFoundException if the partition file has been lost for blocking
> shuffle" to 1.15.0
>
> Best,
> Yingjie
>
> Martijn Visser  于2021年11月25日周四 上午2:40写道：
>
> > Hi all,
> >
> > I would like to start a discussion on releasing Flink 1.14.1. Flink 1.14
> > was released on the 29th of September [1] and so far 107 issues have been
> > resolved, including multiple blockers and critical priorities [2].
> >
> > There are currently 169 open tickets which contain a fixVersion for
> 1.14.1
> > [3]. I'm including the ones that are currently marked as critical or a
> > blocker to verify if these should be included in Flink 1.14.1. It would
> be
> > great if those that are assigned or working on one or more of these
> tickets
> > can give an update on its status.
> >
> > * https://issues.apache.org/jira/browse/FLINK-24543 - Zookeeper
> connection
> > issue causes inconsistent state in Flink -> I think this depends on the
> > outcome of dropping Zookeeper 3.4 as was proposed on the Dev mailing list
> > * https://issues.apache.org/jira/browse/FLINK-25027 - Allow GC of a
> > finished job's JobMaster before the slot timeout is reached
> > * https://issues.apache.org/jira/browse/FLINK-25022 - ClassLoader leak
> > with
> > ThreadLocals on the JM when submitting a job through the REST API
> > * https://issues.apache.org/jira/browse/FLINK-24789 -
> > IllegalStateException
> > with CheckpointCleaner being closed already
> > * https://issues.apache.org/jira/browse/FLINK-24328 - Long term fix for
> > receiving new buffer size before network reader configured -> I'm not
> sure
> > if this would end up in Flink 1.14.1, I think it's more likely that it
> > would be Flink 1.15. Anton/Dawid, could you confirm this?
> > * https://issues.apache.org/jira/browse/FLINK-23946 - Application mode
> > fails fatally when being shut down -> This depends on
> > https://issues.apache.org/jira/browse/FLINK-24038 and I don't see much
> > happening there, so I also expect that this would move to Flink 1.15.
> > David, could you confirm?
> > * https://issues.apache.org/jira/browse/FLINK-22113 - UniqueKey
> constraint
> > is lost with multiple sources join in SQL
> > * https://issues.apache.org/jira/browse/FLINK-21788 - Throw
> > PartitionNotFoundException if the partition file has been lost for
> blocking
> > shuffle -> I'm also expecting that this would move to Flink 1.15, can you
> > confirm Yingjie ?
> >
> > There are quite some other tickets that I've excluded from this list,
> > because they are either test instabilities or are not depending on a
> Flink
> > release to be resolved.
> >
> > Note: there are quite a few test instabilities in the list and help on
> > those is always appreciated. You can check all unassigned tickets
> > instabilities in Jira [4].
> >
> > Are there any other open tickets that we should wait for? Is there a PMC
> > member who would like to manage the release? I'm more than happy to help
> > with monitoring the status of the tickets.
> >
> > Best regards,
> >
> > Martijn
> >
> > [1] https://flink.apache.org/news/2021/09/29/release-1.14.0.html
> > [2]
> >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20fixVersion%20%3D%201.14.1%20ORDER%20BY%20priority%20DESC%2C%20created%20DESC
> > [3]
> >
> >
> https://issues.apache.org/jira/issues?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%201.14.1%20ORDER%20BY%20priority%20DESC%2C%20created%20DESC
> >
> > [4]
> >
> >
> https://issues.apache.org/jira/issues?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%201.14.1%20AND%20labels%20%3D%20test-stability%20AND%20assignee%20in%20(EMPTY)%20ORDER%20BY%20priority%20DESC%2C%20created%20DESC
> >
> > Martijn Visser | Product Manager
> >
> > mart...@ververica.com
> >
> > 
> >
> >
> > Follow us @VervericaData
> >
> > --
> >
> > Join Flink Forward  - The Apache Flink
> > Conference
> >
> > Stream Processing | Event Driven | Real Time
> >
>

Re: [ANNOUNCE] New Apache Flink Committer - Matthias Pohl

[Fabian Paul]

Congrats and well deserved.

Best,
Fabian

On Thu, Dec 2, 2021 at 4:42 PM Ingo Bürk  wrote:
>
> Congrats, Matthias!
>
> On Thu, Dec 2, 2021 at 4:28 PM Till Rohrmann  wrote:
>
> > Hi everyone,
> >
> > On behalf of the PMC, I'm very happy to announce Matthias Pohl as a new
> > Flink committer.
> >
> > Matthias has worked on Flink since August last year. He helped review a ton
> > of PRs. He worked on a variety of things but most notably the tracking and
> > reporting of concurrent exceptions, fixing HA bugs and deprecating and
> > removing our Mesos support. He actively reports issues helping Flink to
> > improve and he is actively engaged in Flink's MLs.
> >
> > Please join me in congratulating Matthias for becoming a Flink committer!
> >
> > Cheers,
> > Till
> >

Re: [ANNOUNCE] New Apache Flink Committer - Ingo Bürk

[Fabian Paul]

Thanks for always pushing Ingo. Congratulations!

Best,
Fabian

On Thu, Dec 2, 2021 at 4:24 PM Till Rohrmann  wrote:
>
> Hi everyone,
>
> On behalf of the PMC, I'm very happy to announce Ingo Bürk as a new Flink
> committer.
>
> Ingo has started contributing to Flink since the beginning of this year. He
> worked mostly on SQL components. He has authored many PRs and helped review
> a lot of other PRs in this area. He actively reported issues and helped our
> users on the MLs. His most notable contributions were Support SQL 2016 JSON
> functions in Flink SQL (FLIP-90), Register sources/sinks in Table API
> (FLIP-129) and various other contributions in the SQL area. Moreover, he is
> one of the few people in our community who actually understands Flink's
> frontend.
>
> Please join me in congratulating Ingo for becoming a Flink committer!
>
> Cheers,
> Till

Re: [DISCUSS] Releasing Flink 1.14.1

[Fabian Paul]

I just opened a PR for
https://issues.apache.org/jira/browse/FLINK-25126 I'll expect to merge
it sometime next week.

Best,
Fabian

On Fri, Dec 3, 2021 at 10:49 AM Martijn Visser  wrote:
>
> Hi all,
>
> Just a status update on the open blockers for 1.14.1:
> * https://issues.apache.org/jira/browse/FLINK-22113 - UniqueKey constraint is 
> lost with multiple sources join in SQL -> I believe most review comments have 
> been fixed and it's just the final review remarks before it's ready.
> * https://issues.apache.org/jira/browse/FLINK-23946 - Application mode fails 
> fatally when being shut down -> @David Morávek can you provide an update?
> * https://issues.apache.org/jira/browse/FLINK-25022 - ClassLoader leak with 
> ThreadLocals on the JM when submitting a job through the REST API -> I think 
> this is just pending on a merge to master and then creating a backport?
> * https://issues.apache.org/jira/browse/FLINK-25126 - Kafka connector tries 
> to commit aborted transaction in batch mode -> This is a new blocker. 
> @fp...@apache.org can you give an update?
> * https://issues.apache.org/jira/browse/FLINK-25132 - KafkaSource cannot work 
> with object-reusing DeserializationSchema -> There's a PR that's being 
> reviewed and then needs a backport.
>
> It would be great if we can finish all these blockers next week to start a 
> release. Do the assignees think that's realistic?
>
> Best regards,
>
> Martijn
>
>
>
>
> On Thu, 2 Dec 2021 at 14:25, Marios Trivyzas  wrote:
>>
>>  https://issues.apache.org/jira/browse/FLINK-22113 will be merged today 
>> (most probably)
>>
>> On Mon, Nov 29, 2021 at 10:15 AM Martijn Visser  
>> wrote:
>>>
>>> Thanks all for the updates! To summarize, these are open tickets that are 
>>> considered blockers for Flink 1.14.1:
>>>
>>> * https://issues.apache.org/jira/browse/FLINK-22113 - UniqueKey constraint 
>>> is lost with multiple sources join in SQL -> @Marios Trivyzas can you give 
>>> an estimate when you expect this to be resolved?
>>> * https://issues.apache.org/jira/browse/FLINK-23946 - Application mode 
>>> fails fatally when being shut down -> A patch is being prepared. @David 
>>> Morávek do you have an estimate when this patch will be there?
>>> * https://issues.apache.org/jira/browse/FLINK-24596 - Bugs in 
>>> sink.buffer-flush before upsert-kafka -> @fp...@apache.org has provided a 
>>> PR is there, so I suspect it would take a couple of days before this is 
>>> merged.
>>> * https://issues.apache.org/jira/browse/FLINK-25022 - ClassLoader leak with 
>>> ThreadLocals on the JM when submitting a job through the REST API -> 
>>> @Chesnay Schepler has provided a PR, so I suspect it would also just take a 
>>> couple of days before this is merged.
>>>
>>> Is there anyone who can help me with creating the actual release when these 
>>> tickets are resolved
>>>
>>> Best regards,
>>>
>>> Martijn
>>>
>>>
>>> On Fri, 26 Nov 2021 at 12:08, Chesnay Schepler  wrote:

 FLINK-25022: I will open a PR later today, and it should be easy to
 backport.
 FLINK-25027: Unlikely to make it for 1.14.1; I also wouldn't consider it
 a blocker

 On 24/11/2021 19:40, Martijn Visser wrote:
 > Hi all,
 >
 > I would like to start a discussion on releasing Flink 1.14.1. Flink 1.14
 > was released on the 29th of September [1] and so far 107 issues have been
 > resolved, including multiple blockers and critical priorities [2].
 >
 > There are currently 169 open tickets which contain a fixVersion for 
 > 1.14.1
 > [3]. I'm including the ones that are currently marked as critical or a
 > blocker to verify if these should be included in Flink 1.14.1. It would 
 > be
 > great if those that are assigned or working on one or more of these 
 > tickets
 > can give an update on its status.
 >
 > * https://issues.apache.org/jira/browse/FLINK-24543 - Zookeeper 
 > connection
 > issue causes inconsistent state in Flink -> I think this depends on the
 > outcome of dropping Zookeeper 3.4 as was proposed on the Dev mailing list
 > * https://issues.apache.org/jira/browse/FLINK-25027 - Allow GC of a
 > finished job's JobMaster before the slot timeout is reached
 > * https://issues.apache.org/jira/browse/FLINK-25022 - ClassLoader leak 
 > with
 > ThreadLocals on the JM when submitting a job through the REST API
 > * https://issues.apache.org/jira/browse/FLINK-24789 - 
 > IllegalStateException
 > with CheckpointCleaner being closed already
 > * https://issues.apache.org/jira/browse/FLINK-24328 - Long term fix for
 > receiving new buffer size before network reader configured -> I'm not 
 > sure
 > if this would end up in Flink 1.14.1, I think it's more likely that it
 > would be Flink 1.15. Anton/Dawid, could you confirm this?
 > * https://issues.apache.org/jira/browse/FLINK-23946 - Application mode
 > fails fatally when being shut down -> This depends on
>>>

Re: [DISCUSS] FLIP-196: Source API stability guarantees

[Fabian Paul]

Hi all,

Thanks Till for starting this discussion. It is great to see these
facts written down since they definitely caused friction in the past
because of different interpretations. Overall I agree with everything
being said in this FLIP. I was just wondering whether we can put the
label explaining table somewhere into our docs. FLIPs are usually only
snapshots for the time being and are hard to search for. It would
allow users to instantly determine the meaning of the used annotation.

I am also missing in this FLIP the connection to the Deprecated
annotation. I think we also need to clarify how this should be used in
conjunction with the API stability.

Best,
Fabian

On Mon, Dec 6, 2021 at 10:03 AM Ingo Bürk  wrote:
>
> Hi Till,
>
> seems I misunderstood it then; thanks for the clarification! And yes, with
> that I would fully agree.
>
>
> Ingo
>
> On Mon, Dec 6, 2021 at 9:59 AM Till Rohrmann  wrote:
>
> > Hi Ingo,
> >
> > No, the added method can have a weaker stability guarantee as long as the
> > user does not have to implement it. In order to give an example the
> > following extension would be ok imo:
> >
> > @Public
> > interface Foobar {
> > @Public
> > int foo();
> >
> > @Experimental
> > default ExperimentalResult bar() {
> >   return ExperimentalResult.notSupported();
> > }
> > }
> >
> > The following extension would not be ok because here the user needs to
> > implement something new:
> >
> > @Public
> > interface Foobar {
> > @Public
> > int foo();
> >
> > @Experimental
> > ExperimentalResult bar();
> > }
> >
> > Moreover, if the user uses bar(), then he opts-in to only get @Experimental
> > stability guarantees.
> >
> > I will add this example to the FLIP for illustrative purposes.
> >
> > Cheers,
> > Till
> >
> > On Fri, Dec 3, 2021 at 6:52 PM Ingo Bürk  wrote:
> >
> > > > Would it be enough to say that for example all classes in the module
> > > flink-java have to be annotated? What I would like to avoid is having to
> > > annotate all classes in some internal module like flink-rpc.
> > >
> > > I don't think it is, but we certainly could restrict it to certain top
> > > level o.a.f.xyz packages.
> > >
> > > > Extending existing classes will only be possible if you can provide a
> > > default implementation
> > >
> > > That I'm totally fine with, but based on that sentence in the FLIP if I
> > > have a public interface and extend it, even with a default
> > implementation,
> > > I _have_ to have this method be stable already as well, right? I couldn't
> > > for example add an experimental method to an interface.
> > >
> > > This would also include all classes used as argument and return type of
> > > such methods too, which seems quite restrictive.
> > >
> > >
> > > Best
> > > Ingo
> > >
> > > On Fri, Dec 3, 2021, 17:51 Till Rohrmann  wrote:
> > >
> > > > >That's still a much weaker requirement, though, as classes can just be
> > > > left unannotated, which is why I prefer annotating all classes
> > regardless
> > > > of location.
> > > >
> > > > Would it be enough to say that for example all classes in the module
> > > > flink-java have to be annotated? What I would like to avoid is having
> > to
> > > > annotate all classes in some internal module like flink-rpc.
> > > >
> > > > > How would you handle e.g. extending an existing Public interface
> > with a
> > > > new method in this case, though? You'd be forced to immediately make
> > the
> > > > new method Public as well, or place it somewhere else entirely, which
> > > leads
> > > > to unfavorable design. I don't think we should disallow extending
> > classes
> > > > with methods of a weaker stability.
> > > >
> > > > Extending existing classes will only be possible if you can provide a
> > > > default implementation. If the user needs to do something, then it is
> > not
> > > > compatible and needs to be handled differently (e.g. by offering a new
> > > > experimental interface that one can use). If we don't enforce this,
> > then
> > > I
> > > > don't see how we can provide source stability guarantees.
> > > >
> > > > Cheers,
> > > > Till
> > > >
> > > > On Fri, Dec 3, 2021 at 5:22 PM Ingo Bürk  wrote:
> > > >
> > > > > Hi Till,
> > > > >
> > > > > > Personally, I'd be fine to say that in API modules (tbd what this
> > is
> > > > > > (probably transitive closure of all APIs)) we require every class
> > to
> > > be
> > > > > > annotated.
> > > > >
> > > > > At least we'll then need the reverse rule: no classes outside *.api.*
> > > > > packages CAN have an API annotation (other than Internal), of course
> > > with
> > > > > many existing violations that need to be accapted.
> > > > >
> > > > > That's still a much weaker requirement, though, as classes can just
> > be
> > > > left
> > > > > unannotated, which is why I prefer annotating all classes regardless
> > of
> > > > > location.
> > > > >
> > > > > > If we have cases that violate the guideline, then I think we either
> > > > have
> > > > > 

Re: [DISCUSS] Releasing Flink 1.14.1

[Fabian Paul]

Hi Martijn,

I just opened the backport for
https://issues.apache.org/jira/browse/FLINK-25132. The changes are
already approved I only wait for a green Azure build.

Best,
Fabian

On Thu, Dec 9, 2021 at 4:01 PM Martijn Visser  wrote:
>
> Hi all,
>
> Thanks for the fixes Jingsong and Zhu!
>
> That means that we still have the following tickets open:
>
> * https://issues.apache.org/jira/browse/FLINK-23946 - Application mode
> fails fatally when being shut down -> A PR is there, just pending a review.
> * https://issues.apache.org/jira/browse/FLINK-25126 - Kafka connector tries
> to commit aborted transaction in batch mode -> I believe this is pending a
> backport, correct @fp...@apache.org  ?
> * https://issues.apache.org/jira/browse/FLINK-25132 - KafkaSource cannot
> work with object-reusing DeserializationSchema -> @renqs...@gmail.com
>  can you provide an ETA for this ticket?
> * https://issues.apache.org/jira/browse/FLINK-25199 - fromValues does not
> emit final MAX watermark -> @Marios Trivyzas  can you
> provide an ETA for this ticket?
> * https://issues.apache.org/jira/browse/FLINK-25227 - Comparing the
> equality of the same (boxed) numeric values returns false -> @Caizhi Weng
>  mentioned that a fix is planned for today/tomorrow.
> I am wondering if this is indeed a blocker for 1.14.1, but given that there
> are still some blockers waiting to be merged we could probably include it.
>
> Best regards,
>
> Martijn
>
> On Thu, 9 Dec 2021 at 07:31, Caizhi Weng  wrote:
>
> > Hi devs!
> >
> > Sorry for the interruptions, but I just found an issue [1] (which I think
> > is a blocking one) in every Flink version, including Flink 1.14.1.
> >
> > For Flink < 1.15, this issue will cause incorrect result when user cast
> > two strings to numerics and compare the numerics.
> >
> > I'm planning for a quick fix today or tomorrow.
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-25227
> >
> > Zhu Zhu  于2021年12月9日周四 10:48写道：
> >
> >> update: backport of FLINK-19142 is done
> >>
> >> Thanks,
> >> Zhu
> >>
> >> Zhu Zhu  于2021年12月8日周三 19:35写道：
> >>
> >> > Hi Martijn,
> >> >
> >> > I'd like to backport the fix of FLINK-19142 to 1.14.1.
> >> > The backport is in progress.
> >> > Will update it here when it is done.
> >> >
> >> > Thanks,
> >> > Zhu
> >> >
> >> > Jingsong Li  于2021年12月8日周三 10:33写道：
> >> >
> >> >> Hi Martijn,
> >> >>
> >> >> We just created a cherry-pick pull-request for
> >> >> https://issues.apache.org/jira/browse/FLINK-20370
> >> >> We could finish it as soon as possible.
> >> >>
> >> >> Best,
> >> >> Jingsong
> >> >>
> >> >> On Fri, Dec 3, 2021 at 10:25 PM Fabian Paul  wrote:
> >> >> >
> >> >> > I just opened a PR for
> >> >> > https://issues.apache.org/jira/browse/FLINK-25126 I'll expect to
> >> merge
> >> >> > it sometime next week.
> >> >> >
> >> >> > Best,
> >> >> > Fabian
> >> >> >
> >> >> > On Fri, Dec 3, 2021 at 10:49 AM Martijn Visser <
> >> mart...@ververica.com>
> >> >> wrote:
> >> >> > >
> >> >> > > Hi all,
> >> >> > >
> >> >> > > Just a status update on the open blockers for 1.14.1:
> >> >> > > * https://issues.apache.org/jira/browse/FLINK-22113 - UniqueKey
> >> >> constraint is lost with multiple sources join in SQL -> I believe most
> >> >> review comments have been fixed and it's just the final review remarks
> >> >> before it's ready.
> >> >> > > * https://issues.apache.org/jira/browse/FLINK-23946 - Application
> >> >> mode fails fatally when being shut down -> @David Morávek can you
> >> provide
> >> >> an update?
> >> >> > > * https://issues.apache.org/jira/browse/FLINK-25022 - ClassLoader
> >> >> leak with ThreadLocals on the JM when submitting a job through the
> >> REST API
> >> >> -> I think this is just pending on a merge to master and then creating
> >> a
> >> >> backport?
> >> >> > > * https://issues.apache.org/jira/browse/FLINK-25126 - Kafka
> >> >> connector tries to commit aborted trans

Re: [DISCUSS] Releasing Flink 1.14.1

[Fabian Paul]

Actually I meant https://issues.apache.org/jira/browse/FLINK-25126
sorry for the confusion.

On Thu, Dec 9, 2021 at 4:55 PM Fabian Paul  wrote:
>
> Hi Martijn,
>
> I just opened the backport for
> https://issues.apache.org/jira/browse/FLINK-25132. The changes are
> already approved I only wait for a green Azure build.
>
> Best,
> Fabian
>
> On Thu, Dec 9, 2021 at 4:01 PM Martijn Visser  wrote:
> >
> > Hi all,
> >
> > Thanks for the fixes Jingsong and Zhu!
> >
> > That means that we still have the following tickets open:
> >
> > * https://issues.apache.org/jira/browse/FLINK-23946 - Application mode
> > fails fatally when being shut down -> A PR is there, just pending a review.
> > * https://issues.apache.org/jira/browse/FLINK-25126 - Kafka connector tries
> > to commit aborted transaction in batch mode -> I believe this is pending a
> > backport, correct @fp...@apache.org  ?
> > * https://issues.apache.org/jira/browse/FLINK-25132 - KafkaSource cannot
> > work with object-reusing DeserializationSchema -> @renqs...@gmail.com
> >  can you provide an ETA for this ticket?
> > * https://issues.apache.org/jira/browse/FLINK-25199 - fromValues does not
> > emit final MAX watermark -> @Marios Trivyzas  can you
> > provide an ETA for this ticket?
> > * https://issues.apache.org/jira/browse/FLINK-25227 - Comparing the
> > equality of the same (boxed) numeric values returns false -> @Caizhi Weng
> >  mentioned that a fix is planned for today/tomorrow.
> > I am wondering if this is indeed a blocker for 1.14.1, but given that there
> > are still some blockers waiting to be merged we could probably include it.
> >
> > Best regards,
> >
> > Martijn
> >
> > On Thu, 9 Dec 2021 at 07:31, Caizhi Weng  wrote:
> >
> > > Hi devs!
> > >
> > > Sorry for the interruptions, but I just found an issue [1] (which I think
> > > is a blocking one) in every Flink version, including Flink 1.14.1.
> > >
> > > For Flink < 1.15, this issue will cause incorrect result when user cast
> > > two strings to numerics and compare the numerics.
> > >
> > > I'm planning for a quick fix today or tomorrow.
> > >
> > > [1] https://issues.apache.org/jira/browse/FLINK-25227
> > >
> > > Zhu Zhu  于2021年12月9日周四 10:48写道：
> > >
> > >> update: backport of FLINK-19142 is done
> > >>
> > >> Thanks,
> > >> Zhu
> > >>
> > >> Zhu Zhu  于2021年12月8日周三 19:35写道：
> > >>
> > >> > Hi Martijn,
> > >> >
> > >> > I'd like to backport the fix of FLINK-19142 to 1.14.1.
> > >> > The backport is in progress.
> > >> > Will update it here when it is done.
> > >> >
> > >> > Thanks,
> > >> > Zhu
> > >> >
> > >> > Jingsong Li  于2021年12月8日周三 10:33写道：
> > >> >
> > >> >> Hi Martijn,
> > >> >>
> > >> >> We just created a cherry-pick pull-request for
> > >> >> https://issues.apache.org/jira/browse/FLINK-20370
> > >> >> We could finish it as soon as possible.
> > >> >>
> > >> >> Best,
> > >> >> Jingsong
> > >> >>
> > >> >> On Fri, Dec 3, 2021 at 10:25 PM Fabian Paul  wrote:
> > >> >> >
> > >> >> > I just opened a PR for
> > >> >> > https://issues.apache.org/jira/browse/FLINK-25126 I'll expect to
> > >> merge
> > >> >> > it sometime next week.
> > >> >> >
> > >> >> > Best,
> > >> >> > Fabian
> > >> >> >
> > >> >> > On Fri, Dec 3, 2021 at 10:49 AM Martijn Visser <
> > >> mart...@ververica.com>
> > >> >> wrote:
> > >> >> > >
> > >> >> > > Hi all,
> > >> >> > >
> > >> >> > > Just a status update on the open blockers for 1.14.1:
> > >> >> > > * https://issues.apache.org/jira/browse/FLINK-22113 - UniqueKey
> > >> >> constraint is lost with multiple sources join in SQL -> I believe most
> > >> >> review comments have been fixed and it's just the final review remarks
> > >> >> before it's ready.
> > >> >> > > * https://issues.apache.org/jira

Re: [DISCUSS] FLIP-191: Extend unified Sink interface to support small file compaction

[Fabian Paul]

t;>
> >> Regards,
> >> Roman
> >>
> >>
> >> On Tue, Nov 9, 2021 at 5:23 AM Reo Lei  wrote:
> >> >
> >> > Hi Fabian,
> >> >
> >> > Thanks for drafting the FLIP and trying to support small file 
> >> > compaction. I
> >> > think this feature is very urgent and valuable for users(at least for 
> >> > me).
> >> >
> >> > Currently I am trying to support streaming rewrite(compact) for Iceberg 
> >> > on
> >> > PR#3323 <https://github.com/apache/iceberg/pull/3323>. As Steven 
> >> > mentioned,
> >> > Iceberg sink and compact data through the following steps:
> >> > Step-1: Some parallel data writer(sinker) to write streaming data as 
> >> > files.
> >> > Step-2: A single parallelism data files committer to commit the completed
> >> > files as soon as possible to make them available.
> >> > Step-3: Some parallel file rewriter(compactor) to collect committed files
> >> > from multiple checkpoints, and rewriter(compact) them together once the
> >> > total file size or number of files reach the threshold.
> >> > Step-4: A single parallelism rewrite(compact) result committer to commit
> >> > the rewritten(compacted) files to replace the old files and make them
> >> > available.
> >> >
> >> >
> >> > If Flink want to support small file compaction, some key point I think is
> >> > necessary:
> >> >
> >> > 1, Compact files from multiple checkpoints.
> >> > I totally agree with Jingsong, because completed file size usually could
> >> > not reach the threshold in a single checkpoint. Especially for 
> >> > partitioned
> >> > table, we need to compact the files of each partition, but usually the 
> >> > file
> >> > size of each partition will be different and may not reach the merge
> >> > threshold. If we compact these files, in a single checkpoint, regardless 
> >> > of
> >> > whether the total file size reaches the threshold, then the value of
> >> > compacting will be diminished and we will still get small files because
> >> > these compacted files are not reach to target size. So we need the
> >> > compactor to collect committed files from multiple checkpoints and 
> >> > compact
> >> > them until they reach the threshold.
> >> >
> >> > 2, Separate write phase and compact phase.
> >> > Users usually hope the data becomes available as soon as possible, and 
> >> > the
> >> >  end-to-end latency is very important. I think we need to separate the
> >> > write and compact phase. For the write phase, there include the Step-1
> >> > and Step-2, we sink data as file and commit it pre checkpoint and 
> >> > regardless
> >> > of whether the file size it is. That could ensure the data will be
> >> > available ASAP. For the compact phase, there include the Step-3
> >> > and Step-4,  the compactor should collect committed files from multiple
> >> > checkpoints and compact them asynchronously once they reach the 
> >> > threshold,
> >> > and the compact committer will commit the  compaction result in the next
> >> > checkpoint. We compact the committed files asynchronously because we 
> >> > don't
> >> > want the compaction to affect the data sink or the whole pipeline.
> >> >
> >> > 3, Exactly once guarantee between write and compact phase.
> >> > Once we separate write phase and compact phase, we need to consider
> >> > how to guarantee
> >> > the exact once semantic between two phases. We should not lose any data 
> >> > or
> >> > files on the compactor(Step-3) in any case and cause the compaction 
> >> > result
> >> > to be inconsistent with before. I think flink should provide an 
> >> > easy-to-use
> >> > interface to make that easier.
> >> >
> >> > 4, Metadata operation and  compaction result validation.
> >> > In the compact phase, there may be not only compact files, but also a lot
> >> > of metadata operations, such as the iceberg needing to read/write 
> >> > manifest
> >> > and do MOR. And we need some interface to support users to do some
> >> > validation of the compaction result. I think these points should be
> >> > considered

Re: Re: [DISCUSS] FLIP-191: Extend unified Sink interface to support small file compaction

[Fabian Paul]

>> > >> 3. Writing small files, then reading and merging them for *every*
>>
>> > >> checkpoint seems worse than only reading them on recovery. I guess I'm
>>
>> > >> missing some cases of reading, so to me it would make sense to mention
>>
>> > >> these cases explicitly in the FLIP motivation section.
>>
>> > >>
>>
>> > >> 4. One way to avoid write-read-merge is by wrapping SinkWriter with
>>
>> > >> another one, which would buffer input elements in a temporary storage
>>
>> > >> (e.g. local file) until a threshold is reached; after that, it would
>>
>> > >> invoke the original SinkWriter. And if a checkpoint barrier comes in
>>
>> > >> earlier, it would send written data to some aggregator. It will
>>
>> > >> increase checkpoint delay (async phase) compared to the current Flink;
>>
>> > >> but not compared to the write-read-merge solution, IIUC.
>>
>> > >> Then such "BufferingSinkWriters" could aggregate input elements from
>>
>> > >> each other, potentially recursively (I mean something like
>>
>> > >> https://cwiki.apache.org/confluence/download/attachments/173082889/DSTL-DFS-DAG.png
>>
>> > >> )
>>
>> > >>
>>
>> > >> 5. Reducing the number of files by reducing aggregator parallelism as
>>
>> > >> opposed to merging on reaching size threshold will likely be less
>>
>> > >> optimal and more difficult to configure. OTH, thresholds might be more
>>
>> > >> difficult to implement and (with recursive merging) would incur higher
>>
>> > >> latency. Maybe that's also something to decide explicitly or at least
>>
>> > >> mention in the FLIP.
>>
>> > >>
>>
>> > >>
>>
>> > >>
>>
>> > >> Regards,
>>
>> > >> Roman
>>
>> > >>
>>
>> > >>
>>
>> > >> On Tue, Nov 9, 2021 at 5:23 AM Reo Lei wrote:
>>
>> > >> >
>>
>> > >> > Hi Fabian,
>>
>> > >> >
>>
>> > >> > Thanks for drafting the FLIP and trying to support small file 
>> > >> > compaction. I
>>
>> > >> > think this feature is very urgent and valuable for users(at least for 
>> > >> > me).
>>
>> > >> >
>>
>> > >> > Currently I am trying to support streaming rewrite(compact) for 
>> > >> > Iceberg on
>>
>> > >> > PR#3323 . As Steven mentioned,
>>
>> > >> > Iceberg sink and compact data through the following steps:
>>
>> > >> > Step-1: Some parallel data writer(sinker) to write streaming data as 
>> > >> > files.
>>
>> > >> > Step-2: A single parallelism data files committer to commit the 
>> > >> > completed
>>
>> > >> > files as soon as possible to make them available.
>>
>> > >> > Step-3: Some parallel file rewriter(compactor) to collect committed 
>> > >> > files
>>
>> > >> > from multiple checkpoints, and rewriter(compact) them together once 
>> > >> > the
>>
>> > >> > total file size or number of files reach the threshold.
>>
>> > >> > Step-4: A single parallelism rewrite(compact) result committer to 
>> > >> > commit
>>
>> > >> > the rewritten(compacted) files to replace the old files and make them
>>
>> > >> > available.
>>
>> > >> >
>>
>> > >> >
>>
>> > >> > If Flink want to support small file compaction, some key point I 
>> > >> > think is
>>
>> > >> > necessary:
>>
>> > >> >
>>
>> > >> > 1, Compact files from multiple checkpoints.
>>
>> > >> > I totally agree with Jingsong, because completed file size usually 
>> > >> > could
>>
>> > >> > not reach the threshold in a single checkpoint. Especially for 
>> > >> > partitioned
>>
>> > >> > table, we need to compact the files of each partition, but usually 
>> > >> > the file
>>
>> > >>

Re: Re: [DISCUSS] FLIP-191: Extend unified Sink interface to support small file compaction

[Fabian Paul]

Hi Till,

Good point, in the scenario with the blocking keyed exchange between
the writer and committer my idea is to make the committer effectively
the global committer. With Sink V2 there is no real difference anymore
between the committer and global committer.
You are right that everything after the committer would be part of the
same failover region but we plan to insert a blocking exchange by
default before all of the custom topologies.

Best,
Fabian

On Thu, Dec 16, 2021 at 11:08 AM Till Rohrmann  wrote:
>
> Hi Fabian,
>
> quick question on your comment 3. If there is a pipelined data exchange
> with a keyBy between the writers/committers and the component that does the
> global commit, then there will only be a single failover region. So is it
> correct that you assumed blocking data exchanges for the scenario you
> described?
>
> Cheers,
> Till
>
> On Thu, Dec 16, 2021 at 9:23 AM Fabian Paul  wrote:
>
> > Hi Yun,
> >
> > Thanks for your fast feedback. Let me clarify your points.
> >
> > 1. We solve it by using StreamExchangeMode.BATCH before any exchange.
> > That obviously doesn’t help with lost TM but we would need to employ
> > HA storage for that. Same issue as now and orthogonal.
> >
> > 2. Extending V1 with V2 or vice versa would require renames of methods
> > (since return types are non-optional) and is making API changes. Even
> > though Experimental, we want to give connector developers the
> > opportunity to provide 1 implementation for all of Flink 1.X. We will
> > offer an internal adapter from V1 to V2, 2 sinkTo , and internally
> > just have one code-path.
> >
> > 3. DataStreamSink would act as a unified view on all the operators and
> > update them all at once when using setParallelism and so on (setName
> > and setUid will receive suffixes per operator).
> > Iceberg actually has a different requirement: They want to have a
> > committer with parallelism 1 but as a coordinator such that
> > embarrassingly parallel pipelines have different fail-over regions. I
> > was thinking that in this case, they need to implement a no-op
> > committer (that just forwards the committables) and use a post-commit
> > topology that achieves that.
> > Another option is that they use the preCommit topology and insert a
> > constant key-by that forwards all committables to a single committer.
> > We are planning to provide building blocks for such pipelines as we
> > go.
> >
> > Best,
> > Fabian
> >
> > On Thu, Dec 16, 2021 at 5:50 AM Yun Gao  wrote:
> > >
> > > Hi Fabian,
> > >
> > > Very thanks for the update! I think the latest version in general looks
> > good from my side
> > > and I think using separate feature interface would be much more easy to
> > understand
> > > and extend in the future. I have some pending issues on the details
> > though:
> > >
> > > 1. The first one is if we could support end-to-end exactly-once with
> > post-committing
> > > topology in the batch mode ? Since for the batch mode, currently we
> > could only commit
> > >  all the transactions after the whole job is finished, otherwise if
> > there are JM failover or the
> > > writer / committer get restarted due to indeterminstic (A -> [B1, B2],
> > A, B1 have finished and
> > >  B2 failed, if -> is rebalance / random / rescale,  all of A, B1, B2
> > would restarted), there might
> > > be repeat records. Previously one possible thought is to move committer
> > and global committer
> > >  to the operator coordinator, but if it is a topology, we might need
> > some other kind of solutions?
> > >
> > > 2. I also want to have a dobule confirmation with the compatibility:
> > since the old sink is also named
> > > with Sink, do we want to put the Sink v2 in a new package ? Besides,
> > since we might want to keep
> > > only have one `sinkTo(Sink sink)` , perhaps we also need to make the
> > Sink v1 to be a subclass of
> > > Sink v2 and extends the stateful and two-phase-commit sinks, right?
> > >
> > > 3. I'd like also have a confirmation on ours thoughts with the
> > `DataStreamSink` returned by the sinkTo method:
> > > The main issue is how do we implement the method like `setParallelism`
> > or `setMaxParallelism` since now the sink
> > > would be translated to multiple transformations? perhaps we could make
> > it the default values for all the transformations
> > > for the sink? A related issue would be for iceberg sink, I think it
> > would need to have onl

Re: [DISCUSS] GHA migration roadmap

[Fabian Paul]

Hi Nico,

Thanks a lot for drafting the proposal. I really like the
fully-fledged phasing model. All in all, I am +1 to move away from
azure and can only second all the points you have mentioned.

I only want to clarify one point. So far my understanding was that the
GHA resources are managed on a GitHub organizational level in contrast
to Azure pipelines where projects have certain resources. What happens
if more and more projects inside the Apache Github organization
migrate to GHA? Will this affect the build queue time?

Best,
Fabian

On Thu, Dec 16, 2021 at 3:59 PM Nicolaus Weidner
 wrote:
>
> Hi all,
>
> as several people know by now, we are planning to move from Azure CI to
> Github Actions. This is motivated by (not an exhaustive list):
> - Not needing to mirror the repo anymore for CI
> - Improving the contributor experience, especially for new contributors
> - GHA development being more active than Azure CI development
>
> In case someone wants to check out the current version of the planned GHA
> workflow, you can find it here:
> https://github.com/ververica/flink/blob/master/.github/workflows/hadoop-2.8.3-scala-2.12-workflow.yml
> Past runs can be seen here: https://github.com/ververica/flink/actions (lots
> of red, but this is almost always not due to the workflow)
>
> I want to put a draft for the migration roadmap up for discussion. It's
> divided into several phases:
>
> *Phase 1: *GHA activated on master (but not required)
> - A single CI machine is converted to run GHA runners (instead of Azure
> runners) and runs the workflow on pushes to master
> - Azure CI remains unchanged and is still the source of truth
> - We can compare runtimes and behavior/failures
> - Timeframe: 2 weeks
>
> *Phase 2: *Additional features
> - Any additional functionality that we want to add to GHA is added (e.g.
> not running the workflow if workflow files were modified)
> - Functionality from FlinkCIBot that we want to keep is ported over
> (syncing with the mirror repo can be dropped, but there are some automated
> checks that we want to keep)
> - We can monitor whether performance is impacted by any change
> - Timeframe: 2 weeks
>
> *Phase 3: *Cron jobs and (some) PR triggers run on GHA
> - GHA cron builds activated (for master and release branches)
> - Note: Includes some backports to all affected branches, else the
> workflows won’t run:
> https://stackoverflow.com/questions/61989951/github-action-workflow-not-running/61992817#61992817
> - GHA builds run for PRs of select committers (the idea is to try out
> builds for all the intended trigger conditions)
> - Timeframe: 1 week
>
> *Up to this point, the existing CI pipeline is mostly unaffected - we only
> took away one CI machine.*
>
> *Phase 4: *Full switch to GHA
> - Set up GHA runners on all machines
> - GHA builds are activated for all PRs
> - Either Azure or GHA build is required
> - GHA runners are activated, Azure runners are deactivated (but not yet
> removed) apart from 1 machine (for stragglers)
> - Azure cron jobs are disabled, but kept around in case we need to revert
> - Timeframe: 1-2 weeks
>
> *Phase 5: *Removal of Azure CI leftovers
> - Only after we are satisfied that GHA is stable (at least 1 month after
> the switch, can be longer)
> - Green GHA build is required from now on
> - Stale PRs that don't have a GHA run will have to trigger a new one (but
> they would most likely have to rebase anyway...)
> - (old) FlinkCIBot is disabled
> - Azure yamls are deleted
> - Azure runners are removed from machines
>
>
> Timing-wise, the full switch to GHA should happen during a quiet time, far
> away from a release. The remaining phases shouldn't have much impact, but
> right before a release is not a good moment, of course.
> Please give us your thoughts and point out anything we missed or that
> doesn't seem to make sense!
>
> Best,
> Nico

Re: Re: [DISCUSS] FLIP-191: Extend unified Sink interface to support small file compaction

[Fabian Paul]

lized) -> pre-commit topology -> committer -> 
> succesful committables (materialized)
> Stage 3: succesful committables (materialized) -> post-commit topology
>
> If any instance of a given stages fails, the whole stage is restarted.
> So, in case of the main pipeline (stage 1) fails, no data will be committed 
> at all. On a rerun, we start fresh (or from the previous stage).
> Only, when all data has been written, we start with committing the data. An 
> issue during committing, will retrigger the commit stage (stage 2) and only 
> that stage. Thus, all committables are stable and remain stable.
> When we are done committing all committables, we run the post-commit topology 
> similarly "atomically".
>
> So now the cavaets:
> - If committer is rerun, all committables are restarted. So the committer 
> needs to be idempotent. That is the same with STREAMING mode now and afaik 
> there is no way around it.
> - If we lose a TM during commit phase, we will run into the original issues 
> of inconstent data as we need to rerun the whole job. That would be solved 
> with HA storage and we haven't found any solution that doesn't require some 
> kind of external storage. However, the commit phase should be rather fast and 
> errors are improbable (low volume).
>
> I'd still like to have an HA storage but that issue is also in Sink V1 and 
> kind of orthogonal. It's also nothing that we can solve without involving 
> more folks (I'm trying to kick start that in the background).
>
> On Thu, Dec 16, 2021 at 1:31 PM Yun Gao  wrote:
> Hi,
>
> Very thanks Fabian for the explanation and it solves most of the issues.
> There is one left issue I want to have a double confirmation is that for
> the edges between writer and committer and in the post-committer topology,
> perhaps the result partition with HA storage is not enough solve all the 
> issues
> directly ? It is due to after the committer and post-committer topology is 
> finished
> and the data is committed, it might still be restarted due to JM failover and 
> the
> deterministic problem (namely the example of  (A -> [B1, B2], A, B1 have 
> finished and
> B2 failed, if -> is rebalance / random / rescale,  all of A, B1, B2 would 
> restarted). Then
> the records would be produced and created for the second times.
>
> We might let the writers to skip producing the new records, but if we have 
> multiple sinks like
> OP1 -> (writer 1 -> committer 1)
>  |--> (writer 2 -> committer 2)
>
> and the failover happens after writer 1 & committer 1 get finished but writer 
> 2 is running,
> if op1 produced different records across the two runs, then the two sinks 
> would produces
> different data, which might be not suitable in some cases. Perhaps we need 
> some support
> from the scheduler side?
>
> But I also agree this could be a separate issue and we could solve it 
> separately in some future
> as long as we know how to solve it~
>
> Best,
> Yun
>
>
> --
> From:Arvid Heise 
> Send Time:2021 Dec. 16 (Thu.) 19:54
> To:dev 
> Cc:Yun Gao 
> Subject:Re: Re: [DISCUSS] FLIP-191: Extend unified Sink interface to support 
> small file compaction
>
> Just a quick amend: There will be no blocking exchange in the pre-writer 
> exchange for performance reasons.
> After the writer, we have tiny data volume and are free to add as many as we 
> see necessary.
>
> On Thu, Dec 16, 2021 at 11:18 AM Fabian Paul  wrote:
> Hi Till,
>
>  Good point, in the scenario with the blocking keyed exchange between
>  the writer and committer my idea is to make the committer effectively
>  the global committer. With Sink V2 there is no real difference anymore
>  between the committer and global committer.
>  You are right that everything after the committer would be part of the
>  same failover region but we plan to insert a blocking exchange by
>  default before all of the custom topologies.
>
>  Best,
>  Fabian
>
>  On Thu, Dec 16, 2021 at 11:08 AM Till Rohrmann  wrote:
>  >
>  > Hi Fabian,
>  >
>  > quick question on your comment 3. If there is a pipelined data exchange
>  > with a keyBy between the writers/committers and the component that does the
>  > global commit, then there will only be a single failover region. So is it
>  > correct that you assumed blocking data exchanges for the scenario you
>  > described?
>  >
>  > Cheers,
>  > Till
>  >
>  > On Thu, Dec 16, 2021 at 9:23 AM Fabian Paul  wrote:
>  >
>  > > Hi Yun,
>  > >
>  > > Thanks for your fast feedback. Let me c

[VOTE] FLIP-191: Extend unified Sink interface to support small file compaction

[Fabian Paul]

Hi everyone,

I'd like to start a vote on FLIP-191: Extend unified Sink interface to
support small file compaction [1] that has been discussed in this
thread [2].

The vote will be open for at least 72 hours unless there is an objection or
not enough votes.

Best,
Fabian

[1] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-191%3A+Extend+unified+Sink+interface+to+support+small+file+compaction
[2] https://lists.apache.org/thread/97kwy315t9r4j02l8v0wotkll4tngb3m

[RESULT][VOTE] FLIP-191: Extend unified Sink interface to support small file compaction

[Fabian Paul]

I am happy to announce that FLIP-191 [1] has been accepted by this vote [2].

There are 5 approving votes, 3 of which are binding:
* Martijn Visser (non-binding)
* Yun Gao (binding)
* Arvid Heise (binding)
* Guowei Ma (binding)
* Jing Ge  (non-binding)

There are no disapproving votes.

Thanks everyone!
Fabian

[1] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-191%3A+Extend+unified+Sink+interface+to+support+small+file+compaction
[2] https://lists.apache.org/thread/97kwy315t9r4j02l8v0wotkll4tngb3m

Re: [VOTE] Create a separate sub project for FLIP-188: flink-store

[Fabian Paul]

Hi all,

I just wanted to give my two cents for the build system discussion. In
general, I agree with David's opinion to start new projects with
Gradle but during the development of the external connector
repository, we found some difficulties that still need to be solved. I
do not want to force another project (with maybe limited Gradle
expertise) to use Gradle right now. After we fully established the
external connector repository with Gradle I can imagine converting the
other external repositories as well.

Best,
Fabian

On Mon, Jan 10, 2022 at 12:04 PM Jark Wu  wrote:
>
> I'm also in favour of "flink-table-store".
>
> Best,
> Jark
>
> On Mon, 10 Jan 2022 at 16:18, David Morávek  wrote:
>>
>> Hi Jingsong,
>>
>> the connector repository prototype I've seen is being built on top of
>> Gradle [1], that's why I was referring to it (I think one idea was also to
>> migrate the main repository to Gradle eventually). I think Martijn / Fabian
>> may be bit more familiar with the connectors repository effort and could
>> shed some light on this.
>>
>> [1] https://github.com/apache/flink-connectors
>>
>> Best,
>> D.
>>
>> On Mon, Jan 10, 2022 at 8:57 AM Yu Li  wrote:
>>
>> > +1 for a separate repository and release pipeline in the same way as
>> > flink-statefun [1], flink-ml [2] and the coming flink-connectors [3].
>> >
>> > +1 for naming it as "flink-table-store" (I'm also ok with
>> > "flink-table-storage", but slightly prefer "flink-table-store" because it's
>> > shorter)
>> >
>> > Thanks for driving this Jingsong, and look forward to a fast evolution of
>> > this direction!
>> >
>> > Best Regards,
>> > Yu
>> >
>> > [1] https://github.com/apache/flink-statefun
>> > [2] https://github.com/apache/flink-ml
>> > [3] https://github.com/apache/flink-connectors
>> >
>> >
>> > On Mon, 10 Jan 2022 at 10:52, Jingsong Li  wrote:
>> >
>> > > Hi David, thanks for your suggestion.
>> > >
>> > > I think we should re-use as many common components with connectors as
>> > > possible. I don't fully understand what you mean, but for this project
>> > > I prefer to use Maven rather than Gradle.
>> > >
>> > > Best,
>> > > Jingsong
>> > >
>> > > On Fri, Jan 7, 2022 at 11:59 PM David Morávek  wrote:
>> > > >
>> > > > +1 for the separate repository under the Flink umbrella
>> > > >
>> > > > as we've already started creating more repositories with connectors,
>> > > would
>> > > > it be possible to re-use the same build infrastructure for this one?
>> > (eg.
>> > > > shared set of Gradle plugins that unify the build experience)?
>> > > >
>> > > > Best,
>> > > > D.
>> > > >
>> > > > On Fri, Jan 7, 2022 at 11:31 AM Jingsong Li 
>> > > wrote:
>> > > >
>> > > > > For more references on `store` and `storage`:
>> > > > >
>> > > > > For example,
>> > > > >
>> > > > > Rocksdb is a library that provides an embeddable, persistent
>> > key-value
>> > > > > store for fast storage. [1]
>> > > > >
>> > > > > Apache HBase [1] is an open-source, distributed, versioned,
>> > > > > column-oriented store modeled after Google' Bigtable. [2]
>> > > > >
>> > > > > [1] https://github.com/facebook/rocksdb
>> > > > > [2] https://github.com/apache/hbase
>> > > > >
>> > > > > Best,
>> > > > > Jingsong
>> > > > >
>> > > > > On Fri, Jan 7, 2022 at 6:17 PM Jingsong Li 
>> > > wrote:
>> > > > > >
>> > > > > > Thanks all,
>> > > > > >
>> > > > > > Combining everyone's comments, I recommend using
>> > `flink-table-store`:
>> > > > > >
>> > > > > > ## table
>> > > > > > something to do with table storage (From Till). Not only
>> > flink-table,
>> > > > > > but also for user-oriented tables.
>> > > > > >
>> > > > > > ## store vs storage
>> > > > > > - The first point I think, store is better pronounced, storage is
>> > > > > > three syllables while store is two syllables
>> > > > > > - Yes, store also stands for shopping. But I think the English
>> > > > > > polysemy is also quite interesting, a store to store various items,
>> > > it
>> > > > > > also feels interesting to represent the feeling that we want to do
>> > > > > > data storage.
>> > > > > > - The first feeling is, storage is a physical object or abstract
>> > > > > > concept, store is a software application or entity
>> > > > > >
>> > > > > > So I prefer `flink-table-store`, what do you think?
>> > > > > >
>> > > > > > (@_@ Naming is too difficult)
>> > > > > >
>> > > > > > Best,
>> > > > > > Jingsong
>> > > > > >
>> > > > > > On Fri, Jan 7, 2022 at 5:37 PM Konstantin Knauf > > >
>> > > > > wrote:
>> > > > > > >
>> > > > > > > +1 to a separate repository assuming this repository will still
>> > be
>> > > > > part of
>> > > > > > > Apache Flink (same PMC, Committers). I am not aware we have
>> > > something
>> > > > > like
>> > > > > > > "sub-projects" officially.
>> > > > > > >
>> > > > > > > I share Till and Timo's concerns regarding "store".
>> > > > > > >
>> > > > > > > On Fri, Jan 7, 2022 at 9:59 AM Till Rohrmann <
>> > trohrm...@apache.org
>> > > >
>> > > > > wrote:
>> > > > > > >
>> > > > > > > > +1 fo

Re: [DISCUSS] FLIP-208: Update KafkaSource to detect EOF based on de-serialized record

[Fabian Paul]

Hi Dong,

Thank you for updating the FLIP and making it applicable for all
sources. I am a bit unsure about the implementation part. I would
propose to add a source mixin interface that implements
`getRecordEvaluator` and sources that want to allow dynamically
stopping implement that interface.

Another question I had was how do we treat sources using the record
evaluator as bounded or unbounded?

Best,
Fabian

On Sat, Jan 8, 2022 at 11:52 AM Dong Lin  wrote:
>
> Hi Martijn and Qingsheng,
>
> The FLIP has been updated to extend the dynamic EOF support for the
> PulsarSource. I have not extended this feature to other sources yet since I
> am not sure it is a requirement to ensure feature consistency across
> different sources. Could you take another look?
>
> Thanks,
> Dong
>
> On Fri, Jan 7, 2022 at 11:49 PM Dong Lin  wrote:
>
> > Hi Martijn,
> >
> > Thanks for the comments! In general I agree we should avoid feature
> > sparsity.
> >
> > In this particular case, connectors are a bit different than most other
> > features in Flink. AFAIK, we plan to move connectors (including Kafka and
> > Pulsar) out of the Flink project in the future, which means that the
> > development of these connectors will be mostly de-centralized (outside of
> > Flink) and be up to their respective maintainers. While I agree that we
> > should provide API/infrastructure in Flink (as this FLIP does) to support
> > feature consistency across connectors, I am not sure we should own the
> > responsibility to actually update all connectors to achieve feature
> > consistency, given that we don't plan to do it in Flink anyway due to its
> > heavy burden.
> >
> > With that being said, I am happy to follow the community guideline if we
> > decide that connector-related FLIP should update every connector's API to
> > ensure feature consistency (to a reasonable extent). For example, in this
> > particular case, it looks like the EOF-detection feature can be applied to
> > every connector (including bounded sources). Is it still sufficient to just
> > update Kafka, Pulsar and Kinesis?
> >
> > Thanks,
> > Dong
> >
> > On Tue, Jan 4, 2022 at 3:31 PM Martijn Visser 
> > wrote:
> >
> >> Hi Dong,
> >>
> >> Thanks for writing the FLIP. It focusses only on the KafkaSource, but I
> >> would expect that if such a functionality is desired, it should be made
> >> available for all unbounded sources (for example, Pulsar and Kinesis). If
> >> it's only available for Kafka, I see it as if we're increasing feature
> >> sparsity while we actually want to decrease that. What do you think?
> >>
> >> Best regards,
> >>
> >> Martijn
> >>
> >> On Tue, 4 Jan 2022 at 08:04, Dong Lin  wrote:
> >>
> >> > Hi all,
> >> >
> >> > We created FLIP-208: Update KafkaSource to detect EOF based on
> >> > de-serialized records. Please find the KIP wiki in the link
> >> >
> >> >
> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-208%3A+Update+KafkaSource+to+detect+EOF+based+on+de-serialized+records
> >> > .
> >> >
> >> > This FLIP aims to address the use-case where users need to stop a Flink
> >> job
> >> > gracefully based on the content of de-serialized records observed in the
> >> > KafkaSource. This feature is needed by users who currently depend on
> >> > KafkaDeserializationSchema::isEndOfStream() to migrate their Flink job
> >> from
> >> > FlinkKafkaConsumer to KafkaSource.
> >> >
> >> > Could you help review this FLIP when you get time? Your comments are
> >> > appreciated!
> >> >
> >> > Cheers,
> >> > Dong
> >> >
> >>
> >

Re: [DISCUSS] FLIP-208: Update KafkaSource to detect EOF based on de-serialized record

[Fabian Paul]

Hi Dong,

I wouldn't change the org.apache.flink.api.connector.source.Source
interface because it either breaks existing sinks or we introduce it
as some kind of optional. I deem both options as not great. My idea is
to introduce a new interface that extends the Source. This way users
who want to develop a source that stops with the record evaluator can
implement the new interface. It also has the nice benefit that we can
give this new type of source a lower stability guarantee than Public
to allow some changes.
In the SourceOperatorFactory we can then access the record evaluator
from the respective sources and pass it to the source operator.

Hopefully, this makes sense. So far I did not find information about
the actual stopping logic in the FLIP maybe you had something
different in mind.

Best,
Fabian

On Tue, Jan 11, 2022 at 1:40 AM Dong Lin  wrote:
>
> Hi Fabian,
>
> Thanks for the comments!
>
> By "add a source mixin interface", are you suggesting to update
> the org.apache.flink.api.connector.source.Source interface to add the API
> "RecordEvaluator getRecordEvaluator()"? If so, it seems to add more
> public API and thus more complexity than the solution in the FLIP. Could
> you help explain more about the benefits of doing this?
>
> Regarding the 2nd question, I think this FLIP does not change whether
> sources are treated as bounded or unbounded. For example, the KafkaSource's
> boundedness will continue to be determined with the API
> KafkaSourceBuilder::setBounded(..) and
> KafkaSourceBuilder::setUnbounded(..). Does this answer your question?
>
> Thanks,
> Dong
>
>
>
>
>
>
>
>
>
> On Mon, Jan 10, 2022 at 8:01 PM Fabian Paul  wrote:
>
> > Hi Dong,
> >
> > Thank you for updating the FLIP and making it applicable for all
> > sources. I am a bit unsure about the implementation part. I would
> > propose to add a source mixin interface that implements
> > `getRecordEvaluator` and sources that want to allow dynamically
> > stopping implement that interface.
> >
> > Another question I had was how do we treat sources using the record
> > evaluator as bounded or unbounded?
> >
> > Best,
> > Fabian
> >
> > On Sat, Jan 8, 2022 at 11:52 AM Dong Lin  wrote:
> > >
> > > Hi Martijn and Qingsheng,
> > >
> > > The FLIP has been updated to extend the dynamic EOF support for the
> > > PulsarSource. I have not extended this feature to other sources yet
> > since I
> > > am not sure it is a requirement to ensure feature consistency across
> > > different sources. Could you take another look?
> > >
> > > Thanks,
> > > Dong
> > >
> > > On Fri, Jan 7, 2022 at 11:49 PM Dong Lin  wrote:
> > >
> > > > Hi Martijn,
> > > >
> > > > Thanks for the comments! In general I agree we should avoid feature
> > > > sparsity.
> > > >
> > > > In this particular case, connectors are a bit different than most other
> > > > features in Flink. AFAIK, we plan to move connectors (including Kafka
> > and
> > > > Pulsar) out of the Flink project in the future, which means that the
> > > > development of these connectors will be mostly de-centralized (outside
> > of
> > > > Flink) and be up to their respective maintainers. While I agree that we
> > > > should provide API/infrastructure in Flink (as this FLIP does) to
> > support
> > > > feature consistency across connectors, I am not sure we should own the
> > > > responsibility to actually update all connectors to achieve feature
> > > > consistency, given that we don't plan to do it in Flink anyway due to
> > its
> > > > heavy burden.
> > > >
> > > > With that being said, I am happy to follow the community guideline if
> > we
> > > > decide that connector-related FLIP should update every connector's API
> > to
> > > > ensure feature consistency (to a reasonable extent). For example, in
> > this
> > > > particular case, it looks like the EOF-detection feature can be
> > applied to
> > > > every connector (including bounded sources). Is it still sufficient to
> > just
> > > > update Kafka, Pulsar and Kinesis?
> > > >
> > > > Thanks,
> > > > Dong
> > > >
> > > > On Tue, Jan 4, 2022 at 3:31 PM Martijn Visser 
> > > > wrote:
> > > >
> > > >> Hi Dong,
> > > >>
> > > >> Thanks for writing the FLIP. It focusses only on th

Re: [DISCUSS] FLIP-208: Update KafkaSource to detect EOF based on de-serialized record

[Fabian Paul]

Hi Dong,

I think I am beginning to understand your idea. Since SourceReaderBase
is marked as PublicEvolving can you also update the FLIP with the
changes you want to make to it? Ideally, connector developers do not
have to change their SourceReaders to implement this new logic.

My idea was to introduce a second source interface that extends the
existing interface and offers only the method getRecordEvaluator().
The record evaluator is still passed as you have described through the
builder and at the end held by the source object. This way the source
framework can automatically use the evaluator without the need that
connector developers have to implement the complicated stopping logic
or change their SourceReaders.

Best,
Fabian


On Wed, Jan 12, 2022 at 2:22 AM Dong Lin  wrote:
>
> Hi Fabian,
>
> Thanks for the comments. Please see my reply inline.
>
> On Tue, Jan 11, 2022 at 11:46 PM Fabian Paul  wrote:
>
> > Hi Dong,
> >
> > I wouldn't change the org.apache.flink.api.connector.source.Source
> > interface because it either breaks existing sinks or we introduce it
> > as some kind of optional. I deem both options as not great. My idea is
> > to introduce a new interface that extends the Source. This way users
> > who want to develop a source that stops with the record evaluator can
> > implement the new interface. It also has the nice benefit that we can
> > give this new type of source a lower stability guarantee than Public
> > to allow some changes.
> >
>
> Currently the eofRecodEvaluator can be passed from
> KafkaSourceBuilder/PulsarSourceBuilder
> to SingleThreadMultiplexSourceReaderBase and SourceReaderBase. This
> approach also allows developers who want to develop a source that stops
> with the record evaluator to implement the new feature. Adding a new
> interface could increase the complexity in our interface and
> infrastructure. I am not sure if it has added benefits compared to the
> existing proposal. Could you explain more?
>
> I am not very sure what "new type of source a lower stability guarantee"
> you are referring to. Could you explain more? It looks like a new feature
> not mentioned in the FLIP. If the changes proposed in this FLIP also
> support the feature you have in mind, could we discuss this in a separate
> FLIP?
>
> In the SourceOperatorFactory we can then access the record evaluator
> > from the respective sources and pass it to the source operator.
> >
> > Hopefully, this makes sense. So far I did not find information about
> > the actual stopping logic in the FLIP maybe you had something
> > different in mind.
> >
>
> By "actual stopping logic", do you mean an example implementation of the
> RecordEvalutor? I think the use-case is described in the motivation
> section, which is about a pipeline processing stock transaction data.
>
> We can support this use-case with this FLIP, by implementing this
> RecordEvaluator that stops reading data from a split when there is a
> message that says "EOF". Users can trigger this feature by sending messages
> with "EOF" in the payload to all partitions of the source Kafka topic.
>
> Does this make sense?
>
>
> >
> > Best,
> > Fabian
> >
> > On Tue, Jan 11, 2022 at 1:40 AM Dong Lin  wrote:
> > >
> > > Hi Fabian,
> > >
> > > Thanks for the comments!
> > >
> > > By "add a source mixin interface", are you suggesting to update
> > > the org.apache.flink.api.connector.source.Source interface to add the API
> > > "RecordEvaluator getRecordEvaluator()"? If so, it seems to add more
> > > public API and thus more complexity than the solution in the FLIP. Could
> > > you help explain more about the benefits of doing this?
> > >
> > > Regarding the 2nd question, I think this FLIP does not change whether
> > > sources are treated as bounded or unbounded. For example, the
> > KafkaSource's
> > > boundedness will continue to be determined with the API
> > > KafkaSourceBuilder::setBounded(..) and
> > > KafkaSourceBuilder::setUnbounded(..). Does this answer your question?
> > >
> > > Thanks,
> > > Dong
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Mon, Jan 10, 2022 at 8:01 PM Fabian Paul  wrote:
> > >
> > > > Hi Dong,
> > > >
> > > > Thank you for updating the FLIP and making it applicable for all
> > > > sources. I am a bit unsure about the implementation part. I would
> > > > propose to add a sou

Re: [VOTE] Deprecate NiFi connector

[Fabian Paul]

Thanks for driving the deprecation efforts.

+1 (binding)

Best,
Fabian

On Mon, Jan 31, 2022 at 11:47 AM Martijn Visser  wrote:
>
> Hi everyone,
>
> I would like to open up a vote to deprecate NiFi in Flink 1.15 and remove
> it in the next version. I've previously mentioned that we were looking for
> maintainers for NiFi, but so far none have come forward [1].
>
> The vote will last for at least 72 hours, and will be accepted by a
> consensus of active committers.
>
> Best regards,
>
> Martijn Visser
> https://twitter.com/MartijnVisser82
>
> [1] https://lists.apache.org/thread/1vs3wmk66vsq6l4npjsfzltft4tz5tkq

Re: [VOTE] Remove Twitter connector

[Fabian Paul]

This connector is really a relict of the past.

+1 (binding)

Best,
Fabian

On Mon, Jan 31, 2022 at 11:47 AM Martijn Visser  wrote:
>
> Hi everyone,
>
> I would like to open up a vote to remove the Twitter connector in Flink
> 1.15. This was brought up previously for a discussion [1].
>
> The vote will last for at least 72 hours, and will be accepted by
> a consensus of active committers.
>
> Best regards,
>
> Martijn Visser
> https://twitter.com/MartijnVisser82
>
> [1] https://lists.apache.org/thread/7sdvp4hj93rh0cz8r50800stzrpgkdm2

Re: [ANNOUNCE] New Flink PMC members: Igal Shilman, Konstantin Knauf and Yun Gao

[Fabian Paul]

Congrats to all three of you, well deserved.

Best,
Fabian

On Wed, Feb 16, 2022 at 2:23 PM Robert Metzger  wrote:
>
> Hi all,
>
> I would like to formally announce a few new Flink PMC members on the dev@
> list. The PMC has not done a good job of always announcing new PMC members
> (and committers) recently. I'll try to keep an eye on this in the future to
> improve the situation.
>
> Nevertheless, I'm very happy to announce some very active community members
> as new PMC members:
>
> - Igal Shilman, added to the PMC in October 2021
> - Konstantin Knauf, added to the PMC in January 2022
> - Yun Gao, added to the PMC in February 2022
>
> Please join me in welcoming them to the Flink PMC!
>
> Best,
> Robert

Re: [DISCUSS] Release Flink 1.14.4

[Fabian Paul]

Hi Konstantin,

Thanks for all the efforts driving the release. From my side,
FLINK-26018 can also be seen as some kind of new feature that was
planned but never implemented. Of course, it would be great to have it
because it currently blocks the migration from the FlinkKafkaConsumer
to the KafkaSource.

Unfortunately, I also found another blocker issue[1]. I try to work on
it this week.

Best,
Fabian

[1] https://issues.apache.org/jira/browse/FLINK-26018

On Tue, Feb 22, 2022 at 8:53 AM Konstantin Knauf  wrote:
>
> Hi everyone,
>
> We still have two more critical issues for Flink 1.14. I'd like to understand 
> if it makes sense to push one of these to the next release.
>
> https://issues.apache.org/jira/browse/FLINK-24607
> @Becket: this seems to be missing the backport, correct?
>
> https://issues.apache.org/jira/browse/FLINK-26018
> @Qingsheng, Fabian, Piotr: I saw that Piotr and Fabian raised some general 
> concerns about the solution. Does it make sense to wait for the resolution of 
> this ticket for Flink 1.14.4?
>
> Thanks,
>
> Konstantin
>
> On Sat, Feb 12, 2022 at 12:42 PM Roman Khachatryan  wrote:
>>
>> Hi,
>>
>> +1 for the proposal.
>>
>> Thanks for volunteering Konstantin!
>>
>> Regards,
>> Roman
>>
>>
>>
>> On Fri, Feb 11, 2022 at 3:00 PM Till Rohrmann  wrote:
>> >
>> > +1 for a 1.14.4 release. The release-1.14 branch already includes 36 fixed
>> > issues, some of them quite severe.
>> >
>> > Cheers,
>> > Till
>> >
>> > On Fri, Feb 11, 2022 at 1:23 PM Martijn Visser 
>> > wrote:
>> >
>> > > Hi Konstantin,
>> > >
>> > > Thanks for opening the discussion. I think FLINK-25732 does indeed 
>> > > warrant
>> > > a speedy Flink 1.14.4 release. I would indeed also like to include
>> > > FLINK-26018.
>> > >
>> > > Best regards,
>> > >
>> > > Martijn
>> > >
>> > > Op vr 11 feb. 2022 om 10:29 schreef Konstantin Knauf 
>> > >
>> > > > Hi everyone,
>> > > >
>> > > > what do you think about a timely Flink 1.14.4 in order to release the 
>> > > > fix
>> > > > for https://issues.apache.org/jira/browse/FLINK-25732. Currently, the
>> > > > landing page of Flink Web User Interface is not working when there are
>> > > > standby Jobmanagers.
>> > > >
>> > > > In addition, I propose to wait for
>> > > > https://issues.apache.org/jira/browse/FLINK-26018 to be resolved.
>> > > >
>> > > > I can volunteer as release manager.
>> > > >
>> > > > Cheers,
>> > > >
>> > > > Konstantin
>> > > >
>> > > > --
>> > > >
>> > > > Konstantin Knauf
>> > > >
>> > > > https://twitter.com/snntrable
>> > > >
>> > > > https://github.com/knaufk
>> > > >
>> > > --
>> > >
>> > > Martijn Visser | Product Manager
>> > >
>> > > mart...@ververica.com
>> > >
>> > > 
>> > >
>> > >
>> > > Follow us @VervericaData
>> > >
>> > > --
>> > >
>> > > Join Flink Forward  - The Apache Flink
>> > > Conference
>> > >
>> > > Stream Processing | Event Driven | Real Time
>> > >
>
>
>
> --
>
> Konstantin Knauf
>
> https://twitter.com/snntrable
>
> https://github.com/knaufk

Re: [DISCUSS] Release Flink 1.14.4

[Fabian Paul]

EDIT: Wrong link before https://issues.apache.org/jira/browse/FLINK-26304

On Tue, Feb 22, 2022 at 4:55 PM Fabian Paul  wrote:
>
> Hi Konstantin,
>
> Thanks for all the efforts driving the release. From my side,
> FLINK-26018 can also be seen as some kind of new feature that was
> planned but never implemented. Of course, it would be great to have it
> because it currently blocks the migration from the FlinkKafkaConsumer
> to the KafkaSource.
>
> Unfortunately, I also found another blocker issue[1]. I try to work on
> it this week.
>
> Best,
> Fabian
>
> [1] https://issues.apache.org/jira/browse/FLINK-26018
>
> On Tue, Feb 22, 2022 at 8:53 AM Konstantin Knauf  wrote:
> >
> > Hi everyone,
> >
> > We still have two more critical issues for Flink 1.14. I'd like to 
> > understand if it makes sense to push one of these to the next release.
> >
> > https://issues.apache.org/jira/browse/FLINK-24607
> > @Becket: this seems to be missing the backport, correct?
> >
> > https://issues.apache.org/jira/browse/FLINK-26018
> > @Qingsheng, Fabian, Piotr: I saw that Piotr and Fabian raised some general 
> > concerns about the solution. Does it make sense to wait for the resolution 
> > of this ticket for Flink 1.14.4?
> >
> > Thanks,
> >
> > Konstantin
> >
> > On Sat, Feb 12, 2022 at 12:42 PM Roman Khachatryan  wrote:
> >>
> >> Hi,
> >>
> >> +1 for the proposal.
> >>
> >> Thanks for volunteering Konstantin!
> >>
> >> Regards,
> >> Roman
> >>
> >>
> >>
> >> On Fri, Feb 11, 2022 at 3:00 PM Till Rohrmann  wrote:
> >> >
> >> > +1 for a 1.14.4 release. The release-1.14 branch already includes 36 
> >> > fixed
> >> > issues, some of them quite severe.
> >> >
> >> > Cheers,
> >> > Till
> >> >
> >> > On Fri, Feb 11, 2022 at 1:23 PM Martijn Visser 
> >> > wrote:
> >> >
> >> > > Hi Konstantin,
> >> > >
> >> > > Thanks for opening the discussion. I think FLINK-25732 does indeed 
> >> > > warrant
> >> > > a speedy Flink 1.14.4 release. I would indeed also like to include
> >> > > FLINK-26018.
> >> > >
> >> > > Best regards,
> >> > >
> >> > > Martijn
> >> > >
> >> > > Op vr 11 feb. 2022 om 10:29 schreef Konstantin Knauf 
> >> > > 
> >> > >
> >> > > > Hi everyone,
> >> > > >
> >> > > > what do you think about a timely Flink 1.14.4 in order to release 
> >> > > > the fix
> >> > > > for https://issues.apache.org/jira/browse/FLINK-25732. Currently, the
> >> > > > landing page of Flink Web User Interface is not working when there 
> >> > > > are
> >> > > > standby Jobmanagers.
> >> > > >
> >> > > > In addition, I propose to wait for
> >> > > > https://issues.apache.org/jira/browse/FLINK-26018 to be resolved.
> >> > > >
> >> > > > I can volunteer as release manager.
> >> > > >
> >> > > > Cheers,
> >> > > >
> >> > > > Konstantin
> >> > > >
> >> > > > --
> >> > > >
> >> > > > Konstantin Knauf
> >> > > >
> >> > > > https://twitter.com/snntrable
> >> > > >
> >> > > > https://github.com/knaufk
> >> > > >
> >> > > --
> >> > >
> >> > > Martijn Visser | Product Manager
> >> > >
> >> > > mart...@ververica.com
> >> > >
> >> > > <https://www.ververica.com/>
> >> > >
> >> > >
> >> > > Follow us @VervericaData
> >> > >
> >> > > --
> >> > >
> >> > > Join Flink Forward <https://flink-forward.org/> - The Apache Flink
> >> > > Conference
> >> > >
> >> > > Stream Processing | Event Driven | Real Time
> >> > >
> >
> >
> >
> > --
> >
> > Konstantin Knauf
> >
> > https://twitter.com/snntrable
> >
> > https://github.com/knaufk

Re: [DISCUSS] Looking for maintainers for Cassandra connector or decide to remove connector

[Fabian Paul]

Hi Marco,

Great to hear that you put some thought into the topic. Judging from
the past we already tried once to support multiple external versions
within one connector (ElasticSearch) and it complicates things a lot.
So if it makes your development easier to create a different module
that should be fine. Usually, we try not to break/remove existing
connectors if they are not deprecated yet. In your case, I'd recommend
first developing the unified connector and then deprecating the old
connector.

Regarding the external connector repository, we plan to only
externalize implementations that are based on the unified Source and
Sink interfaces so that we can slowly deprecate the old interfaces.

Best,
Fabian

Re: [ANNOUNCE] Flink 1.15 Feature Freeze

[Fabian Paul]

Hi all,

I would like to merge the following PR [1]. It has been approved
before the feature freeze but no one had time to merge it,
unfortunately. The feature is very contained and only adds a simple
capability to the Elastic connector when used with Flink SQL. If there
are no concerns until end of day I am going to merge it.

Best,
Fabian

[1] https://github.com/apache/flink/pull/18058

On Fri, Feb 18, 2022 at 12:37 PM Till Rohrmann  wrote:
>
> Thanks for letting us know Wenlong. Since the SQL client testing hasn't
> really started yet, I think it is ok to merge this PR.
>
> Cheers,
> Till
>
> On Thu, Feb 17, 2022 at 2:32 PM wenlong.lwl  wrote:
>
> > Hi,all, I want to merge a pr (https://github.com/apache/flink/pull/18363)
> > belonging to FLIP-190, which was approved yesterday, but not merged before
> > code freeze because of CI queueing and failed by other changes.
> > WDYT?
> >
> > Best,
> > Wenlong
> >
> > On Thu, 17 Feb 2022 at 02:08, Till Rohrmann  wrote:
> >
> > > Hi everyone,
> > >
> > > The deadline for merging new features for Flink 1.15 has passed.
> > >
> > > * From now on, only bug-fixes and documentation fixes / improvements are
> > > allowed to be merged into the master branch.
> > >
> > > * New features merged after this point can be reverted. If you need an
> > > exception to this rule, please open a discussion on dev@ list and reach
> > > out
> > > to us.
> > >
> > > We plan to wait for the master branch to get a bit more stabilized before
> > > cutting the "release-1.15" branch, in order to reduce the overhead of
> > > having to manage two branches. That also means potentially delaying
> > merging
> > > new features for Flink 1.16 into the master branch. If you are blocked on
> > > this, please let us know and we can come up with a compromise for the
> > > branch cutting time.
> > >
> > > What you can do to help with the release testing phase:
> > >
> > > * The first release testing sync will be on *February 22, 9am CET*.
> > > Everyone is welcome to join. The link can be found on the release wiki
> > page
> > > [1].
> > >
> > > * Please prepare for the release testing by creating Jira tickets for
> > > documentation and testing tasks for the new features. Tickets should be
> > > opened with Priority Blocker, FixVersion 1.15.0 and Label release-testing
> > > (testing tasks only).
> > >
> > > * There are currently 92 test-stability issues affecting the 1.15.0
> > release
> > > [2]. It is greatly appreciated if you can help address some of them.
> > >
> > > Cheers,
> > > Joe, Yun & Till
> > >
> > > [1] https://cwiki.apache.org/confluence/display/FLINK/1.15+Release
> > > [2] https://issues.apache.org/jira/issues/?filter=12351363
> > >
> >

Re: [VOTE] Release 1.9.3, release candidate #1

[Fabian Paul]

+1 (non-binding)

- Verified signature
- Built from source (Java8)
- Run custom jobs on Kubernetes

Regards,
Fabian

> On 18. Apr 2020, at 04:37, Dian Fu  wrote:
> 
> Hi everyone,
> 
> Please review and vote on the release candidate #1 for the version 1.9.3,
> as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
> 
> 
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release and binary convenience releases to be
> deployed to dist.apache.org [2], which are signed with the key with
> fingerprint 6B6291A8502BA8F0913AE04DDEB95B05BF075300 [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "release-1.9.3-rc1" [5],
> * website pull request listing the new release and adding announcement blog
> post [6].
> 
> The vote will be open for at least 72 hours. It is adopted by majority 
> approval, with at least 3 PMC affirmative votes.
> 
> Thanks,
> Dian
> 
> [1] 
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12346867
> [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.9.3-rc1/
> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> [4] https://repository.apache.org/content/repositories/orgapacheflink-1353/
> [5] 
> https://github.com/apache/flink/commit/6d23b2c38c7a8fd8855063238c744923e1985a63
> [6] https://github.com/apache/flink-web/pull/329

Re: [ANNOUNCE] Apache Flink 1.11.0, release candidate #2

[Fabian Paul]

Hi,

Thanks for the great efforts in preparing the second rc. I was just going 
through the published artifacts and it seems that some are missing in the 
latest release.

In comparison you can look at 

https://repository.apache.org/content/repositories/orgapacheflink-1370/org/apache/flink/
 

 with the full list of artifacts for the first rc and 
https://repository.apache.org/content/repositories/orgapacheflink-1374/org/apache/flink/
 

 with only a subset for the second one.

Did you only upload the artifacts which have not been changed?

Best,
Fabian

Re: [ANNOUNCE] Apache Flink 1.11.0, release candidate #2

[Fabian Paul]

Hi,

Thanks again for uploading the missing artifacts. Unfortunately this rc does 
not fully compile due to [1].

Would it be possible for testing purposed to quickly include this fix into the 
rc or do you think it is necessary to open a complete new one?


[1] https://issues.apache.org/jira/browse/FLINK-18411 


Best,
Fabian

[jira] [Created] (FLINK-27480) KafkaSources sharing the groupId might lead to InstanceAlreadyExistException warning

[Fabian Paul (Jira)]

Fabian Paul created FLINK-27480:
---

 Summary: KafkaSources sharing the groupId might lead to 
InstanceAlreadyExistException warning
 Key: FLINK-27480
 URL: https://issues.apache.org/jira/browse/FLINK-27480
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Kafka
Affects Versions: 1.14.4, 1.15.0, 1.16.0
Reporter: Fabian Paul


More and more frequently, users ran into an issue by not correctly configuring 
the KafkaSource 
([https://stackoverflow.com/questions/72026997/flink-instancealreadyexistsexception-while-migrating-to-the-kafkasource)]
 and setting non-distinguishable groupIds for the source. Internally the used 
KafkaConsumer tries to register with the metric system and incorporates the 
groupId as part of the metric, leading to name collision.

We should update the documentation to explain the situation properly.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (FLINK-27484) Reduce ArchUnit violations in the project

[Fabian Paul (Jira)]

Fabian Paul created FLINK-27484:
---

 Summary: Reduce ArchUnit violations in the project
 Key: FLINK-27484
 URL: https://issues.apache.org/jira/browse/FLINK-27484
 Project: Flink
  Issue Type: Improvement
  Components: API / Core, Connectors / Common, Runtime / Configuration
Reporter: Fabian Paul


When ArchUnit was introduced we deliberately ignored the existing violations. 
This is the umbrella ticket to hold the efforts to reduce the exposure.

 

In the long run, this gives our users clarity about using or not using a 
certain part of the codebase.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (FLINK-27486) Reduce ArchUnit violations in connector base module

[Fabian Paul (Jira)]

Fabian Paul created FLINK-27486:
---

 Summary: Reduce ArchUnit violations in connector base module
 Key: FLINK-27486
 URL: https://issues.apache.org/jira/browse/FLINK-27486
 Project: Flink
  Issue Type: Sub-task
  Components: Connectors / Common
Affects Versions: 1.16.0
Reporter: Fabian Paul






--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (FLINK-27493) Forward all numeric Kafka metrics to Flink's metrics system

[Fabian Paul (Jira)]

Fabian Paul created FLINK-27493:
---

 Summary: Forward all numeric Kafka metrics to Flink's metrics 
system
 Key: FLINK-27493
 URL: https://issues.apache.org/jira/browse/FLINK-27493
 Project: Flink
  Issue Type: New Feature
  Components: Connectors / Kafka
Affects Versions: 1.16.0
Reporter: Fabian Paul


With the upgrade of the Kafka version to 2.8, it is now possible to access more 
metrics from the KafkaConsumer/KafkaProducer. So far we only forward metrics 
that are of type Double but ignore other numeric values. It might be worthwhile 
to forward all numerics metrics.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (FLINK-29509) Set correct subtaskId during recovery of committables

[Fabian Paul (Jira)]

Fabian Paul created FLINK-29509:
---

 Summary: Set correct subtaskId during recovery of committables
 Key: FLINK-29509
 URL: https://issues.apache.org/jira/browse/FLINK-29509
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Common
Affects Versions: 1.15.2, 1.17.0, 1.16.1
Reporter: Fabian Paul


When we recover the `CheckpointCommittableManager` we ignore the subtaskId it 
is recovered on. 
[https://github.com/apache/flink/blob/d191bda7e63a2c12416cba56090e5cd75426079b/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CheckpointCommittableManagerImpl.java#L58]

This becomes a problem when a sink uses a post-commit topology because multiple 
committer operators might forward committable summaries coming from the same 
subtaskId.

 

It should be possible to use the subtaskId already present in the 
`CommittableCollector` when creating the `CheckpointCommittableManager`s.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29512) Align SubtaskCommittableManager checkpointId with CheckpointCommittableManagerImpl checkpointId during recovery

[Fabian Paul (Jira)]

Fabian Paul created FLINK-29512:
---

 Summary: Align SubtaskCommittableManager checkpointId with 
CheckpointCommittableManagerImpl checkpointId during recovery
 Key: FLINK-29512
 URL: https://issues.apache.org/jira/browse/FLINK-29512
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Common
Affects Versions: 1.15.2, 1.17.0, 1.16.1
Reporter: Fabian Paul


Similar to the issue described in 
https://issues.apache.org/jira/browse/FLINK-29509 during the recovery of 
committables, the subtaskCommittables checkpointId is set to always 1 
[https://github.com/apache/flink/blob/9152af41c2d401e5eacddee1bb10d1b6bea6c61a/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommittableCollectorSerializer.java#L193]
 while the holding CheckpointCommittableManager is initialized with the 
checkpointId that is written into state 
[https://github.com/apache/flink/blob/9152af41c2d401e5eacddee1bb10d1b6bea6c61a/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommittableCollectorSerializer.java#L155
 
.|https://github.com/apache/flink/blob/9152af41c2d401e5eacddee1bb10d1b6bea6c61a/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/sink/committables/CommittableCollectorSerializer.java#L155.]

 

This leads to that during a recovery, the post-commit topology will receive a 
committable summary with the recovered checkpoint id and multiple 
`CommittableWithLinage`s with the reset checkpointId causing orphaned 
`CommittableWithLinages` without a `CommittableSummary` failing the job.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-29583) Ensure correct subtaskId and checkpointId is set during committer state migration

[Fabian Paul (Jira)]

Fabian Paul created FLINK-29583:
---

 Summary: Ensure correct subtaskId and checkpointId is set during 
committer state migration
 Key: FLINK-29583
 URL: https://issues.apache.org/jira/browse/FLINK-29583
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Common
Affects Versions: 1.15.2, 1.16.0, 1.17.0
Reporter: Fabian Paul


We already discovered two problems during recovery of committers when a post 
commit topology is used

 
 * https://issues.apache.org/jira/browse/FLINK-29509
 * https://issues.apache.org/jira/browse/FLINK-29512

Both problems also apply when recovering Flink 1.14 unified sinks committer 
state and migrate it to the extended unified model.

As part of this ticket we should fix both issues for the migration and also 
increase the test coverage for the migration cases i.e. add test cases in 
CommitterOperatorTest and CommittableCollectorSerializerTest.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-30238) Unified Sink committer does not clean up state on final savepoint

[Fabian Paul (Jira)]

Fabian Paul created FLINK-30238:
---

 Summary: Unified Sink committer does not clean up state on final 
savepoint
 Key: FLINK-30238
 URL: https://issues.apache.org/jira/browse/FLINK-30238
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Common
Affects Versions: 1.15.3, 1.17.0, 1.16.1
Reporter: Fabian Paul


During stop-with-savepoint the committer only commits the pending committables 
on notifyCheckpointComplete.

This has several downsides.
 * Last committableSummary has checkpoint id LONG.MAX and is never cleared from 
the state leading to that stop-with-savepoint does not work when the pipeline 
recovers from a savepoint 
 * While the committables are committed during stop-with-savepoint they are not 
forwarded to post-commit topology, potentially losing data and preventing to 
close open transactions.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35419) scan.bounded.latest-offset makes queries never finish if the latest message is a EndTxn Kafka marker

[Fabian Paul (Jira)]

Fabian Paul created FLINK-35419:
---

 Summary: scan.bounded.latest-offset makes queries never finish if 
the latest message is a EndTxn Kafka marker
 Key: FLINK-35419
 URL: https://issues.apache.org/jira/browse/FLINK-35419
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kafka
Affects Versions: 1.19.0, 1.17.0, 1.16.0, 1.8.0
Reporter: Fabian Paul


When running the kafka connector in bounded mode, the stop condition can be 
defined as the latest offset when the job starts.

 

Unfortunately, Kafka's latest offset calculation also includes special marker 
records, such as transaction markers, in the overall count.

 

When Flink waits for a job to finish, it compares the number of records read 
until the point with the original latest offset [1]. Since the consumer will 
never see the special marker records, the latest offset is never reached, and 
the job gets stuck. 

 

To reproduce the issue, you can write into a Kafka topic and make sure that the 
latest record is a transaction end event. Afterwards you can start a Flink job 
configured with `scan.bounded.latest-offset` pointing to that topic.

 

[1]https://github.com/confluentinc/flink/blob/59c5446c4aac0d332a21b456f4a3f82576104b80/flink-connectors/confluent-connector-kafka/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/reader/KafkaPartitionSplitReader.java#L128



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-16525) TwoPhaseCommitSinkFunction subtask logs misleading name

[Fabian Paul (Jira)]

Fabian Paul created FLINK-16525:
---

 Summary: TwoPhaseCommitSinkFunction subtask logs misleading name
 Key: FLINK-16525
 URL: https://issues.apache.org/jira/browse/FLINK-16525
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Task
Reporter: Fabian Paul


The current name() function in TwoPhaseCommitSinkFunction tries to describe the 
currently running subtask with its class name, the index of the subtask and the 
number of parallel subtasks.

Since the starting index of the subtask is 0, and the starting number for the 
parallelism is 1, it could lead to the following log message.
{code:java}
15:59:41,448 INFO  
org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction  - 
FlinkKafkaProducer 0/1 - checkpoint 1 complete, committing transaction 
TransactionHolder{handle=KafkaTransactionState [transactionalId=null, 
producerId=-1, epoch=-1], transactionStartTime=1583852371370} from checkpoint 1
{code}
Although only one subtask is running it describes the subtask as 0/1 which 
might indicate more than one subtask.

I would suggest incrementing the first number after the class name by 1 to 
better indicate how many subtasks are running.

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16553) KafkaFetcher topic/partition metrics

[Fabian Paul (Jira)]

Fabian Paul created FLINK-16553:
---

 Summary: KafkaFetcher topic/partition metrics
 Key: FLINK-16553
 URL: https://issues.apache.org/jira/browse/FLINK-16553
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Kafka, Runtime / Metrics
Reporter: Fabian Paul


When using the Kafka universal connector, currently not all KafkaFetcher 
metrics ([link| 
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/internals/FetcherMetricsRegistry.java]
 which are exposed through the KafkaConsumer are accessible within the Flink 
metrics system.

Especially, all metrics which are related to topics and partitions are not 
available. The KafkaConsumer internally only registers those metrics after it 
has fetched some records.

Unfortunately, at the moment Flink only checks the available metrics right 
after the initialization of the KafkaConsumer when no records are polled, yet.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16570) Difficulties to select correct metric with long name in dropdown of Flink UI task menu

[Fabian Paul (Jira)]

Fabian Paul created FLINK-16570:
---

 Summary: Difficulties to select correct metric with long name in 
dropdown of Flink UI task menu
 Key: FLINK-16570
 URL: https://issues.apache.org/jira/browse/FLINK-16570
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Web Frontend
Reporter: Fabian Paul
 Attachments: metrics_dropdown.png

As seen in the attached image it is currently difficult to select the correct 
metrics when the metric name exceeds the length of the dropdown because the 
full name cannot be seen.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-16883) No support for log4j2 configuration formats besides properties

[Fabian Paul (Jira)]

Fabian Paul created FLINK-16883:
---

 Summary: No support for log4j2 configuration formats besides 
properties
 Key: FLINK-16883
 URL: https://issues.apache.org/jira/browse/FLINK-16883
 Project: Flink
  Issue Type: Improvement
  Components: Command Line Client
Affects Versions: 1.11.0
Reporter: Fabian Paul


If `flink.console.sh` is used to start a Flink cluster the env java opts 
precede the log settings. 
([link|[https://github.com/apache/flink/blob/1444fd68115594b872201242886b4d789f4b26a5/flink-dist/src/main/flink-bin/bin/flink-console.sh#L73]]

This way the log settings `log4.configurationFile` will always overwrite 
previous keys. Since the `log4j.configurationFile` is set to `log4j.properties` 
it is not possible to leverage other formats than properties for the 
configuration.

 

My proposal would be to switch the order of the configurations that the log 
settings precede the env java opts. Users could then overwrite the default file 
with their configurations.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-21879) ActiveResourceManagerTest.testWorkerRegistrationTimeoutNotCountingAllocationTime fails on AZP

[Fabian Paul (Jira)]

Fabian Paul created FLINK-21879:
---

 Summary: 
ActiveResourceManagerTest.testWorkerRegistrationTimeoutNotCountingAllocationTime
 fails on AZP
 Key: FLINK-21879
 URL: https://issues.apache.org/jira/browse/FLINK-21879
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.13.0
Reporter: Fabian Paul


https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=15047&view=logs&j=34f41360-6c0d-54d3-11a1-0292a2def1d9&t=2d56e022-1ace-542f-bf1a-b37dd63243f2&l=6760
{code:java}
[ERROR] 
testWorkerRegistrationTimeoutNotCountingAllocationTime(org.apache.flink.runtime.resourcemanager.active.ActiveResourceManagerTest)
  Time elapsed: 0.388 s  <<< FAILURE!
java.lang.AssertionError: 

Expected: an instance of 
org.apache.flink.runtime.registration.RegistrationResponse$Success
 but:  is a 
org.apache.flink.runtime.taskexecutor.TaskExecutorRegistrationRejection
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
at org.junit.Assert.assertThat(Assert.java:956)
at org.junit.Assert.assertThat(Assert.java:923)
at 
org.apache.flink.runtime.resourcemanager.active.ActiveResourceManagerTest$13.lambda$new$2(ActiveResourceManagerTest.java:789)
at 
org.apache.flink.runtime.resourcemanager.active.ActiveResourceManagerTest$Context.runTest(ActiveResourceManagerTest.java:857)
at 
org.apache.flink.runtime.resourcemanager.active.ActiveResourceManagerTest$13.(ActiveResourceManagerTest.java:770)
at 
org.apache.flink.runtime.resourcemanager.active.ActiveResourceManagerTest.testWorkerRegistrationTimeoutNotCountingAllocationTime(ActiveResourceManagerTest.java:753)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at 
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)

{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22256) Persist checkpoint type information

[Fabian Paul (Jira)]

Fabian Paul created FLINK-22256:
---

 Summary: Persist checkpoint type information
 Key: FLINK-22256
 URL: https://issues.apache.org/jira/browse/FLINK-22256
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Checkpointing
Reporter: Fabian Paul


As a user, it is retrospectively difficult to determine what kind of checkpoint 
(i.e. incremental, unaligned, ...) was performed when looking only at the 
persisted checkpoint metadata.

The only way would be to look into the execution configuration of the job which 
might not be available anymore and can be scattered across the application code 
and cluster configuration.

It would be highly beneficial if such information would be part of the 
persisted metadata to not track these external pointers.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22257) Clarify Flink ConfigOptions Usage

[Fabian Paul (Jira)]

Fabian Paul created FLINK-22257:
---

 Summary: Clarify Flink ConfigOptions Usage
 Key: FLINK-22257
 URL: https://issues.apache.org/jira/browse/FLINK-22257
 Project: Flink
  Issue Type: Improvement
Reporter: Fabian Paul


For users, it is hard to determine which ConfigOptions are relevant for the 
different stages of a Flink application.

Beginning from the translation of the user program to the execution on the 
cluster. In particular which options can be configured through the different 
channels.
 * Cluster configuration (i.e. flink-conf.yaml)
 * Application configuration, code-based

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22434) Dispatcher does not store suspended jobs in execution graph store

[Fabian Paul (Jira)]

Fabian Paul created FLINK-22434:
---

 Summary: Dispatcher does not store suspended jobs in execution 
graph store
 Key: FLINK-22434
 URL: https://issues.apache.org/jira/browse/FLINK-22434
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Reporter: Fabian Paul
Assignee: Fabian Paul
 Fix For: 1.11.4, 1.14.0, 1.12.3, 1.13.1


Only globally terminated jobs are currently stored in the execution graph store 
after termination. In case the JobManager is shutdown and jobs are still 
running, these jobs will be suspended which is a non-globally terminated state.

The problem surfaces when a user tries to access information about the job 
during termination, leading to a job not found response. By storing all 
terminated jobs in the execution graph store this should be fixed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-22696) Enable Confluent Schema Registry Test on jdk 11

[Fabian Paul (Jira)]

Fabian Paul created FLINK-22696:
---

 Summary: Enable Confluent Schema Registry Test on jdk 11
 Key: FLINK-22696
 URL: https://issues.apache.org/jira/browse/FLINK-22696
 Project: Flink
  Issue Type: Improvement
  Components: Test Infrastructure
Reporter: Fabian Paul
Assignee: Fabian Paul






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-23123) Implement at-least-once Kafka Sink

[Fabian Paul (Jira)]

Fabian Paul created FLINK-23123:
---

 Summary: Implement at-least-once Kafka Sink
 Key: FLINK-23123
 URL: https://issues.apache.org/jira/browse/FLINK-23123
 Project: Flink
  Issue Type: Sub-task
Reporter: Fabian Paul






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-23124) Implement exactly-once Kafka Sink

[Fabian Paul (Jira)]

Fabian Paul created FLINK-23124:
---

 Summary: Implement exactly-once Kafka Sink
 Key: FLINK-23124
 URL: https://issues.apache.org/jira/browse/FLINK-23124
 Project: Flink
  Issue Type: Sub-task
Reporter: Fabian Paul






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-23248) SinkWriter is not close when failing

[Fabian Paul (Jira)]

Fabian Paul created FLINK-23248:
---

 Summary: SinkWriter is not close when failing
 Key: FLINK-23248
 URL: https://issues.apache.org/jira/browse/FLINK-23248
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Task
Affects Versions: 1.12.4, 1.13.1, 1.14.0
Reporter: Fabian Paul


Currently the SinkWriter is only closed when the operator finishes in 
`AbstractSinkWriterOperator#close()` but we also must close the SinkWrite on 
`AbstractSinkWriterOperator#dispose()` to release possible acquired resources 
when failing

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-23639) Migrate Table API to new KafkaSink

[Fabian Paul (Jira)]

Fabian Paul created FLINK-23639:
---

 Summary: Migrate Table API to new KafkaSink
 Key: FLINK-23639
 URL: https://issues.apache.org/jira/browse/FLINK-23639
 Project: Flink
  Issue Type: Sub-task
  Components: Connectors / Kafka, Table SQL / API
Reporter: Fabian Paul


With the KafkaSink ported to FLIP-143 we should also adapt the Table API to 
leverage the new KafkaSink



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-23640) Create a KafkaRecordSerializationSchemas valueOnly helper

[Fabian Paul (Jira)]

Fabian Paul created FLINK-23640:
---

 Summary: Create a KafkaRecordSerializationSchemas valueOnly helper
 Key: FLINK-23640
 URL: https://issues.apache.org/jira/browse/FLINK-23640
 Project: Flink
  Issue Type: Sub-task
Reporter: Fabian Paul


Commonly users only want to serialize the value of a Kafka record if they are 
not using a key schema. For these users, we can provide an easier entry point 
comparable to \{{ KafkaValueOnlyDeserializerWrapper }}.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-23664) Write documentation for new KafkaSink

[Fabian Paul (Jira)]

Fabian Paul created FLINK-23664:
---

 Summary: Write documentation for new KafkaSink
 Key: FLINK-23664
 URL: https://issues.apache.org/jira/browse/FLINK-23664
 Project: Flink
  Issue Type: Sub-task
Reporter: Fabian Paul






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-23710) Move sink to org.apache.kafka.conntor.kafka.sink package

[Fabian Paul (Jira)]

Fabian Paul created FLINK-23710:
---

 Summary: Move sink to org.apache.kafka.conntor.kafka.sink package
 Key: FLINK-23710
 URL: https://issues.apache.org/jira/browse/FLINK-23710
 Project: Flink
  Issue Type: Sub-task
Reporter: Fabian Paul






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-23735) Migrate BufferedUpsertSinkFunction to FLIP-143

[Fabian Paul (Jira)]

Fabian Paul created FLINK-23735:
---

 Summary: Migrate BufferedUpsertSinkFunction to FLIP-143
 Key: FLINK-23735
 URL: https://issues.apache.org/jira/browse/FLINK-23735
 Project: Flink
  Issue Type: Sub-task
Reporter: Fabian Paul


The BufferedUpsertSinkFunction is still using the old sink interfaces and 
relies on the old Kafka DataStream connector FlinkKafkaProducer.

We need to migrate it to the new Sink API to also leverage the new KafkaSink 
connector and finally deprecate the FlinkKafkaProducer and all its belongings.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-23814) Test FLIP-143 KafkaSink

[Fabian Paul (Jira)]

Fabian Paul created FLINK-23814:
---

 Summary: Test FLIP-143 KafkaSink
 Key: FLINK-23814
 URL: https://issues.apache.org/jira/browse/FLINK-23814
 Project: Flink
  Issue Type: Sub-task
  Components: Connectors / Kafka
Reporter: Fabian Paul
 Fix For: 1.14.0


The following scenarios are worthwhile to test
 * Start simple job with None/At-least once delivery guarantee and write 
records to kafka topic
 * Start simple job with exactly-once delivery guarantee and write records to 
kafka topic. The records should only be visible with a `read-committed` consumer
 * Stop a job with exactly-once delivery guarantee and restart it with 
different parallelism (scale-down, scale-up)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-23838) Add FLIP-33 metrics to new KafkaSink

[Fabian Paul (Jira)]

Fabian Paul created FLINK-23838:
---

 Summary: Add FLIP-33 metrics to new KafkaSink
 Key: FLINK-23838
 URL: https://issues.apache.org/jira/browse/FLINK-23838
 Project: Flink
  Issue Type: Sub-task
  Components: Connectors / Kafka
Reporter: Fabian Paul
 Fix For: 1.14.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-23875) ReducingUpsertSink can loose record during failover

[Fabian Paul (Jira)]

Fabian Paul created FLINK-23875:
---

 Summary:  ReducingUpsertSink can loose record during failover
 Key: FLINK-23875
 URL: https://issues.apache.org/jira/browse/FLINK-23875
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kafka, Table SQL / API
Affects Versions: 1.14.0
Reporter: Fabian Paul


When trying to rework the Table API Kafka connector to make it compatible with 
the new KafkaSink I noticed that currently the buffer which is used to reduce 
the update-before and update-after calls is not snapshotted which can result in 
data loss if the job fails while the buffer is not empty.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-24006) MailboxExecutorImplTest#testIsIdle does not test the correct behaviour

[Fabian Paul (Jira)]

Fabian Paul created FLINK-24006:
---

 Summary: MailboxExecutorImplTest#testIsIdle does not test the 
correct behaviour
 Key: FLINK-24006
 URL: https://issues.apache.org/jira/browse/FLINK-24006
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Task, Tests
Affects Versions: 1.13.2, 1.12.5, 1.14.0
Reporter: Fabian Paul


The test was introduced to ensure that the mailbox idleness is still counting 
new messages although the mailbox loop might have been stopped.

 

Unfortunately, the test does not stop the mailbox processor currently which 
leads to than the test even passes without the actual code changes of 
https://issues.apache.org/jira/browse/FLINK-19109



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-24032) StreamSink does not receive periodic watermarks

[Fabian Paul (Jira)]

Fabian Paul created FLINK-24032:
---

 Summary: StreamSink does not receive periodic watermarks
 Key: FLINK-24032
 URL: https://issues.apache.org/jira/browse/FLINK-24032
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Task
Affects Versions: 1.14.0
Reporter: Fabian Paul


In the following scenario, the sink will never receive watermarks 
  
{code:java}
env.readFile(...)
.assignTimestampsAndWatermarks(format, file)
.rebalance()
.addSink(...);
{code}
I also noticed that when changing the code to the following the watermarks flow 
to the sink
{code:java}
env.readFile(...)
.assignTimestampsAndWatermarks(format, file)
.rebalance()
.process(new ProcessFunction() {...})
.addSink(...);
{code}
An example test case is accessible here 
[https://github.com/fapaul/flink/blob/9b749ac80cd128a7f288da45db313bafa39d8008/flink-tests/src/test/java/org/apache/flink/test/streaming/api/FileReadingWatermarkITCase.java#L68]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-24051) Make consumer.group-id optional for KafkaSource

[Fabian Paul (Jira)]

Fabian Paul created FLINK-24051:
---

 Summary: Make consumer.group-id optional for KafkaSource
 Key: FLINK-24051
 URL: https://issues.apache.org/jira/browse/FLINK-24051
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Kafka
Affects Versions: 1.14.0, 1.15.0
Reporter: Fabian Paul


For most of the users it is not necessary to generate a group-id and the source 
itself can provide a meaningful group-id during startup.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-24055) Deprecate FlinkKafkaConsumer

[Fabian Paul (Jira)]

Fabian Paul created FLINK-24055:
---

 Summary: Deprecate FlinkKafkaConsumer
 Key: FLINK-24055
 URL: https://issues.apache.org/jira/browse/FLINK-24055
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Kafka
Affects Versions: 1.14.0, 1.15.0
Reporter: Fabian Paul


With the introduction of the KafkaSource 
https://issues.apache.org/jira/browse/FLINK-18323 we should deprecate the 
FlinkKafkaConsumer to hint users to start the migration.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-24079) BufferedUpsertSinkFunction can loose records during failover

[Fabian Paul (Jira)]

Fabian Paul created FLINK-24079:
---

 Summary: BufferedUpsertSinkFunction can loose records during 
failover
 Key: FLINK-24079
 URL: https://issues.apache.org/jira/browse/FLINK-24079
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kafka, Table SQL / Ecosystem
Affects Versions: 1.13.2
Reporter: Fabian Paul


The internally used buffer is not snapshotted on checkpoint which can lead to 
loosing on failure. We need to snapshot the buffer similarly to FLINK-23875.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-24281) Migrate all existing tests to new Kafka Sink

[Fabian Paul (Jira)]

Fabian Paul created FLINK-24281:
---

 Summary: Migrate all existing tests to new Kafka Sink
 Key: FLINK-24281
 URL: https://issues.apache.org/jira/browse/FLINK-24281
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Kafka
Affects Versions: 1.14.0, 1.15.0
Reporter: Fabian Paul


The FlinkKafkaProducer is deprecated since 1.14 but a lot of existing tests are 
still using.

We should replace it with the KafkaSink because it completely subsumes it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-24282) KafkaRecordSerializationSchema TopicSelector is not serializable

[Fabian Paul (Jira)]

Fabian Paul created FLINK-24282:
---

 Summary: KafkaRecordSerializationSchema TopicSelector is not 
serializable
 Key: FLINK-24282
 URL: https://issues.apache.org/jira/browse/FLINK-24282
 Project: Flink
  Issue Type: Bug
Affects Versions: 1.14.0
Reporter: Fabian Paul


To dynamically calculate the outgoing topic we allow passing a lambda. 
Unfortunately, it is currently not marked as serializable hence the following 
code fails in during closure cleaning when used within a job.
 
{code:java}
KafkaRecordSerializationSchema.builder()
.setTopic(topic)
.setValueSerializationSchema(serSchema)
.setPartitioner(partitioner)
.build())
{code}

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-24292) Update Flink's Kafka examples to use KafkaSink

[Fabian Paul (Jira)]

Fabian Paul created FLINK-24292:
---

 Summary: Update Flink's Kafka examples to use KafkaSink
 Key: FLINK-24292
 URL: https://issues.apache.org/jira/browse/FLINK-24292
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Kafka
Affects Versions: 1.14.0, 1.15.0
Reporter: Fabian Paul






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-24308) Translate KafkaSink docs to chinese

[Fabian Paul (Jira)]

Fabian Paul created FLINK-24308:
---

 Summary: Translate KafkaSink docs to chinese
 Key: FLINK-24308
 URL: https://issues.apache.org/jira/browse/FLINK-24308
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Affects Versions: 1.14.0
Reporter: Fabian Paul


With https://issues.apache.org/jira/browse/FLINK-23664 only the English 
documentation was updated. We also have to update the Chinese docs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)