Re: [DISCUSS] Connector releases for Flink 1.19

[Jing Ge]

Hi,

Thanks Danny for driving it! Do we have an umbrella ticket for Flink 1.19
connectors release?

@Sergei
Thanks for the hint wrt JDBC connector. Where could users know that it
already supports 1.19?

Best regards,
Jing

On Fri, May 17, 2024 at 4:07 AM Sergey Nuyanzin  wrote:

> >, it looks like opensearch-2.0.0 has been created now, all good.
> yep, thanks to Martijn
>
> I've created RCs for Opensearch connector
>
> On Tue, May 14, 2024 at 12:38 PM Danny Cranmer 
> wrote:
>
> > Hello,
> >
> > @Sergey Nuyanzin , it looks like opensearch-2.0.0
> > has been created now, all good.
> >
> > @Hongshun Wang, thanks, since the CDC connectors are not yet released I
> > had omitted them from this task. But happy to include them, thanks for
> the
> > support.
> >
> > Thanks,
> > Danny
> >
> > On Mon, May 13, 2024 at 3:40 AM Hongshun Wang 
> > wrote:
> >
> >> Hello Danny,
> >> Thanks for pushing this forward.  I am available to assist with the CDC
> >> connector[1].
> >>
> >> [1] https://github.com/apache/flink-cdc
> >>
> >> Best
> >> Hongshun
> >>
> >> On Sun, May 12, 2024 at 8:48 PM Sergey Nuyanzin 
> >> wrote:
> >>
> >> > I'm in a process of preparation of RC for OpenSearch connector
> >> >
> >> > however it seems I need PMC help: need to create opensearch-2.0.0 on
> >> jira
> >> > since as it was proposed in another ML[1] to have 1.x for OpenSearch
> >> > v1 and 2.x for OpenSearch v2
> >> >
> >> > would be great if someone from PMC could help here
> >> >
> >> > [1] https://lists.apache.org/thread/3w1rnjp5y612xy5k9yv44hy37zm9ph15
> >> >
> >> > On Wed, Apr 17, 2024 at 12:42 PM Ferenc Csaky
> >> >  wrote:
> >> > >
> >> > > Thank you Danny and Sergey for pushing this!
> >> > >
> >> > > I can help with the HBase connector if necessary, will comment the
> >> > > details to the relevant Jira ticket.
> >> > >
> >> > > Best,
> >> > > Ferenc
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > On Wednesday, April 17th, 2024 at 11:17, Danny Cranmer <
> >> > dannycran...@apache.org> wrote:
> >> > >
> >> > > >
> >> > > >
> >> > > > Hello all,
> >> > > >
> >> > > > I have created a parent Jira to cover the releases [1]. I have
> >> > assigned AWS
> >> > > > and MongoDB to myself and OpenSearch to Sergey. Please assign the
> >> > > > relevant issue to yourself as you pick up the tasks.
> >> > > >
> >> > > > Thanks!
> >> > > >
> >> > > > [1] https://issues.apache.org/jira/browse/FLINK-35131
> >> > > >
> >> > > > On Tue, Apr 16, 2024 at 2:41 PM Muhammet Orazov
> >> > > > mor+fl...@morazow.com.invalid wrote:
> >> > > >
> >> > > > > Thanks Sergey and Danny for clarifying, indeed it
> >> > > > > requires committer to go through the process.
> >> > > > >
> >> > > > > Anyway, please let me know if I can be any help.
> >> > > > >
> >> > > > > Best,
> >> > > > > Muhammet
> >> > > > >
> >> > > > > On 2024-04-16 11:19, Danny Cranmer wrote:
> >> > > > >
> >> > > > > > Hello,
> >> > > > > >
> >> > > > > > I have opened the VOTE thread for the AWS connectors release
> >> [1].
> >> > > > > >
> >> > > > > > > If I'm not mistaking (please correct me if I'm wrong) this
> >> > request is
> >> > > > > > > not
> >> > > > > > > about version update it is about new releases for connectors
> >> > > > > >
> >> > > > > > Yes, correct. If there are any other code changes required
> then
> >> > help
> >> > > > > > would be appreciated.
> >> > > > > >
> >> > > > > > > Are you going to create an umbrella issue for it?
> >> > > > > >
> >> > > > > > We do not usually create JIRA issues for releases. That being
> >> said
> >> > it
> >> > > > > > sounds like a good idea to have one place to track the status
> of
> >> > the
> >> > > > > > connector releases and pre-requisite code changes.
> >> > > > > >
> >> > > > > > > I would like to work on this task, thanks for initiating it!
> >> > > > > >
> >> > > > > > The actual release needs to be performed by a committer.
> >> However,
> >> > help
> >> > > > > > getting the connectors building against Flink 1.19 and testing
> >> the
> >> > RC
> >> > > > > > is
> >> > > > > > appreciated.
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > > Danny
> >> > > > > >
> >> > > > > > [1]
> >> > https://lists.apache.org/thread/0nw9smt23crx4gwkf6p1dd4jwvp1g5s0
> >> > > > > >
> >> > > > > > On Tue, Apr 16, 2024 at 6:34 AM Sergey Nuyanzin
> >> > snuyan...@gmail.com
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > Thanks for volunteering Muhammet!
> >> > > > > > > And thanks Danny for starting the activity.
> >> > > > > > >
> >> > > > > > > If I'm not mistaking (please correct me if I'm wrong)
> >> > > > > > >
> >> > > > > > > this request is not about version update it is about new
> >> > releases for
> >> > > > > > > connectors
> >> > > > > > > btw for jdbc connector support of 1.19 and 1.20-SNAPSHOT is
> >> > already
> >> > > > > > > done
> >> > > > > > >
> >> > > > > > > I would volunteer for Opensearch connector since currently
> I'm
> >> > working
> >> > > > > > > on
> >> > > > > > > support of Opensearch v2
> >> > > > > > > 

Re: [VOTE] FLIP-453: Promote Unified Sink API V2 to Public and Deprecate SinkFunction

[Jing Ge]

+1(binding) Thanks Martijn!

Best regards,
Jing

On Wed, May 15, 2024 at 7:00 PM Muhammet Orazov
 wrote:

> Thanks Martijn driving this! +1 (non-binding)
>
> Best,
> Muhammet
>
> On 2024-05-14 06:43, Martijn Visser wrote:
> > Hi everyone,
> >
> > With no more discussions being open in the thread [1] I would like to
> > start
> > a vote on FLIP-453: Promote Unified Sink API V2 to Public and Deprecate
> > SinkFunction [2]
> >
> > The vote will be open for at least 72 hours unless there is an
> > objection or
> > insufficient votes.
> >
> > Best regards,
> >
> > Martijn
> >
> > [1] https://lists.apache.org/thread/hod6bg421bzwhbfv60lwsck7r81dvo59
> > [2]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-453%3A+Promote+Unified+Sink+API+V2+to+Public+and+Deprecate+SinkFunction
>

Re: [ANNOUNCE] New Apache Flink Committer - Zakelly Lan

[Jing Ge]

Congratulations Zakelly!

Best regards,
Jing

On Mon, Apr 15, 2024 at 4:26 PM Xia Sun  wrote:

> Congratulations Zakelly!
>
>  Best,
>  Xia
>
> Leonard Xu  于2024年4月15日周一 16:16写道：
>
> > Congratulations Zakelly!
> >
> >
> > Best,
> > Leonard
> > > 2024年4月15日 下午3:56，Samrat Deb  写道：
> > >
> > > Congratulations Zakelly!
> >
> >
>

Re: [VOTE] FLIP-399: Flink Connector Doris

[Jing Ge]

+1(binding)

Best regards,
Jing

On Tue, Apr 9, 2024 at 8:54 PM Feng Jin  wrote:

> +1 (non-binding)
>
> Best,
> Feng
>
> On Tue, Apr 9, 2024 at 5:56 PM gongzhongqiang 
> wrote:
>
> > +1 (non-binding)
> >
> > Best,
> >
> > Zhongqiang Gong
> >
> > wudi <676366...@qq.com.invalid> 于2024年4月9日周二 10:48写道：
> >
> > > Hi devs,
> > >
> > > I would like to start a vote about FLIP-399 [1]. The FLIP is about
> > > contributing the Flink Doris Connector[2] to the Flink community.
> > > Discussion thread [3].
> > >
> > > The vote will be open for at least 72 hours unless there is an
> objection
> > or
> > > insufficient votes.
> > >
> > >
> > > Thanks,
> > > Di.Wu
> > >
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-399%3A+Flink+Connector+Doris
> > > [2] https://github.com/apache/doris-flink-connector
> > > [3] https://lists.apache.org/thread/p3z4wsw3ftdyfs9p2wd7bbr2gfyl3xnh
> > >
> > >
> >
>

Re: Re: [ANNOUNCE] Apache Paimon is graduated to Top Level Project

[Jing Ge]

Congrats!

Best regards,
Jing

On Thu, Mar 28, 2024 at 1:27 PM Feifan Wang  wrote:

> Congratulations!——
>
> Best regards,
>
> Feifan Wang
>
>
>
>
> At 2024-03-28 20:02:43, "Yanfei Lei"  wrote:
> >Congratulations!
> >
> >Best,
> >Yanfei
> >
> >Zhanghao Chen  于2024年3月28日周四 19:59写道：
> >>
> >> Congratulations!
> >>
> >> Best,
> >> Zhanghao Chen
> >> 
> >> From: Yu Li 
> >> Sent: Thursday, March 28, 2024 15:55
> >> To: d...@paimon.apache.org 
> >> Cc: dev ; user 
> >> Subject: Re: [ANNOUNCE] Apache Paimon is graduated to Top Level Project
> >>
> >> CC the Flink user and dev mailing list.
> >>
> >> Paimon originated within the Flink community, initially known as Flink
> >> Table Store, and all our incubating mentors are members of the Flink
> >> Project Management Committee. I am confident that the bonds of
> >> enduring friendship and close collaboration will continue to unite the
> >> two communities.
> >>
> >> And congratulations all!
> >>
> >> Best Regards,
> >> Yu
> >>
> >> On Wed, 27 Mar 2024 at 20:35, Guojun Li 
> wrote:
> >> >
> >> > Congratulations!
> >> >
> >> > Best,
> >> > Guojun
> >> >
> >> > On Wed, Mar 27, 2024 at 5:24 PM wulin  wrote:
> >> >
> >> > > Congratulations~
> >> > >
> >> > > > 2024年3月27日 15:54，王刚  写道：
> >> > > >
> >> > > > Congratulations~
> >> > > >
> >> > > >> 2024年3月26日 10:25，Jingsong Li  写道：
> >> > > >>
> >> > > >> Hi Paimon community,
> >> > > >>
> >> > > >> I’m glad to announce that the ASF board has approved a
> resolution to
> >> > > >> graduate Paimon into a full Top Level Project. Thanks to
> everyone for
> >> > > >> your help to get to this point.
> >> > > >>
> >> > > >> I just created an issue to track the things we need to modify
> [2],
> >> > > >> please comment on it if you feel that something is missing. You
> can
> >> > > >> refer to apache documentation [1] too.
> >> > > >>
> >> > > >> And, we already completed the GitHub repo migration [3], please
> update
> >> > > >> your local git repo to track the new repo [4].
> >> > > >>
> >> > > >> You can run the following command to complete the remote repo
> tracking
> >> > > >> migration.
> >> > > >>
> >> > > >> git remote set-url origin https://github.com/apache/paimon.git
> >> > > >>
> >> > > >> If you have a different name, please change the 'origin' to your
> remote
> >> > > name.
> >> > > >>
> >> > > >> Please join me in celebrating!
> >> > > >>
> >> > > >> [1]
> >> > >
> https://incubator.apache.org/guides/transferring.html#life_after_graduation
> >> > > >> [2] https://github.com/apache/paimon/issues/3091
> >> > > >> [3] https://issues.apache.org/jira/browse/INFRA-25630
> >> > > >> [4] https://github.com/apache/paimon
> >> > > >>
> >> > > >> Best,
> >> > > >> Jingsong Lee
> >> > >
> >> > >
>

Re: Re: [VOTE] FLIP-423: Disaggregated State Storage and Management (Umbrella FLIP)

[Jing Ge]

+1(binding)

Best Regards,
Jing

On Thu, Mar 28, 2024 at 1:28 PM Feifan Wang  wrote:

> +1 (non-binding)
>
> ——
>
> Best regards,
>
> Feifan Wang
>
>
>
>
> At 2024-03-28 20:20:21, "Piotr Nowojski"  wrote:
> >+1 (binding)
> >
> >Piotrek
> >
> >czw., 28 mar 2024 o 11:43 Yuan Mei  napisał(a):
> >
> >> Hi devs,
> >>
> >> I'd like to start a vote on the FLIP-423: Disaggregated State Storage
> and
> >> Management (Umbrella FLIP) [1]. The discussion thread is here [2].
> >>
> >> The vote will be open for at least 72 hours unless there is an
> objection or
> >> insufficient votes.
> >>
> >> [1] https://cwiki.apache.org/confluence/x/R4p3EQ
> >> [2] https://lists.apache.org/thread/ct8smn6g9y0b8730z7rp9zfpnwmj8vf0
> >>
> >>
> >> Best,
> >> Yuan
> >>
>

Re: Re: [VOTE] FLIP-425: Asynchronous Execution Model

[Jing Ge]

+1(binding)

Best Regards,
Jing

On Thu, Mar 28, 2024 at 1:30 PM Feifan Wang  wrote:

> +1 (non-binding)
>
>
>
> ——
>
> Best regards,
>
> Feifan Wang
>
>
>
>
> At 2024-03-28 20:20:39, "Piotr Nowojski"  wrote:
> >+1 (binding)
> >
> >Piotrek
> >
> >czw., 28 mar 2024 o 11:44 Yuan Mei  napisał(a):
> >
> >> +1 (binding)
> >>
> >> Best,
> >> Yuan
> >>
> >> On Thu, Mar 28, 2024 at 4:33 PM Xuannan Su 
> wrote:
> >>
> >> > +1 (non-binding)
> >> >
> >> > Best regards,
> >> > Xuannan
> >> >
> >> > On Wed, Mar 27, 2024 at 6:28 PM Yanfei Lei 
> wrote:
> >> > >
> >> > > Hi everyone,
> >> > >
> >> > > Thanks for all the feedback about the FLIP-425: Asynchronous
> Execution
> >> > > Model [1]. The discussion thread is here [2].
> >> > >
> >> > > The vote will be open for at least 72 hours unless there is an
> >> > > objection or insufficient votes.
> >> > >
> >> > > [1] https://cwiki.apache.org/confluence/x/S4p3EQ
> >> > > [2]
> https://lists.apache.org/thread/wxn1j848fnfkqsnrs947wh1wmj8n8z0h
> >> > >
> >> > > Best regards,
> >> > > Yanfei
> >> >
> >>
>

Re: [VOTE] FLIP-424: Asynchronous State APIs

[Jing Ge]

+1 (binding)

Thanks!

Best regards,
Jing

On Wed, Mar 27, 2024 at 11:23 AM Zakelly Lan  wrote:

> Hi devs,
>
> I'd like to start a vote on the FLIP-424: Asynchronous State APIs [1]. The
> discussion thread is here [2].
>
> The vote will be open for at least 72 hours unless there is an objection or
> insufficient votes.
>
> [1] https://cwiki.apache.org/confluence/x/SYp3EQ
> [2] https://lists.apache.org/thread/nmd9qd0k8l94ygcfgllxms49wmtz1864
>
>
> Best,
> Zakelly
>

Re: [VOTE] Release 1.19.0, release candidate #2

[Jing Ge]

+1 (not-binding)

- verified signature
- verified checksum
- verified that source distribution does not contain binaries
- built from source
- reviewed the web PR
- reviewed the release note
- checked the tag
- checked the repo

Best regards,
Jing

On Thu, Mar 14, 2024 at 2:37 PM Sergey Nuyanzin  wrote:

> +1 (non-binding)
>
> - Checked the pre-built jars are generated with jdk8
> - Verified signature and checksum
> - Verified no binary in source
> - Verified source code tag
> - Reviewed release note
> - Reviewed web PR
> - Built from source
> - Run a simple job successfully
>
> On Thu, Mar 14, 2024 at 2:21 PM Martijn Visser 
> wrote:
>
> > +1 (binding)
> >
> > - Validated hashes
> > - Verified signature
> > - Verified that no binaries exist in the source archive
> > - Build the source with Maven via mvn clean install -Pcheck-convergence
> > -Dflink.version=1.19.0
> > - Verified licenses
> > - Verified web PR
> > - Started a cluster and the Flink SQL client, successfully read and wrote
> > with the Kafka connector to Confluent Cloud with AVRO and Schema Registry
> > enabled
> >
> > On Thu, Mar 14, 2024 at 1:32 PM gongzhongqiang <
> gongzhongqi...@apache.org>
> > wrote:
> >
> > > +1 (non-binding)
> > >
> > > - Verified no binary files in source code
> > > - Verified signature and checksum
> > > - Build source code and run a simple job successfully
> > > - Reviewed the release announcement PR
> > >
> > > Best,
> > >
> > > Zhongqiang Gong
> > >
> > > Ferenc Csaky  于2024年3月14日周四 20:07写道：
> > >
> > > >  +1 (non-binding)
> > > >
> > > > - Verified checksum and signature
> > > > - Verified no binary in src
> > > > - Built from src
> > > > - Reviewed release note PR
> > > > - Reviewed web PR
> > > > - Tested a simple datagen query and insert to blackhole sink via SQL
> > > > Gateway
> > > >
> > > > Best,
> > > > Ferenc
> > > >
> > > >
> > > >
> > > >
> > > > On Thursday, March 14th, 2024 at 12:14, Jane Chan <
> > qingyue@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > >
> > > > >
> > > > > Hi Lincoln,
> > > > >
> > > > > Thank you for the prompt response and the effort to provide clarity
> > on
> > > > this
> > > > > matter.
> > > > >
> > > > > Best,
> > > > > Jane
> > > > >
> > > > > On Thu, Mar 14, 2024 at 6:02 PM Lincoln Lee lincoln.8...@gmail.com
> > > > wrote:
> > > > >
> > > > > > Hi Jane,
> > > > > >
> > > > > > Thank you for raising this question. I saw the discussion in the
> > Jira
> > > > > > (include Matthias' point)
> > > > > > and sought advice from several PMCs (including the previous RMs),
> > the
> > > > > > majority of people
> > > > > > are in favor of merging the bugfix into the release branch even
> > > during
> > > > the
> > > > > > release candidate
> > > > > > (RC) voting period, so we should accept all bugfixes (unless
> there
> > > is a
> > > > > > specific community
> > > > > > rule preventing it).
> > > > > >
> > > > > > Thanks again for contributing to the community!
> > > > > >
> > > > > > Best,
> > > > > > Lincoln Lee
> > > > > >
> > > > > > Matthias Pohl matthias.p...@aiven.io.invalid 于2024年3月14日周四
> > 17:50写道：
> > > > > >
> > > > > > > Update on FLINK-34227 [1] which I mentioned above: Chesnay
> helped
> > > > > > > identify
> > > > > > > a concurrency issue in the JobMaster shutdown logic which seems
> > to
> > > > be in
> > > > > > > the code for quite some time. I created a PR fixing the issue
> > > hoping
> > > > that
> > > > > > > the test instability is resolved with it.
> > > > > > >
> > > > > > > The concurrency issue doesn't really explain why it only
> started
> > to
> > > > > > > appear
> > > > > > > recently in a specific CI setup (GHA with AdaptiveScheduler).
> > There
> > > > is no
> > > > > > > hint in the git history indicating that it's caused by some
> newly
> > > > > > > introduced change. That is why I wouldn't make FLINK-34227 a
> > reason
> > > > to
> > > > > > > cancel rc2. Instead, the fix can be provided in subsequent
> patch
> > > > > > > releases.
> > > > > > >
> > > > > > > Matthias
> > > > > > >
> > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-34227
> > > > > > >
> > > > > > > On Thu, Mar 14, 2024 at 8:49 AM Jane Chan
> qingyue@gmail.com
> > > > wrote:
> > > > > > >
> > > > > > > > Hi Yun, Jing, Martijn and Lincoln,
> > > > > > > >
> > > > > > > > I'm seeking guidance on whether merging the bugfix[1][2] at
> > this
> > > > stage
> > > > > > > > is
> > > > > > > > appropriate. I want to ensure that the actions align with the
> > > > current
> > > > > > > > release process and do not disrupt the ongoing preparations.
> > > > > > > >
> > > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-29114
> > > > > > > > [2] https://github.com/apache/flink/pull/24492
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Jane
> > > > > > > >
> > > > > > > > On Thu, Mar 14, 2024 at 1:33 PM Yun Tang myas...@live.com
> > wrote:
> > > > > > > >
> > > > > > > > > +1 (non-binding)
> > > > > > > > >
> > > > > > > > > *
> > > > > > > > > 

Re: [DISCUSS] FLIP-424: Asynchronous State APIs

[Jing Ge]

indeed! I missed that part. Thanks for the hint!

Best regards,
Jing

On Wed, Mar 13, 2024 at 6:02 AM Zakelly Lan  wrote:

> Hi Jing,
>
> The deprecation and removal of original APIs is beyond the scope of
> current FLIP, but I do add/highlight such information under "Compatibility,
> Deprecation, and Migration Plan" section.
>
>
> Best,
> Zakelly
>
> On Wed, Mar 13, 2024 at 9:18 AM Yunfeng Zhou 
> wrote:
>
>> Hi Zakelly,
>>
>> Thanks for your responses. I agree with it that we can keep the design
>> as it is for now and see if others have any better ideas for these
>> questions.
>>
>> Best,
>> Yunfeng
>>
>> On Tue, Mar 12, 2024 at 5:23 PM Zakelly Lan 
>> wrote:
>> >
>> > Hi Xuannan,
>> >
>> > Thanks for your comments, I modified the FLIP accordingly.
>> >
>> > Hi Yunfeng,
>> >
>> > Thanks for sharing your opinions!
>> >
>> >> Could you provide some hint on use cases where users need to mix sync
>> >> and async state operations in spite of the performance regression?
>> >> This information might help address our concerns on design. If the
>> >> mixed usage is simply something not recommended, I would prefer to
>> >> prohibit such usage from API.
>> >
>> > In fact, there is no scenario where users MUST use the sync APIs, but
>> it is much easier to use for those who are not familiar with asynchronous
>> programming. If they want to migrate their job from Flink 1.x to 2.0
>> leveraging some benefits from asynchronous APIs, they may try the mixed
>> usage. It is not user-friendly to directly throw exceptions at runtime, I
>> think our better approach is to warn users and recommend avoiding this. I
>> added an example in this FLIP.
>> >
>> > Well, I do not insist on allowing mixed usage of APIs if others reach
>> an agreement that we won't support that . I think the most important is to
>> keep the API easy to use and understand, thus I propose a unified state
>> declaration and explicit meaning in method name. WDYT?
>> >
>> >> Sorry I missed the new sink API. I do still think that it would be
>> >> better to make the package name more informative, and ".v2." does not
>> >> contain information for new Flink users who did not know the v1 of
>> >> state API. Unlike internal implementation and performance
>> >> optimization, API will hardly be compromised for now and updated in
>> >> future, so I still suggest we improve the package name now if
>> >> possible. But given the existing practice of sink v2 and
>> >> AbstractStreamOperatorV2, the current package name would be acceptable
>> >> to me if other reviewers of this FLIP agrees on it.
>> >
>> > Actually, I don't like 'v2' either. So if there is another good name,
>> I'd be happy to apply. This is a compromise to the current situation. Maybe
>> we could refine this after the retirement of original state APIs.
>> >
>> >
>> > Thanks & Best,
>> > Zakelly
>> >
>> >
>> > On Tue, Mar 12, 2024 at 1:42 PM Yunfeng Zhou <
>> flink.zhouyunf...@gmail.com> wrote:
>> >>
>> >> Hi Zakelly,
>> >>
>> >> Thanks for the quick response!
>> >>
>> >> > Actually splitting APIs into two sets ... warn them in runtime.
>> >>
>> >> Could you provide some hint on use cases where users need to mix sync
>> >> and async state operations in spite of the performance regression?
>> >> This information might help address our concerns on design. If the
>> >> mixed usage is simply something not recommended, I would prefer to
>> >> prohibit such usage from API.
>> >>
>> >> > In fact ... .sink2`.
>> >>
>> >> Sorry I missed the new sink API. I do still think that it would be
>> >> better to make the package name more informative, and ".v2." does not
>> >> contain information for new Flink users who did not know the v1 of
>> >> state API. Unlike internal implementation and performance
>> >> optimization, API will hardly be compromised for now and updated in
>> >> future, so I still suggest we improve the package name now if
>> >> possible. But given the existing practice of sink v2 and
>> >> AbstractStreamOperatorV2, the current package name would be acceptable
>> >> to me if other reviewers of this FLIP agrees on it.
>> >>
>> >> Best,
>> >> Yunfeng
>> >>
>> >> On Mon, Mar 11, 2024 at 5:27 PM Zakelly Lan 
>> wrote:
>> >> >
>> >> > Hi Yunfeng,
>> >> >
>> >> > Thanks for your comments!
>> >> >
>> >> > +1 for JingGe's suggestion to introduce an AsyncState API, instead of
>> >> > > having both get() and asyncGet() in the same State class. As a
>> >> > > supplement to its benefits, this design could help avoid having
>> users
>> >> > > to use sync and async API in a mixed way (unless they create both a
>> >> > > State and an AsyncState from the same state descriptor), which is
>> >> > > supposed to bring suboptimal performance according to the FLIP's
>> >> > > description.
>> >> >
>> >> >
>> >> > Actually splitting APIs into two sets of classes also brings some
>> >> > difficulties. In this case, users must explicitly define their usage
>> before
>> >> > actually doing state access. It is a little strange 

Re: [VOTE] FLIP-420: Add API annotations for RocksDB StateBackend user-facing classes

[Jing Ge]

+1 (binding) Thanks!

Best regards,
Jing

On Sun, Mar 10, 2024 at 5:32 PM Jing Ge  wrote:

> Hi Jinzhong,
>
> Thanks for driving this topic and sorry for just joining the discussion
> now. I replied in your discussion thread. Would you like to take a look
> and let's keep the discussion there? I will come back to this thread and
> vote once the discussion is done. Thanks!
>
> Best regards,
> Jing
>
> On Thu, Mar 7, 2024 at 4:39 AM Zakelly Lan  wrote:
>
>> +1 non-binding
>>
>> Thanks for proposing this.
>>
>> Best,
>> Zakelly
>>
>> On Thu, Mar 7, 2024 at 10:13 AM Yanfei Lei  wrote:
>>
>> > +1(binding) for this vote.
>> >
>> > Hangxiang Yu  于2024年3月7日周四 09:54写道：
>> > >
>> > > +1 (binding)
>> > >
>> > > On Thu, Mar 7, 2024 at 9:34 AM Yun Tang  wrote:
>> > >
>> > > > > +1 for this FLIP.
>> > > > Sorry for not being clear in my previous reply, it's a binding vote.
>> > > >
>> > > > Best
>> > > > Yun Tang
>> > > > 
>> > > > From: Jeyhun Karimov 
>> > > > Sent: Thursday, March 7, 2024 4:40
>> > > > To: dev@flink.apache.org 
>> > > > Subject: Re: [VOTE] FLIP-420: Add API annotations for RocksDB
>> > StateBackend
>> > > > user-facing classes
>> > > >
>> > > > Hi Jinzhong,
>> > > >
>> > > > Thanks for the FLIP.
>> > > >
>> > > > +1 (non-binding)
>> > > >
>> > > > Regards,
>> > > > Jeyhun
>> > > >
>> > > > On Wed, Mar 6, 2024 at 5:09 PM Yun Tang  wrote:
>> > > >
>> > > > > +1 for this FLIP.
>> > > > >
>> > > > >
>> > > > >
>> > > > > Best
>> > > > > Yun Tang
>> > > > > 
>> > > > > From: Jinzhong Li 
>> > > > > Sent: Wednesday, March 6, 2024 20:29
>> > > > > To: dev@flink.apache.org 
>> > > > > Subject: [VOTE] FLIP-420: Add API annotations for RocksDB
>> > StateBackend
>> > > > > user-facing classes
>> > > > >
>> > > > > Hi All,
>> > > > >
>> > > > > I'd like to start a vote on the FLIP-420: Add API annotations for
>> > RocksDB
>> > > > > StateBackend user-facing classes[1].
>> > > > > The discussion thread is here [2].
>> > > > >
>> > > > > The vote will be open for at least 72 hours unless there is an
>> > objection
>> > > > or
>> > > > > not enough votes.
>> > > > >
>> > > > >
>> > > > > [1]https://cwiki.apache.org/confluence/x/JQs4EQ
>> > > > > [2]
>> https://lists.apache.org/thread/4t71lz2j2ft8hf90ylvtomynhr2qthoo
>> > > > >
>> > > > >
>> > > > > Best,
>> > > > > Jinzhong Li
>> > > > >
>> > > >
>> > >
>> > >
>> > > --
>> > > Best,
>> > > Hangxiang.
>> >
>> >
>> >
>> > --
>> > Best,
>> > Yanfei
>> >
>>
>

Re: [DISCUSS]FLIP-420: Add API annotations for RocksDB StateBackend user-facing classes

[Jing Ge]

Hi Jinzhong,

Thanks for your clarification!


> As mentioned in Flink's “API compatibility guarantees”[1], "any API without
> such an annotation is considered internal to Flink, with no guarantees
> being provided."
> I think these interfaces are treated as internal interfaces regardless of
> whether they are marked @Internal.
>
> So, it's acceptable for me to either leave it unmarked or mark it as
> @Internal.
>

It is a question of consistency. I am fine with @Internal and once we
use @Internal, we should add it to all classes within a module to keep the
consistency. Otherwise, when developers read some interfaces
have @Internal, some other classes have no annotations. It turns out there
are different meanings of with or without @Internal. But it is not a big
deal. If no other one has the same concern. I will not block you :-)



> As for FLIP-197, I found that the proposal in this flip has not actually
> been implemented. Because FLINK-25352 [2] has not been resolved，it looks
> like we can't add more info into the PublicEvolving annotation.
>
>
Fair enough! Thanks for the hint!

Best regards,
Jing



On Mon, Mar 11, 2024 at 9:03 AM Jinzhong Li 
wrote:

> Hi Jing.
>
> Thanks for your suggestion.
>
> >>  I was wondering if SingleStateIterator and RocksDBRestoreOperation are
> exposed to users even if they are interfaces.
>
> I agree that very very few users implement these two interfaces, except for
> some advanced users who implement their own StateBackend based on
> rocksdbStateBackend. (The "StateBackend" and "EmbeddedRocksDBStateBackend"
> are PublicEvolving interface)
>
> >> Only adding @Internal annotation to these two interfaces could be
> considered as an implicit hint that users can replace the default behaviour
> with their own implementation.



> As mentioned in Flink's “API compatibility guarantees”[1], "any API without
> such an annotation is considered internal to Flink, with no guarantees
> being provided."
> I think these interfaces are treated as internal interfaces regardless of
> whether they are marked @Internal.
>
> So, it's acceptable for me to either leave it unmarked or mark it as
> @Internal.
>



> >> Another suggestion is that we should start following FLIP-197[1](which
> is already accepted by the community) to add more info into the
> PublicEvolving annotation in order to kick off the graduation process.
>
>

> As for FLIP-197, I found that the proposal in this flip has not actually
> been implemented. Because FLINK-25352 [2] has not been resolved，it looks
> like we can't add more info into the PublicEvolving annotation.
>
>

> [1]
>
> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/upgrading/#api-compatibility-guarantees
> [2] https://issues.apache.org/jira/browse/FLINK-25352
>
> Best,
> Jinzhong
>
>
> On Mon, Mar 11, 2024 at 12:29 AM Jing Ge 
> wrote:
>
> > Hi Jinzhong,
> >
> > Sorry for the late reply. The key guideline of adding visibility
> annotation
> > is whether the interface/class is a customer-facing API. I was wondering
> if
> > SingleStateIterator and RocksDBRestoreOperation are exposed to users even
> > if they are interfaces, i.e. users can use their own implementation
> > classes. The flink-statebackend-rocksdb module has many more classes than
> > the FLIP described. Only adding @Internal annotation to these two
> > interfaces could be considered as an implicit hint that users can replace
> > the default behaviour with their own implementation.
> >
> > Another suggestion is that we should start following FLIP-197[1](which is
> > already accepted by the community) to add more info into the
> PublicEvolving
> > annotation in order to kick off the graduation process. WDYT?
> >
> > Best regards,
> > Jing
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-197%3A+API+stability+graduation+process
> >
> > On Mon, Feb 26, 2024 at 2:22 PM Jinzhong Li 
> > wrote:
> >
> > > Hi all,
> > >
> > > Thanks for all the feedback. It seems there are no more questions
> > > unaddressed.  I would like to open the voting thread after three days.
> > >
> > > Please let me know if you have any concerns, thanks!
> > >
> > > Best,
> > > Jinzhong Li
> > >
> > > On Mon, Feb 26, 2024 at 11:29 AM Yanfei Lei 
> wrote:
> > >
> > > > @Yun Tang
> > > > Thanks for the information, +1 for marking
> > > > `ConfigurableRocksDBOptionsFactory` as `PublicEvolving `.
> > > >
> > > > Best,
> > 

Re: [DISCUSS] FLIP-424: Asynchronous State APIs

[Jing Ge]

Hi Zakelly,

Thanks for your clarification! I would suggest explicitly adding
description(better highlight) into the FLIP that the original State API
will be deprecated. My gut feeling is that it is very important for anyone,
who will review the new design, to understand the long-term intention.

Best regards,
Jing

On Mon, Mar 11, 2024 at 8:01 AM Zakelly Lan  wrote:

> Hi Jing,
>
> Thanks for your comments!
>
> Sorry for not making this clear. Actually these APIs are in some newly
> introduced classes, which are located in a different package name with the
> original ones. I suggest we name it "State API V2" and the package name
> will be 'org.apache.flink.api.common.state.v2'. They work closely with the
> Datastream V2 and are annotated with @Experimental in first few versions,
> and will be promoted to @PublicEvolving alongside the DataStream V2. I
> agree that the name of interfaces should add 'async', and we will also
> support synchronous APIs in these new API classes. This approach allows us
> to:
>
>1. Have enough flexibility for rapid development with @Experimental
>annotation.
>2. Provide both sync and async style APIs for greater ease of use.
>3. Release ourselves from legacy constraints that prevent altering
>interface signatures, and we can do something like reorganizing the
>exceptions as FLIP-368[1] proposed.
>
> State API V2 will become the exclusive API set available to users when
> working with DataStream API V2. We may discuss the deprecation of original
> ones in future.
>
> WDYT?
>
>
> Best,
> Zakelly
>
>
> On Sun, Mar 10, 2024 at 8:34 PM Jing Ge 
> wrote:
>
>> Hi Zakelly,
>>
>> Thanks for your proposal. The FLIP looks in good shape. +1 for it! I'd
>> like
>> to ask some questions to understand your thoughts more precisely.
>>
>> 1. StateFuture is a new interface. At first glance, it should
>> be @Experimental. But according to our API Arch rule[1], it should be at
>> least @PublicEvolving, if it will be used by any existing PublicEvloving
>> classes. You might want to add this info to your FLIP, if we want to go
>> with this option.
>>
>> 2. The return types of methods in State and related sub-interfaces are
>> StateFuture. Since the old State interfaces already have those
>> methods
>> and Java does not allow method overload with the same method but different
>> return types. Do you want to change the old methods or use new interfaces?
>> My understanding is that, according to the description in the
>> "Compatibility, Deprecation, and Migration Plan'' section in the FLIP, new
>> interfaces will be defined alongside the old interfaces. I guess the
>> long-term intention of this FLIP is not to deprecate the synchronous State
>> API. Both State APIs will be supported for different scenarios. In this
>> case, does it make sense to:
>>
>> 2.1 annotated all new interfaces with @Experimental to have the
>> flexibility for further modifications?
>> 2.2 use different names e.g. AsyncState etc. to avoid potential
>> human mistakes while coding(e.g. import wrong package by mistake etc.) and
>> ease the job development with state?
>>
>> Best regards,
>> Jing
>>
>>
>> [1]
>>
>> https://github.com/apache/flink/blob/d6a4eb966fbc47277e07b79e7c64939a62eb1d54/flink-architecture-tests/flink-architecture-tests-production/src/main/java/org/apache/flink/architecture/rules/ApiAnnotationRules.java#L99
>>
>> On Thu, Mar 7, 2024 at 9:49 AM Zakelly Lan  wrote:
>>
>> > Hi devs,
>> >
>> > I'd like to start a discussion on a sub-FLIP of FLIP-423: Disaggregated
>> > State Storage and Management[1], which is a joint work of Yuan Mei,
>> Zakelly
>> > Lan, Jinzhong Li, Hangxiang Yu, Yanfei Lei and Feng Wang:
>> >
>> >  - FLIP-424: Asynchronous State APIs [2]
>> >
>> > This FLIP introduces new APIs for asynchronous state access.
>> >
>> > Please make sure you have read the FLIP-423[1] to know the whole story,
>> and
>> > we'll discuss the details of FLIP-424[2] under this mail. For the
>> > discussion of overall architecture or topics related with multiple
>> > sub-FLIPs, please post in the previous mail[3].
>> >
>> > Looking forward to hearing from you!
>> >
>> > [1] https://cwiki.apache.org/confluence/x/R4p3EQ
>> > [2] https://cwiki.apache.org/confluence/x/SYp3EQ
>> > [3] https://lists.apache.org/thread/ct8smn6g9y0b8730z7rp9zfpnwmj8vf0
>> >
>> >
>> > Best,
>> > Zakelly
>> >
>>
>

Re: [DISCUSS] FLIP-425: Asynchronous Execution Model

[Jing Ge]

Hi Yanfei,

Thanks for your clarification! Now I got a much clear picture and I am
still trying to understand your thoughts for some of those questions:


> How many mails are encapsulated depends on how the user writes the
> code. The statements in a `then()` will be wrapped into a mail.
> StateFuture is a restricted version of CompletableFuture, their basic
> semantics are consistent.
>

Conceptually, users can write a chain of many async calls, i.e. many then()
calls. And all these calls for Record A must be executed in order, while
Record B should stay at the Blocking buffer. What is the benefit for users
to build a chain of mails instead of just one mail(it is still async)? Is
there any best practices or guidelines to teach/tell users when and how
many async calls in a chain could/should be built?

> The challenge arises in determining when all the processing logic
associated with Record A is fully executed. To address this, we have
adopted a reference counting mechanism that tracks ongoing operations
(either processing input or executing a callback) related to a single
record.

> We describe this in the "Error handling"[2] section. This FLIP also
> adopts the design from FLIP-368, ensuring that all state interfaces
> throw unchecked exceptions and, consequently, do not declare any
> exceptions in their signatures. In cases where an exception occurs
> while accessing the state, the job should fail.
>

My question was not about how exceptions will be defined. I am not sure how
unchecked exceptions handling will be implemented. I was wondering if
exceptions in the mail chain would have an impact on the reference
counting? E.g. in Fig 5, if an exception happened in the value(),
update(int), or function within then(), any -1 counting might be missed?
Maybe a UT to cover all kinds of cases, i.e. happy paths and unhappy paths,
would make sense.

Best regards,
Jing

On Mon, Mar 11, 2024 at 3:58 AM Yanfei Lei  wrote:

> Hi Jing,
>
> Thanks for your thoughtful feedback!
>
> > does it mean that there will be three mails for Read, Update, and Output
> ?
>
> With this example, there are two mails. The Read is processed by
> `mailboxDefaultAction`[1], and the Update and Output are encapsulated
> as mail.
>
> > does it make sense to encapsulate one mail instead of 3 mails with more
> overhead?
>
>

> How many mails are encapsulated depends on how the user writes the
> code. The statements in a `then()` will be wrapped into a mail.
> StateFuture is a restricted version of CompletableFuture, their basic
> semantics are consistent.
>
>

> > Would you like to add more description for cases when exceptions
> happened? E.g. when reading or/and updating State throws IOExceptions.
>
>

> We describe this in the "Error handling"[2] section. This FLIP also
> adopts the design from FLIP-368, ensuring that all state interfaces
> throw unchecked exceptions and, consequently, do not declare any
> exceptions in their signatures. In cases where an exception occurs
> while accessing the state, the job should fail.
>



> > Is it correct to understand that AEC is stateless?
>
> Great perspective, yes, it can be understood that way.
> AEC is a task-level component. When the job fails or is restarted, the
> runtime status in AEC will be reset.
> In fact, we have considered taking a snapshot of the status in AEC and
> storing it in a checkpoint like "unaligned checkpoint", but since
> callback cannot be serialized, this idea is not feasible for the time
> being.
>
> > would you like to add Pseudo-code for the inFilghtReocordNum decrement
> to help us understand the logic better?
>
> This part of the code is a bit scattered, we will try to abstract a
> pseudo-code. You can first refer to the RecordContext-related code [3]
> in the PoC to understand it.
>
> [1]
> https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/mailbox/MailboxProcessor.java#L81
> [2]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-425%3A+Asynchronous+Execution+Model#FLIP425:AsynchronousExecutionModel-ErrorHandling
> [3]
> https://github.com/ververica/flink-poc/blob/disagg-poc-2/flink-runtime/src/main/java/org/apache/flink/runtime/state/async/RecordContext.java#L77
>
>
> Best,
> Yanfei
>
> Jing Ge  于2024年3月10日周日 23:47写道：
> >
> > Hi Yanfei,
> >
> > Thanks for your proposal! The FLIP contains a lot of great new ideas. I'd
> > like to ask some questions to make sure we are on the same page.
> >
> > > For the asynchronous interface, Record A should run with Read, Update
> and
> > Output, while Record B should stay at the Blocking buffer.
> >
> > 1. With this exa

Re: [VOTE] FLIP-420: Add API annotations for RocksDB StateBackend user-facing classes

[Jing Ge]

Hi Jinzhong,

Thanks for driving this topic and sorry for just joining the discussion
now. I replied in your discussion thread. Would you like to take a look
and let's keep the discussion there? I will come back to this thread and
vote once the discussion is done. Thanks!

Best regards,
Jing

On Thu, Mar 7, 2024 at 4:39 AM Zakelly Lan  wrote:

> +1 non-binding
>
> Thanks for proposing this.
>
> Best,
> Zakelly
>
> On Thu, Mar 7, 2024 at 10:13 AM Yanfei Lei  wrote:
>
> > +1(binding) for this vote.
> >
> > Hangxiang Yu  于2024年3月7日周四 09:54写道：
> > >
> > > +1 (binding)
> > >
> > > On Thu, Mar 7, 2024 at 9:34 AM Yun Tang  wrote:
> > >
> > > > > +1 for this FLIP.
> > > > Sorry for not being clear in my previous reply, it's a binding vote.
> > > >
> > > > Best
> > > > Yun Tang
> > > > 
> > > > From: Jeyhun Karimov 
> > > > Sent: Thursday, March 7, 2024 4:40
> > > > To: dev@flink.apache.org 
> > > > Subject: Re: [VOTE] FLIP-420: Add API annotations for RocksDB
> > StateBackend
> > > > user-facing classes
> > > >
> > > > Hi Jinzhong,
> > > >
> > > > Thanks for the FLIP.
> > > >
> > > > +1 (non-binding)
> > > >
> > > > Regards,
> > > > Jeyhun
> > > >
> > > > On Wed, Mar 6, 2024 at 5:09 PM Yun Tang  wrote:
> > > >
> > > > > +1 for this FLIP.
> > > > >
> > > > >
> > > > >
> > > > > Best
> > > > > Yun Tang
> > > > > 
> > > > > From: Jinzhong Li 
> > > > > Sent: Wednesday, March 6, 2024 20:29
> > > > > To: dev@flink.apache.org 
> > > > > Subject: [VOTE] FLIP-420: Add API annotations for RocksDB
> > StateBackend
> > > > > user-facing classes
> > > > >
> > > > > Hi All,
> > > > >
> > > > > I'd like to start a vote on the FLIP-420: Add API annotations for
> > RocksDB
> > > > > StateBackend user-facing classes[1].
> > > > > The discussion thread is here [2].
> > > > >
> > > > > The vote will be open for at least 72 hours unless there is an
> > objection
> > > > or
> > > > > not enough votes.
> > > > >
> > > > >
> > > > > [1]https://cwiki.apache.org/confluence/x/JQs4EQ
> > > > > [2]
> https://lists.apache.org/thread/4t71lz2j2ft8hf90ylvtomynhr2qthoo
> > > > >
> > > > >
> > > > > Best,
> > > > > Jinzhong Li
> > > > >
> > > >
> > >
> > >
> > > --
> > > Best,
> > > Hangxiang.
> >
> >
> >
> > --
> > Best,
> > Yanfei
> >
>

Re: [DISCUSS]FLIP-420: Add API annotations for RocksDB StateBackend user-facing classes

[Jing Ge]

Hi Jinzhong,

Sorry for the late reply. The key guideline of adding visibility annotation
is whether the interface/class is a customer-facing API. I was wondering if
SingleStateIterator and RocksDBRestoreOperation are exposed to users even
if they are interfaces, i.e. users can use their own implementation
classes. The flink-statebackend-rocksdb module has many more classes than
the FLIP described. Only adding @Internal annotation to these two
interfaces could be considered as an implicit hint that users can replace
the default behaviour with their own implementation.

Another suggestion is that we should start following FLIP-197[1](which is
already accepted by the community) to add more info into the PublicEvolving
annotation in order to kick off the graduation process. WDYT?

Best regards,
Jing

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-197%3A+API+stability+graduation+process

On Mon, Feb 26, 2024 at 2:22 PM Jinzhong Li 
wrote:

> Hi all,
>
> Thanks for all the feedback. It seems there are no more questions
> unaddressed.  I would like to open the voting thread after three days.
>
> Please let me know if you have any concerns, thanks!
>
> Best,
> Jinzhong Li
>
> On Mon, Feb 26, 2024 at 11:29 AM Yanfei Lei  wrote:
>
> > @Yun Tang
> > Thanks for the information, +1 for marking
> > `ConfigurableRocksDBOptionsFactory` as `PublicEvolving `.
> >
> > Best,
> > Yanfei
> >
> > Yun Tang  于2024年2月23日周五 19:54写道：
> > >
> > > Hi Jinzhong,
> > >
> > > Thanks for driving this topic, and +1 for fixing the lack of
> annotation.
> > >
> > > @Yanfei the `ConfigurableRocksDBOptionsFactory` interface is introduced
> > for user extension, you can refer to the doc[1], which shows an example
> of
> > how to use this interface.
> > >
> > >
> > > [1]
> >
> https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/large_state_tuning/#tuning-rocksdb-memory
> > >
> > >
> > > Best
> > > Yun Tang
> > > 
> > > From: Yanfei Lei 
> > > Sent: Thursday, February 22, 2024 15:39
> > > To: dev@flink.apache.org 
> > > Subject: Re: [DISCUSS]FLIP-420: Add API annotations for RocksDB
> > StateBackend user-facing classes
> > >
> > > Hi Jinzhong,
> > > Thanks for driving this!
> > >
> > > 1. I'm wondering if `ConfigurableRocksDBOptionsFactory` will be used
> > > by users,  currently it looks like only developers use it in rocksdb
> > > state backend module. And Its only non-testing subclass
> > > "DefaultConfigurableOptionsFactory" is marked @Deprecated.
> > > 2. Regarding @Internal,  according to the comments, it is used for
> > > "Annotation to mark methods within stable, public APIs as an internal
> > > developer API."  So marking "SingleStateIterator" and
> > > "RocksDBRestoreOperation" as @Internal is acceptable for me.
> > >
> > > Best,
> > > Yanfei
> > >
> > > Jinzhong Li  于2024年1月25日周四 12:16写道：
> > > >
> > > > Hi Zakelly,
> > > >
> > > > Thanks for your comments!
> > > >
> > > > 1）I agree that almost no user would use "RocksDBStateUploader" and
> > > > "RocksDBStateDownloader" to do something. It's fine for me to keep
> them
> > > > unmarked.
> > > > 2）Regarding "SingleStateIterator", I think it's acceptable to either
> > leave
> > > > it unmarked or mark it as @Internal. I just consider that
> > > > SingleStateIterator is one interface with the "public" modifier and
> it
> > is
> > > > harmless to annotate it as @Internal.
> > > >
> > > >
> > > >
> > > >
> > > > Hi Hangxiang,
> > > >
> > > > Thanks for the reminder!
> > > >
> > > > It makes sense to mark RocksDBStateBackendFactory as Deprecated.
> > > >
> > > > Best,
> > > > Jinzhong Li
> > > >
> > > >
> > > > On Thu, Jan 25, 2024 at 10:22 AM Hangxiang Yu 
> > wrote:
> > > >
> > > > > Hi Jinzhong.
> > > > > Thanks for driving this!
> > > > > Some suggestions:
> > > > > 1. As RocksDBStateBackend marked as Deprecated, We should also
> > > > > mark RocksDBStateBackendFactory as Deprecated
> > > > > 2. Since 1.19 will be freezed in 1.26. Let's adjust the target
> > version to
> > > > > 1.20
> > > > >
> > > > >
> > > > > On Wed, Jan 24, 2024 at 11:50 PM Zakelly Lan <
> zakelly@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Jinzhong,
> > > > > >
> > > > > > Thanks for driving this! +1 for fixing the lack of annotation.
> > > > > >
> > > > > > I'm wondering if we really need to annotate
> *RocksDBStateUploader*
> > and
> > > > > > *RocksDBStateDownloader
> > > > > > *with @Internal, as they seem to be ordinary classes without
> > interacting
> > > > > > with other modules.
> > > > > > Also, I have reservations about annotating *SingleStateIterator*,
> > but I'd
> > > > > > like to hear others' opinions and won't insist on this.
> > > > > >
> > > > > > Best,
> > > > > > Zakelly
> > > > > >
> > > > > > On Wed, Jan 24, 2024 at 10:26 PM Jinzhong Li <
> > lijinzhong2...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi devs,
> > > > > > >
> > > > > > > I’m opening this thread to discuss about FLIP-420: Add API
> > 

Re: [DISCUSS] FLIP Suggestion: Externalize Kudu Connector from Bahir

[Jing Ge]

Hi Ferenc,

Thanks for the proposal! +1 for it!

Similar to what Leonard mentioned. I would suggest:
1. Use the "release" to define the release version of the Kudu connector
itself.
2. Optionally, add one more row underneath to describe which Flink versions
this release will be compatible with, e.g. 1.17, 1.18. I think it makes
sense to support at least two last Flink releases. An example could be
found at [1]

Best regards,
Jing

 [1] https://lists.apache.org/thread/jcjfy3fgpg5cdnb9noslq2c77h0gtcwp

On Sun, Mar 10, 2024 at 3:46 PM Yanquan Lv  wrote:

> Hi Ferenc, +1 for this FLIP.
>
> Ferenc Csaky  于2024年3月9日周六 01:49写道：
>
> > Thank you Jeyhun, Leonard, and Hang for your comments! Let me
> > address them from earliest to latest.
> >
> > > How do you plan the review process in this case (e.g. incremental
> > over existing codebase or cumulative all at once) ?
> >
> > I think incremental would be less time consuming and complex for
> > reviewers so I would leaning towards that direction. I would
> > imagine multiple subtasks for migrating the existing code, and
> > updating the deprecated interfaces, so those should be separate PRs and
> > the release can be initiated when everything is merged.
> >
> > > (1) About the release version, could you specify kudu connector version
> > instead of flink version 1.18 as external connector version is different
> > with flink?
> > > (2) About the connector config options, could you enumerate these
> > options so that we can review they’re reasonable or not?
> >
> > I added these to the FLIP, copied the current configs options as is,
> PTAL.
> >
> > > (3) Metrics is also key part of connector, could you add the supported
> > connector metrics to public interface as well?
> >
> > The current Bahir conenctor code does not include any metrics and I did
> > not plan to include them into the scope of this FLIP.
> >
> > > I think that how to state this code originally lived in Bahir may be in
> > the
> > FLIP.
> >
> > I might miss your point, but the FLIP contains this: "Migrating the
> > current code keeping the history and noting it explicitly it was forked
> > from the Bahir repository [2]." Pls. share if you meant something else.
> >
> > Best,
> > Ferenc
> >
> >
> >
> > On Friday, March 8th, 2024 at 10:42, Hang Ruan 
> > wrote:
> >
> > >
> > >
> > > Hi, Ferenc.
> > >
> > > Thanks for the FLIP discussion. +1 for the proposal.
> > > I think that how to state this code originally lived in Bahir may be in
> > the
> > > FLIP.
> > >
> > > Best,
> > > Hang
> > >
> > > Leonard Xu xbjt...@gmail.com 于2024年3月7日周四 14:14写道：
> > >
> > > > Thanks Ferenc for kicking off this discussion, I left some comments
> > here:
> > > >
> > > > (1) About the release version, could you specify kudu connector
> version
> > > > instead of flink version 1.18 as external connector version is
> > different
> > > > with flink ?
> > > >
> > > > (2) About the connector config options, could you enumerate these
> > options
> > > > so that we can review they’re reasonable or not?
> > > >
> > > > (3) Metrics is also key part of connector, could you add the
> supported
> > > > connector metrics to public interface as well?
> > > >
> > > > Best,
> > > > Leonard
> > > >
> > > > > 2024年3月6日 下午11:23，Ferenc Csaky ferenc.cs...@pm.me.INVALID 写道：
> > > > >
> > > > > Hello devs,
> > > > >
> > > > > Opening this thread to discuss a FLIP [1] about externalizing the
> > Kudu
> > > > > connector, as recently
> > > > > the Apache Bahir project were moved to the attic [2]. Some details
> > were
> > > > > discussed already
> > > > > in another thread [3]. I am proposing to externalize this connector
> > and
> > > > > keep it maintainable,
> > > > > and up to date.
> > > > >
> > > > > Best regards,
> > > > > Ferenc
> > > > >
> > > > > [1]
> > > > >
> >
> https://docs.google.com/document/d/1vHF_uVe0FTYCb6PRVStovqDeqb_C_FKjt2P5xXa7uhE
> > > > > [2] https://bahir.apache.org/
> > > > > [3]
> https://lists.apache.org/thread/2nb8dxxfznkyl4hlhdm3vkomm8rk4oyq
> >
>

Re: [DISCUSS] FLIP-425: Asynchronous Execution Model

[Jing Ge]

Hi Yanfei,

Thanks for your proposal! The FLIP contains a lot of great new ideas. I'd
like to ask some questions to make sure we are on the same page.

> For the asynchronous interface, Record A should run with Read, Update and
Output, while Record B should stay at the Blocking buffer.

1. With this example, does it mean that there will be three mails for Read,
Update, and Output ?
2. If yes, since the Read, Update, and Output have to be executed before
Record B, does it make sense to encapsulate one mail instead of 3 mails
with more overhead? There must be some thoughts behind the design. Look
forward to it.

> The challenge arises in determining when all the processing logic
associated with Record A is fully executed. To address this, we have
adopted a reference counting mechanism that tracks ongoing operations
(either processing input or executing a callback) related to a single
record.

The design reminds me of the JVM reference counting for GC. Would you like
to add more description for cases when exceptions happened? E.g. when
reading or/and updating State throws IOExceptions.

> In more detail, AEC uses a inFilghtReocordNum  variable to trace the
current number of records in progress. Every time the AEC receives a new
record, the inFilghtReocordNum  increases by 1; when all processing and
callback for this record have completed, the inFilghtReocordNum  decreases
by 1. When processing one checkpoint mail, the current task thread will
give up the time slice through the yield() method of the mailbox executor,
so that the ongoing state request’s callback and the blocking state
requests will be drained first until inFlightRecordNum reduces to 0.

1. Speaking of draining, is it correct to understand that AEC is stateless?
E.g. AEC could be easily scaled out if it became a bottleneck.
2. There are Pseudo-code for the inFilghtReocordNum increment, would you
like to add Pseudo-code for the inFilghtReocordNum decrement to help us
understand the logic better?

The FLIP shows overall a great design! +1 for it! Looking forward to your
thoughts, thanks!

Best regards,
Jing

On Thu, Mar 7, 2024 at 10:05 AM Yanfei Lei  wrote:

> Hi devs,
>
> I'd like to start a discussion on FLIP-425: Asynchronous Execution
> Model[1], which is a sub-FLIP of FLIP-423: Disaggregated State Storage
> and Management[2].
>
> FLIP-425 introduces a non-blocking execution model leveraging the
> asynchronous APIs introduced in FLIP-424[3].
> For the whole story please read the FLIP-423[2], and this thread is
> aimed to discuss the details of "FLIP-425: Asynchronous Execution
> Model".
>
> Regarding the details of this FLIP, there have been some discussions
> here[4], mainly focusing on framework overhead profiling, watermark
> processing, etc. Please see link[4] for the context.
>
> Looking forward to hearing from you!
>
>
> [1] https://cwiki.apache.org/confluence/x/S4p3EQ
> [2] https://cwiki.apache.org/confluence/x/R4p3EQ
> [3] https://cwiki.apache.org/confluence/x/SYp3EQ
> [4] https://lists.apache.org/thread/ct8smn6g9y0b8730z7rp9zfpnwmj8vf0
>
> Best,
> Yanfei
>

Re: [DISCUSS] FLIP-424: Asynchronous State APIs

[Jing Ge]

Hi Zakelly,

Thanks for your proposal. The FLIP looks in good shape. +1 for it! I'd like
to ask some questions to understand your thoughts more precisely.

1. StateFuture is a new interface. At first glance, it should
be @Experimental. But according to our API Arch rule[1], it should be at
least @PublicEvolving, if it will be used by any existing PublicEvloving
classes. You might want to add this info to your FLIP, if we want to go
with this option.

2. The return types of methods in State and related sub-interfaces are
StateFuture. Since the old State interfaces already have those methods
and Java does not allow method overload with the same method but different
return types. Do you want to change the old methods or use new interfaces?
My understanding is that, according to the description in the
"Compatibility, Deprecation, and Migration Plan'' section in the FLIP, new
interfaces will be defined alongside the old interfaces. I guess the
long-term intention of this FLIP is not to deprecate the synchronous State
API. Both State APIs will be supported for different scenarios. In this
case, does it make sense to:

2.1 annotated all new interfaces with @Experimental to have the
flexibility for further modifications?
2.2 use different names e.g. AsyncState etc. to avoid potential
human mistakes while coding(e.g. import wrong package by mistake etc.) and
ease the job development with state?

Best regards,
Jing


[1]
https://github.com/apache/flink/blob/d6a4eb966fbc47277e07b79e7c64939a62eb1d54/flink-architecture-tests/flink-architecture-tests-production/src/main/java/org/apache/flink/architecture/rules/ApiAnnotationRules.java#L99

On Thu, Mar 7, 2024 at 9:49 AM Zakelly Lan  wrote:

> Hi devs,
>
> I'd like to start a discussion on a sub-FLIP of FLIP-423: Disaggregated
> State Storage and Management[1], which is a joint work of Yuan Mei, Zakelly
> Lan, Jinzhong Li, Hangxiang Yu, Yanfei Lei and Feng Wang:
>
>  - FLIP-424: Asynchronous State APIs [2]
>
> This FLIP introduces new APIs for asynchronous state access.
>
> Please make sure you have read the FLIP-423[1] to know the whole story, and
> we'll discuss the details of FLIP-424[2] under this mail. For the
> discussion of overall architecture or topics related with multiple
> sub-FLIPs, please post in the previous mail[3].
>
> Looking forward to hearing from you!
>
> [1] https://cwiki.apache.org/confluence/x/R4p3EQ
> [2] https://cwiki.apache.org/confluence/x/SYp3EQ
> [3] https://lists.apache.org/thread/ct8smn6g9y0b8730z7rp9zfpnwmj8vf0
>
>
> Best,
> Zakelly
>

Re: [DISCUSS] FLIP-399: Flink Connector Doris

[Jing Ge]

Thanks for the clarification. It makes sense to me.

Best regards,
Jing

On Wed, Mar 6, 2024 at 10:03 AM wudi <676366...@qq.com.invalid> wrote:

> Hi Jing Ge, thanks for your suggestions.
>
> 1. Currently, the Flink Doris Connector is compatible with Flink versions
> 1.15-1.18. SupportsCommitter[1] seems to be introduced in Flink 1.19, and
> most users may not have upgraded their Flink environments to that version
> yet. Modifying it now could lead to incompatibilities. I think we can
> postpone the modification and make it together with other connectors. What
> do you think?
>
> 2. Yes, currently DorisSource only supports batch reading, typically used
> for data synchronization and ETL. Streaming reading is not supported yet,
> which requires the capability of Doris Binlog (mentioned in the Doris 2024
> RoadMap[2]). Streaming reading can be used to capture incremental events
> from the database, making it more convenient for users to process real-time
> data newly added to Doris.
>
> 3. E2ECase[3] for DorisSource have been added, and the TestPlan in the
> FLIP has been modified.
>
> [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-372%3A+Enhance+and+synchronize+Sink+API+to+match+the+Source+API
> [2] https://github.com/apache/doris/issues/30669
> [3]
> https://github.com/apache/doris-flink-connector/blob/master/flink-doris-connector/src/test/java/org/apache/doris/flink/tools/cdc/DorisDorisE2ECase.java
>
> Brs,
> di.wu
>
>
> > 2024年3月6日 06:12，Jing Ge  写道：
> >
> > Hi Di,
> >
> > Thanks for your proposal. +1 for the contribution. I'd like to know your
> > thoughts about the following questions:
> >
> > 1. According to your clarification of the exactly-once, thanks for it
> BTW,
> > no PreCommitTopology is required. Does it make sense to let DorisSink[1]
> > implement SupportsCommitter, since the TwoPhaseCommittingSink is
> > deprecated[2] before turning the Doris connector into a Flink connector?
> > 2. OLAP engines are commonly used as the tail/downstream of a data
> pipeline
> > to support further e.g. ad-hoc query or cube with feasible
> pre-aggregation.
> > Just out of curiosity, would you like to share some real use cases that
> > will use OLAP engines as the source of a streaming data pipeline? Or it
> > will only be used as the source for the batch?
> > 3. The E2E test only covered sink[3], if I am not mistaken. Would you
> like
> > to test the source in E2E too?
> >
> > [1]
> >
> https://github.com/apache/doris-flink-connector/blob/43e0e5cf9b832854ea228fb093077872e3a311b6/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/DorisSink.java#L55
> > [2]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-372%3A+Enhance+and+synchronize+Sink+API+to+match+the+Source+API
> > [3]
> >
> https://github.com/apache/doris-flink-connector/blob/43e0e5cf9b832854ea228fb093077872e3a311b6/flink-doris-connector/src/test/java/org/apache/doris/flink/tools/cdc/MySQLDorisE2ECase.java#L96
> >
> > Best regards,
> > Jing
> >
> > On Tue, Mar 5, 2024 at 11:18 AM wudi <676366...@qq.com.invalid> wrote:
> >
> >> Hi, Jeyhun Karimov.
> >> Thanks for your question.
> >>
> >> - How to ensure Exactly-Once?
> >> 1. When the Checkpoint Barrier arrives, DorisSink will trigger the
> >> precommit api of StreamLoad to complete the persistence of data in Doris
> >> (the data will not be visible at this time), and will also pass this
> TxnID
> >> to the Committer.
> >> 2. When this Checkpoint of the entire Job is completed, the Committer
> will
> >> call the commit api of StreamLoad and commit TxnID to complete the
> >> visibility of the transaction.
> >> 3. When the task is restarted, the Txn with successful precommit and
> >> failed commit will be aborted based on the label-prefix, and Doris'
> abort
> >> API will be called. (At the same time, Doris will also abort
> transactions
> >> that have not been committed for a long time)
> >>
> >> ps: At the same time, this part of the content has been updated in FLIP
> >>
> >> - Because the default table model in Doris is Duplicate (
> >> https://doris.apache.org/docs/data-table/data-model/), which does not
> >> have a primary key, batch writing may cause data duplication, but UNIQ
> The
> >> model has a primary key, which ensures the idempotence of writing, thus
> >> achieving Exactly-Once
> >>
> >> Brs,
> >> di.wu
> >>
> >>
> >>> 2024年3月2日 17:50，Jeyhun Karimov  写道：
> >

Re: [DISCUSS] FLIP-423 ～FLIP-428: Introduce Disaggregated State Storage and Management in Flink 2.0

[Jing Ge]

+1 for Gyula's suggestion. I just finished FLIP-423 which introduced the
intention of the big change and high level architecture. Great content btw!
The only public interface change for this FLIP is one new config to use
ForSt. It makes sense to have one dedicated discussion thread for each
concrete system design.

@Zakelly The links in your mail do not work except the last one, because
the FLIP-xxx has been included into the url like
https://lists.apache.org/thread/nmd9qd0k8l94ygcfgllxms49wmtz1864FLIP-425.

NIT fix:

FLIP-424: https://lists.apache.org/thread/nmd9qd0k8l94ygcfgllxms49wmtz1864

FLIP-425: https://lists.apache.org/thread/wxn1j848fnfkqsnrs947wh1wmj8n8z0h

FLIP-426: https://lists.apache.org/thread/bt931focfl9971cwq194trmf3pkdsxrf

FLIP-427: https://lists.apache.org/thread/vktfzqvb7t4rltg7fdlsyd9sfdmrc4ft

FLIP-428: https://lists.apache.org/thread/vr8f91p715ct4lop6b3nr0fh4z5p312b

Best regards,
Jing




On Thu, Mar 7, 2024 at 10:14 AM Zakelly Lan  wrote:

> Hi everyone,
>
> Thank you all for a lively discussion here, and it is a good time to move
> forward to more detailed discussions. Thus we open several threads for
> sub-FLIPs:
>
> FLIP-424: https://lists.apache.org/thread/nmd9qd0k8l94ygcfgllxms49wmtz1864
> FLIP-425
> :
> https://lists.apache.org/thread/wxn1j848fnfkqsnrs947wh1wmj8n8z0h
> FLIP-426
> :
> https://lists.apache.org/thread/bt931focfl9971cwq194trmf3pkdsxrf
> FLIP-427
> :
> https://lists.apache.org/thread/vktfzqvb7t4rltg7fdlsyd9sfdmrc4ft
> FLIP-428
> :
> https://lists.apache.org/thread/vr8f91p715ct4lop6b3nr0fh4z5p312b
>
> If you want to talk about the overall architecture, roadmap, milestones or
> something related with multiple FLIPs, please post it here. Otherwise you
> can discuss some details in separate mails. Let's try to avoid repeated
> discussion in different threads. I will sync important messages here if
> there are any in the above threads.
>
> And reply to @Jeyhun: We will ensure the content between those FLIPs is
> consistent.
>
>
> Best,
> Zakelly
>
> On Thu, Mar 7, 2024 at 2:16 PM Yuan Mei  wrote:
>
> > I have been a bit busy these few weeks and sorry for responding late.
> >
> > The original thinking of keeping discussion within one thread is for
> easier
> > tracking and avoid for repeated discussion in different threads.
> >
> > For details, It might be good to start in different threads if needed.
> >
> > We will think of a way to better organize the discussion.
> >
> > Best
> > Yuan
> >
> >
> > On Thu, Mar 7, 2024 at 4:38 AM Jeyhun Karimov 
> > wrote:
> >
> > > Hi,
> > >
> > > + 1 for the suggestion.
> > > Maybe we can the discussion with the FLIPs with minimum dependencies
> > (from
> > > the other new/proposed FLIPs).
> > > Based on our discussion on a particular FLIP, the subsequent (or its
> > > dependent) FLIP(s) can be updated accordingly?
> > >
> > > Regards,
> > > Jeyhun
> > >
> > > On Wed, Mar 6, 2024 at 5:34 PM Gyula Fóra 
> wrote:
> > >
> > > > Hey all!
> > > >
> > > > This is a massive improvement / work. I just started going through
> the
> > > > Flips and have a more or less meta comment.
> > > >
> > > > While it's good to keep the overall architecture discussion here, I
> > think
> > > > we should still have separate discussions for each FLIP where we can
> > > > discuss interface details etc. With so much content if we start
> adding
> > > > minor comments here that will lead to nowhere but those discussions
> are
> > > > still important and we should have them in separate threads (one for
> > each
> > > > FLIP)
> > > >
> > > > What do you think?
> > > > Gyula
> > > >
> > > > On Wed, Mar 6, 2024 at 8:50 AM Yanfei Lei 
> wrote:
> > > >
> > > > > Hi team,
> > > > >
> > > > > Thanks for your discussion. Regarding FLIP-425, we have
> supplemented
> > > > > several updates to answer high-frequency questions:
> > > > >
> > > > > 1. We captured a flame graph of the Hashmap state backend in
> > > > > "Synchronous execution with asynchronous APIs"[1], which reveals
> that
> > > > > the framework overhead (including reference counting,
> future-related
> > > > > code and so on) consumes about 9% of the keyed operator CPU time.
> > > > > 2. We added a set of comparative experiments for watermark
> > processing,
> > > > > the performance of Out-Of-Order mode is 70% better than
> > > > > strictly-ordered mode under ~140MB state size. Instructions on how
> to
> > > > > run this test have also been added[2].
> > > > > 3. Regarding the order of StreamRecord, whether it has state access
> > or
> > > > > not. We supplemented a new *Strict order of 'processElement'*[3].
> > > > >
> > > > > [1]
> > > > >
> > > >
> > >
> >
> 

Re: [DISCUSS] Apache Bahir retired

[Jing Ge]

Hi Ferenc,

+1 for it! The proposal looks good, thanks! I would suggest starting a new
discuss thread following the new FLIP process created by Martijn[1].

[1] https://issues.apache.org/jira/browse/FLINK-34515

Best regards,
Jing

On Fri, Mar 1, 2024 at 5:25 PM Ferenc Csaky 
wrote:

> Hi,
>
> According to the current standards I created a Google doc for the FLIP
> [1]. I pointed it to the this discussion but if a dedicated discussion
> should be started for it, pls. let me know. PTAL.
>
> Regards,
> Ferenc
>
> [1]
> https://docs.google.com/document/d/1vHF_uVe0FTYCb6PRVStovqDeqb_C_FKjt2P5xXa7uhE/edit
>
>
>
>
> On Thursday, February 29th, 2024 at 09:48, Ferenc Csaky
>  wrote:
>
> >
> >
> > Thank you Marton and Martijn, I will proceed with the FLIP then.
> >
> > Best,
> > Ferenc
> >
> >
> >
> >
> > On Wednesday, February 28th, 2024 at 21:15, Martijn Visser
> martijnvis...@apache.org wrote:
> >
> > > Hi all,
> > >
> > > +1 to have a connector FLIP to propose a Kudu connector. I'm +0 overall
> > > because I don't see a lot of activity happening in newly proposed
> > > connectors, but if there's demand for it and people want to volunteer
> with
> > > contributions, there's no reason to block it.
> > >
> > > Best regards,
> > >
> > > Martijn
> > >
> > > On Wed, Feb 28, 2024 at 4:31 PM Márton Balassi
> balassi.mar...@gmail.com
> > >
> > > wrote:
> > >
> > > > Hi team,
> > > >
> > > > Thanks for bringing this up, Feri. I am +1 for maintaining the Kudu
> > > > connector as an external Flink connector.
> > > >
> > > > As per the legal/trademark questions this is actually fair game
> because one
> > > > does not donate code to a specific Apache project, technically it is
> > > > donated to the Apache Software foundation. Consequently moving
> between ASF
> > > > projects is fine, I would add a line to the NOTICE file stating that
> this
> > > > code originally lived in Bahir once we forked it.
> > > >
> > > > Although I did not find an easy to link precedent this is also
> implied in
> > > > the Attic Bahir site [1] ("notify us if you fork outside Apache")
> and in
> > > > this [2] Apache community dev list chat. We should notify the Attic
> team in
> > > > any case. :-)
> > > >
> > > > [1] https://attic.apache.org/projects/bahir.html
> > > > [2] https://lists.apache.org/thread/p31mz4x4dcvd43f026d5p05rpglzfyrt
> > > >
> > > > On Tue, Feb 27, 2024 at 10:09 AM Ferenc Csaky
> ferenc.cs...@pm.me.invalid
> > > > wrote:
> > > >
> > > > > Thank you Leonard for sharing your thoughts on this topic.
> > > > >
> > > > > I agree that complying with the Flink community connector
> > > > > development process would be a must, if there are no legal or
> > > > > copyright issues, I would be happy to take that task for this
> > > > > particular case.
> > > > >
> > > > > I am no legal/copyright expert myslef, but Bahir uses the Apache
> > > > > 2.0 license as well, so I believe it should be possible without
> too many
> > > > > complications, but I try to look for help on that front.
> > > > >
> > > > > FYI we are using and supporting a downstream fork of the Kudu
> connector
> > > > > on
> > > > > top of Flink 1.18 without any major modifications, so it is pretty
> up to
> > > > > date upstream as well.
> > > > >
> > > > > Regards,
> > > > > Ferenc
> > > > >
> > > > > On Monday, February 26th, 2024 at 10:29, Leonard Xu
> xbjt...@gmail.com
> > > > > wrote:
> > > > >
> > > > > > Hey, Ferenc
> > > > > >
> > > > > > Thanks for initiating this discussion. Apache Bahir is a great
> project
> > > > > > that provided significant assistance to many Apache Flink/Spark
> users.
> > > > > > It's
> > > > > > pity news that it has been retired.
> > > > > >
> > > > > > I believe that connectivity is crucial for building the
> ecosystem of
> > > > > > the
> > > > > > Flink such a computing engine. The community, or at least I,
> would
> > > > > > actively
> > > > > > support the introduction and maintenance of new connectors.
> Therefore,
> > > > > > adding a Kudu connector or other connectors from Bahir makes
> sense to me,
> > > > > > as long as we adhere to the development process for connectors
> in the
> > > > > > Flink
> > > > > > community[1].
> > > > > > I recently visited the Bahir Flink repository. Although the last
> > > > > > release
> > > > > > of Bahir Flink was in August ’22[2] which is compatible with
> Flink 1.14,
> > > > > > its latest code is compatible with Flink 1.17[3]. So, based on
> the
> > > > > > existing
> > > > > > codebase, developing an official Apache Flink connector for Kudu
> or other
> > > > > > connectors should be manageable. One point to consider is that
> if we're
> > > > > > not
> > > > > > developing a connector entirely from scratch but based on an
> existing
> > > > > > repository, we must ensure that there are no copyright issues.
> Here, "no
> > > > > > issues" means satisfying both Apache Bahir's and Apache Flink's
> copyright
> > > > > > requirements. Honestly, I'm not an expert in copyright or legal
> 

Re: [DISCUSS] FLIP-399: Flink Connector Doris

[Jing Ge]

Hi Di,

Thanks for your proposal. +1 for the contribution. I'd like to know your
thoughts about the following questions:

1. According to your clarification of the exactly-once, thanks for it BTW,
no PreCommitTopology is required. Does it make sense to let DorisSink[1]
implement SupportsCommitter, since the TwoPhaseCommittingSink is
deprecated[2] before turning the Doris connector into a Flink connector?
2. OLAP engines are commonly used as the tail/downstream of a data pipeline
to support further e.g. ad-hoc query or cube with feasible pre-aggregation.
Just out of curiosity, would you like to share some real use cases that
will use OLAP engines as the source of a streaming data pipeline? Or it
will only be used as the source for the batch?
3. The E2E test only covered sink[3], if I am not mistaken. Would you like
to test the source in E2E too?

[1]
https://github.com/apache/doris-flink-connector/blob/43e0e5cf9b832854ea228fb093077872e3a311b6/flink-doris-connector/src/main/java/org/apache/doris/flink/sink/DorisSink.java#L55
[2]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-372%3A+Enhance+and+synchronize+Sink+API+to+match+the+Source+API
[3]
https://github.com/apache/doris-flink-connector/blob/43e0e5cf9b832854ea228fb093077872e3a311b6/flink-doris-connector/src/test/java/org/apache/doris/flink/tools/cdc/MySQLDorisE2ECase.java#L96

Best regards,
Jing

On Tue, Mar 5, 2024 at 11:18 AM wudi <676366...@qq.com.invalid> wrote:

> Hi, Jeyhun Karimov.
> Thanks for your question.
>
> - How to ensure Exactly-Once?
> 1. When the Checkpoint Barrier arrives, DorisSink will trigger the
> precommit api of StreamLoad to complete the persistence of data in Doris
> (the data will not be visible at this time), and will also pass this TxnID
> to the Committer.
> 2. When this Checkpoint of the entire Job is completed, the Committer will
> call the commit api of StreamLoad and commit TxnID to complete the
> visibility of the transaction.
> 3. When the task is restarted, the Txn with successful precommit and
> failed commit will be aborted based on the label-prefix, and Doris' abort
> API will be called. (At the same time, Doris will also abort transactions
> that have not been committed for a long time)
>
> ps: At the same time, this part of the content has been updated in FLIP
>
> - Because the default table model in Doris is Duplicate (
> https://doris.apache.org/docs/data-table/data-model/), which does not
> have a primary key, batch writing may cause data duplication, but UNIQ The
> model has a primary key, which ensures the idempotence of writing, thus
> achieving Exactly-Once
>
> Brs,
> di.wu
>
>
> > 2024年3月2日 17:50，Jeyhun Karimov  写道：
> >
> > Hi,
> >
> > Thanks for the proposal. +1 for the FLIP.
> > I have a few questions:
> >
> > - How exactly the two (Stream Load's two-phase commit and Flink's
> two-phase
> > commit) combination will ensure the e2e exactly-once semantics?
> >
> > - The FLIP proposes to combine Doris's batch writing with the primary key
> > table to achieve Exactly-Once semantics. Could you elaborate more on
> that?
> > Why it is not the default behavior but a workaround?
> >
> > Regards,
> > Jeyhun
> >
> > On Sat, Mar 2, 2024 at 10:14 AM Yanquan Lv  wrote:
> >
> >> Thanks for driving this.
> >> The content is very detailed, it is recommended to add a section on Test
> >> Plan for more completeness.
> >>
> >> Di Wu  于2024年1月25日周四 15:40写道：
> >>
> >>> Hi all,
> >>>
> >>> Previously, we had some discussions about contributing Flink Doris
> >>> Connector to the Flink community [1]. I want to further promote this
> >> work.
> >>> I hope everyone will help participate in this FLIP discussion and
> provide
> >>> more valuable opinions and suggestions.
> >>> Thanks.
> >>>
> >>> [1] https://lists.apache.org/thread/lvh8g9o6qj8bt3oh60q81z0o1cv3nn8p
> >>>
> >>> Brs,
> >>> di.wu
> >>>
> >>>
> >>>
> >>> On 2023/12/07 05:02:46 wudi wrote:
> 
>  Hi all,
> 
>  As discussed in the previous email [1], about contributing the Flink
> >>> Doris Connector to the Flink community.
> 
> 
>  Apache Doris[2] is a high-performance, real-time analytical database
> >>> based on MPP architecture, for scenarios where Flink is used for data
> >>> analysis, processing, or real-time writing on Doris, Flink Doris
> >> Connector
> >>> is an effective tool.
> 
>  At the same time, Contributing Flink Doris Connector to the Flink
> >>> community will further expand the Flink Connectors ecosystem.
> 
>  So I would like to start an official discussion FLIP-399: Flink
> >>> Connector Doris[3].
> 
>  Looking forward to comments, feedbacks and suggestions from the
> >>> community on the proposal.
> 
>  [1] https://lists.apache.org/thread/lvh8g9o6qj8bt3oh60q81z0o1cv3nn8p
>  [2]
> >> https://doris.apache.org/docs/dev/get-starting/what-is-apache-doris/
>  [3]
> >>>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-399%3A+Flink+Connector+Doris
> 
> 
>  Brs,

Re: [DISCUSS] Add "Special Thanks" Page on the Flink Website

[Jing Ge]

+1 and thanks for the proposal!

Best regards,
Jing

On Tue, Mar 5, 2024 at 10:26 AM tison  wrote:

> I like this idea, so +1 for setting up.
>
> For anyone who have the access, this is a related thread about
> project-wise sponsor in the foundation level [1].
>
> Best,
> tison.
>
> [1] https://lists.apache.org/thread/2nv0x9gfk9lfnpb2315xgywyx84y97v6
>
> Jark Wu  于2024年3月5日周二 17:17写道：
> >
> > Sorry, I posted the wrong [7] link. The Flink benchmark ML link is:
> > https://lists.apache.org/thread/bkw6ozoflgltwfwmzjtgx522hyssfko6
> >
> >
> > On Tue, 5 Mar 2024 at 16:56, Jark Wu  wrote:
> >
> > > Hi all,
> > >
> > >
> > >
> > > I want to propose adding a "Special Thanks" page to our Apache Flink
> website [1]
> > >
> > > to honor and appreciate the
> > > companies and organizations that have sponsored
> > >
> > >
> > > machines or services for our project. The establishment of such a page
> serves as
> > >
> > >
> > > a public acknowledgment of our sponsors' contributions and
> simultaneously acts
> > >
> > >
> > > as a positive encouragement for other entities to consider supporting
> our project.
> > >
> > >
> > >
> > > Adding Per-Project Thanks pages is allowed by ASF policy[2], which
> says "PMCs
> > >
> > >
> > > may wish to provide recognition for third parties that provide
> software or services
> > >
> > >
> > > to the project's committers to further the goals of the project. These
> are typically
> > >
> > > called Per-Project Thanks pages".  Many Apache projects have added such
> > >
> > > pages, for example, Apache HBase[3] and Apache Mina[4].
> > >
> > >
> > > To initiate this idea, I have drafted a preliminary page under the
> > > "About" menu
> > >
> > > on the
> > > Flink website to specifically thank Alibaba and Ververica, by following
> > >
> > > the ASF guidelines and the Apache Mina project.
> > >
> > >
> > > page image:
> > >
> https://github.com/apache/flink/assets/5378924/e51aaffe-565e-46d1-90af-3900904afcc0
> > >
> > >
> > >
> > > Below companies are on the thanks list for their donation to Flink
> testing infrastructure:
> > >
> > > - Alibaba donated 8 machines (32vCPU,64GB) for running Flink CI builds
> [5].
> > >
> > >
> > > - Ververica donated 2 machines for hosting flink-ci repositories [6]
> and running Flink benchmarks [7].
> > >
> > >
> > > I may miss some other donations or companies, please add them if you
> know.
> > >
> > > Looking forward to your feedback about this proposal!
> > >
> > >
> > > Best,
> > >
> > > Jark
> > >
> > >
> > > [1]: https://flink.apache.org/
> > >
> > > [2]: https://www.apache.org/foundation/marks/linking#projectthanks
> > >
> > > [3]: https://hbase.apache.org/sponsors.html
> > >
> > > [4]: https://mina.apache.org/special-thanks.html
> > >
> > > [5]:
> > >
> https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines#AzurePipelines-AvailableCustomBuildMachines
> > >
> > > [6]:
> > >
> https://cwiki.apache.org/confluence/display/FLINK/Continuous+Integration
> > >
> > > [7]:
> > >
> https://lists.apache.org/thread.html/41a68c775753a7841896690c75438e0a497634102e676db880f30225@%3Cdev.flink.apache.org%3E
> > >
>

Re: [ANNOUNCE] New Apache Flink Committer - Jiabao Sun

[Jing Ge]

Congrats! Jiabao!

Best regards,
Jing

On Mon, Feb 19, 2024 at 2:38 PM Jeyhun Karimov  wrote:

> Congratulations, Jiabao!
> Well deserved!
>
> On Mon, Feb 19, 2024 at 2:26 PM gongzhongqiang 
> wrote:
>
> > Congratulations, Jiabao!
> >
> > Best,
> > Zhongqiang Gong
> >
> > Qingsheng Ren  于2024年2月19日周一 17:53写道：
> >
> > > Hi everyone,
> > >
> > > On behalf of the PMC, I'm happy to announce Jiabao Sun as a new Flink
> > > Committer.
> > >
> > > Jiabao began contributing in August 2022 and has contributed 60+
> commits
> > > for Flink main repo and various connectors. His most notable
> contribution
> > > is being the core author and maintainer of MongoDB connector, which is
> > > fully functional in DataStream and Table/SQL APIs. Jiabao is also the
> > > author of FLIP-377 and the main contributor of JUnit 5 migration in
> > runtime
> > > and table planner modules.
> > >
> > > Beyond his technical contributions, Jiabao is an active member of our
> > > community, participating in the mailing list and consistently
> > volunteering
> > > for release verifications and code reviews with enthusiasm.
> > >
> > > Please join me in congratulating Jiabao for becoming an Apache Flink
> > > committer!
> > >
> > > Best,
> > > Qingsheng (on behalf of the Flink PMC)
> > >
> >
>

[jira] [Created] (FLINK-34446) regression: alias is not in lateral join

[Jing Ge (Jira)]

Jing Ge created FLINK-34446:
---

 Summary: regression: alias is not in lateral join
 Key: FLINK-34446
 URL: https://issues.apache.org/jira/browse/FLINK-34446
 Project: Flink
  Issue Type: Bug
Reporter: Jing Ge


found one regression issue. Query working Flink 1.17.2, but failing with Flink 
1.18.+

 
{code:java}
-- Query working Flink 1.17.2, but failing with Flink 1.18.+

-- -- [ERROR] Could not execute SQL statement. Reason:

-- -- org.apache.calcite.sql.validate.SqlValidatorException: Table 's' not found

SELECT
a_or_b,
id, 
splits
FROM sample as s ,
LATERAL TABLE(split(s.id,'[01]')) lt(splits)
CROSS JOIN (VALUES ('A'), ('B')) AS cj(a_or_b); {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[ANNOUNCE] Flink 1.19 Cross-team testing & sync summary on 02/13/2024

[Jing Ge]

Hi folks,

enclosed please find the summary of the sync up meeting.

*- Cross-team testing*

*Unassigned release testing tickets[1] are looking for contributors to pick
them up. The guideline of how to test the feature could be found in the
ticket description.*

There are 3 release testing instruction tickets[2] still waiting for
response since authors are taking the Chinese new Loong year.

*- Release-1.19 nightly build*

Azure CI pipeline has been configured to trigger the 1.19 release branch
nightly build, 1.16 has been removed[3].
*- CI issues*

There are some CI test or instability issues that are under evaluation[4].

Huge shoutout to Matthias for observing CI stability and continuously
improving the release process.


Best regards,
Martijn, Lincoln, Yun, and Jing




[1]
https://issues.apache.org/jira/browse/FLINK-34399?jql=project%20%3D%20FLINK%20AND%20parent%20%3D%20FLINK-34285%20AND%20labels%20%3D%20release-testing%20AND%20assignee%20is%20EMPTY%20%20ORDER%20BY%20updatedDate
[2]
https://issues.apache.org/jira/browse/FLINK-34391?jql=project%20%3D%20FLINK%20AND%20parent%20%3D%20FLINK-34285%20AND%20labels%20is%20EMPTY%20AND%20status%20%3D%20OPEN%20%20ORDER%20BY%20updatedDate
[3] https://issues.apache.org/jira/browse/FLINK-34282
[4] https://cwiki.apache.org/confluence/display/FLINK/1.19+Release

Re: [DISCUSS] Alternative way of posting FLIPs

[Jing Ge]

+1 for option 1. The Github discussions look more like an overlap to the ML
instead of a wiki tool like Confluence.

Best regards,
Jing

On Fri, Feb 9, 2024 at 10:08 AM Yun Tang  wrote:

> For the first solution, I am concerned about whether we can view the
> history of design docs, which is supported by Confluence wiki and GitHub
> discussions. From my understanding, even the discussion history could let
> others know the evolution of this feature and the history of a design doc
> is also really important.
>
> Best
> Yun Tang
> 
> From: Piotr Nowojski 
> Sent: Thursday, February 8, 2024 14:17
> To: dev@flink.apache.org 
> Subject: Re: [DISCUSS] Alternative way of posting FLIPs
>
> +1 for the first option as well
>
> Best,
> Piotrek
>
> śr., 7 lut 2024 o 16:48 Matthias Pohl 
> napisał(a):
>
> > +1 for option 1 since it's a reasonable temporary workaround
> >
> > Moving to GitHub discussions would either mean moving the current FLIP
> > collection or having the FLIPs in two locations. Both options do not seem
> > to be optimal. Another concern I had was that GitHub Discussions wouldn't
> > allow integrating diagrams that easily. But it looks like they support
> > Mermaid [1] for diagrams.
> >
> > One flaw of the GoogleDocs approach is, though, that we have to rely on
> > diagrams being provided as PNG/JPG/SVG rather than draw.io diagrams.
> > draw.io
> > is more tightly integrated with the Confluence wiki which allows
> > editing/updating diagrams in the wiki rather than using some external
> tool.
> > Google Draw is also not that convenient to use in my opinion. Anyway,
> > that's a minor issue, I guess.
> >
> > Matthias
> >
> > [1]
> >
> >
> https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/creating-diagrams
> >
> > On Wed, Feb 7, 2024 at 3:30 PM Lincoln Lee 
> wrote:
> >
> > > Thanks Martijn moving this forward!
> > >
> > > +1 for the first solution, because as of now it looks like this is a
> > > temporary solution and we're still looking forward to the improvement
> by
> > > ASF Infra, when the access is ok for contributors, we can back to the
> > > current workflow.
> > >
> > > For solution 2, one visible downside is that it becomes inconvenient to
> > > look for flips (unless we permanently switch to github discussion).
> > >
> > > Looking forward to hearing more thoughts.
> > >
> > > Best,
> > > Lincoln Lee
> > >
> > >
> > > Martijn Visser  于2024年2月7日周三 21:51写道：
> > >
> > > > Hi all,
> > > >
> > > > ASF Infra has confirmed to me that only ASF committers can access the
> > > > ASF Confluence site since a recent change. One of the results of this
> > > > decision is that users can't signup and access Confluence, so only
> > > > committers+ can create FLIPs.
> > > >
> > > > ASF Infra hopes to improve this situation when they move to the Cloud
> > > > shortly (as in: some months), but they haven't committed on an actual
> > > > date. The idea would be that we find a temporary solution until
> anyone
> > > > can request access to Confluence.
> > > >
> > > > There are a couple of ways we could resolve this situation:
> > > > 1. Contributors create a Google Doc and make that view-only, and post
> > > > that Google Doc to the mailing list for a discussion thread. When the
> > > > discussions have been resolved, the contributor ask on the Dev
> mailing
> > > > list to a committer/PMC to copy the contents from the Google Doc, and
> > > > create a FLIP number for them. The contributor can then use that FLIP
> > > > to actually have a VOTE thread.
> > > > 2. We could consider moving FLIPs to "Discussions" on Github, like
> > > > Airflow does at https://github.com/apache/airflow/discussions
> > > > 3. Perhaps someone else has another good idea.
> > > >
> > > > Looking forward to your thoughts.
> > > >
> > > > Best regards,
> > > >
> > > > Martijn
> > > >
> > >
> >
>

Re: Security fixes for Flink 1.18 (flink-shaded)

[Jing Ge]

Hi Hong,

We were facing exactly the same guava issues and after analyzing those
CVEs, we got the same conclusion as what Chesnay mentioned.

Best regards,
Jing

On Fri, Feb 2, 2024 at 10:18 AM Chesnay Schepler  wrote:

> Guava CVEs don't apply because it's all about using it's createTempDir
> method which we don't use.
>
> Zookeeper CVE doesn't really apply because it's a server-side issue.
>
> On 02/02/2024 09:42, Martijn Visser wrote:
> > To add to this: we can't upgrade to flink-shaded 18.0, since we've just
> > reverted that for Flink 1.19 because of the performance regression. We
> will
> > need a new flink-shaded version to deal with these performance
> regressions.
> >
> > On Fri, Feb 2, 2024 at 9:39 AM Martijn Visser 
> > wrote:
> >
> >> Hi Hong,
> >>
> >> I do have objections: upgrading Flink-Shaded in a patch version is
> >> something that we should not take lightly, since it involves components
> >> that are used in the core functionality of Flink. We've seen in the past
> >> that changes in Flink Shaded have an impact on stability and
> performance. I
> >> would like to see how Flink is affected by these CVEs, since in almost
> all
> >> cases these are false-positives for Flink.
> >>
> >> Best regards,
> >>
> >> Martijn
> >>
> >> On Thu, Feb 1, 2024 at 4:22 PM Hong Liang  wrote:
> >>
> >>> Hi all,
> >>>
> >>> Recently, we detected some active CVEs on the flink-shaded-guava and
> >>> flink-shaded-zookeeper package used in Flink 1.18. Since Flink 1.18 is
> >>> still in support for security fixes, we should consider fixing this.
> >>> However, since the vulnerable package is coming from flink-shaded, I
> >>> wanted
> >>> to check if there are thoughts from the community around releasing a
> patch
> >>> version of flink-shaded.
> >>>
> >>> Problem:
> >>> Flink 1.18 uses guava 31.1-jre from flink-shaded-guava 17.0, which is
> >>> affected by CVE-2023-2976 (HIGH) [1] and CVE-2020-8908 (LOW) [2]. Flink
> >>> 1.18 also uses zookeeper 3.7.1, which is affected by CVE-2023-44981
> >>> (CRITICAL) [3].
> >>>
> >>> To fix, I can think of two options:
> >>> Option 1:
> >>> Upgrade Flink 1.18 to use flink.shaded.version 18.0. This is easiest
> as we
> >>> can backport the change for Flink 1.19 directly (after the performance
> >>> regression is addressed) [4]. However, there are also upgrades to
> jackson,
> >>> asm and netty in flink.shaded.version 1.18.
> >>>
> >>> Option 2:
> >>> Release flink.shaded.version 17.1, with just a bump in zookeeper and
> guava
> >>> versions. Then, upgrade Flink 1.18 to use this new flink.shaded.version
> >>> 17.1. This is harder, but keeps the changes contained and minimal.
> >>>
> >>> Given the version bump is on flink-shaded, which is relocated to keep
> the
> >>> usage of libraries contained within the flink runtime itself, I am
> >>> inclined
> >>> to go with Option 1, even though the change is slightly larger than
> just
> >>> the security fixes.
> >>>
> >>> Do people have any objections?
> >>>
> >>>
> >>> Regards,
> >>> Hong
> >>>
> >>> [1] https://nvd.nist.gov/vuln/detail/CVE-2023-2976
> >>> [2] https://nvd.nist.gov/vuln/detail/CVE-2020-8908
> >>> [3] https://nvd.nist.gov/vuln/detail/CVE-2023-44981
> >>> [4] https://issues.apache.org/jira/browse/FLINK-33705
> >>>
>
>

Re: [ANNOUNCE] Apache Flink 1.18.1 released

[Jing Ge]

Hi folks,

The bug has been fixed and PR at docker-library/official-images has been
merged. The official images are available now.

Best regards,
Jing

On Mon, Jan 22, 2024 at 11:39 AM Jing Ge  wrote:

> Hi folks,
>
> I am still working on the official images because of the issue
> https://issues.apache.org/jira/browse/FLINK-34165. Images under
> apache/flink are available.
>
> Best regards,
> Jing
>
> On Sun, Jan 21, 2024 at 11:06 PM Jing Ge  wrote:
>
>> Thanks Leonard for the feedback! Also thanks @Jark Wu  
>> @Chesnay
>> Schepler  and each and everyone who worked closely
>> with me for this release. We made it together!
>>
>> Best regards,
>> Jing
>>
>> On Sun, Jan 21, 2024 at 9:25 AM Leonard Xu  wrote:
>>
>>> Thanks Jing for driving the release, nice work!
>>>
>>> Thanks all who involved this release!
>>>
>>> Best,
>>> Leonard
>>>
>>> > 2024年1月20日 上午12:01，Jing Ge  写道：
>>> >
>>> > The Apache Flink community is very happy to announce the release of
>>> Apache
>>> > Flink 1.18.1, which is the first bugfix release for the Apache Flink
>>> 1.18
>>> > series.
>>> >
>>> > Apache Flink® is an open-source stream processing framework for
>>> > distributed, high-performing, always-available, and accurate data
>>> streaming
>>> > applications.
>>> >
>>> > The release is available for download at:
>>> > https://flink.apache.org/downloads.html
>>> >
>>> > Please check out the release blog post for an overview of the
>>> improvements
>>> > for this bugfix release:
>>> >
>>> https://flink.apache.org/2024/01/19/apache-flink-1.18.1-release-announcement/
>>> >
>>> > Please note: Users that have state compression should not migrate to
>>> 1.18.1
>>> > (nor 1.18.0) due to a critical bug that could lead to data loss. Please
>>> > refer to FLINK-34063 for more information.
>>> >
>>> > The full release notes are available in Jira:
>>> >
>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353640
>>> >
>>> > We would like to thank all contributors of the Apache Flink community
>>> who
>>> > made this release possible! Special thanks to @Qingsheng Ren @Leonard
>>> Xu
>>> > @Xintong Song @Matthias Pohl @Martijn Visser for the support during
>>> this
>>> > release.
>>> >
>>> > A Jira task series based on the Flink release wiki has been created for
>>> > 1.18.1 release. Tasks that need to be done by PMC have been explicitly
>>> > created separately. It will be convenient for the release manager to
>>> reach
>>> > out to PMC for those tasks. Any future patch release could consider
>>> cloning
>>> > it and follow the standard release process.
>>> > https://issues.apache.org/jira/browse/FLINK-33824
>>> >
>>> > Feel free to reach out to the release managers (or respond to this
>>> thread)
>>> > with feedback on the release process. Our goal is to constantly
>>> improve the
>>> > release process. Feedback on what could be improved or things that
>>> didn't
>>> > go so well are appreciated.
>>> >
>>> > Regards,
>>> > Jing
>>>
>>>

Re: [SUMMARY] Flink 1.19 Release Sync 01/23/2024

[Jing Ge]

Hi Feng,

Thanks for letting us know, it is a great feature. +1 to extend the code
freeze deadline.

Best regards,
Jing

On Thu, Jan 25, 2024 at 8:08 AM Feng Jin  wrote:

> Hi  everyone,
>
> Xuyang and I are currently working on FLIP-387[1], which aims to support
> named parameters for functions and procedures.
> This will make it more convenient for users to utilize functions and
> procedures with multiple parameters.
>
> We have divided the task into four sub-tasks, and we are currently working
> on them:
> https://issues.apache.org/jira/browse/FLINK-34055
> https://issues.apache.org/jira/browse/FLINK-34056
> https://issues.apache.org/jira/browse/FLINK-34057
>
> These tasks have already been developed and reviewed, and we expect them
> to be merged today(Jan 25th).
>
> However, there is still one remaining task:
> https://issues.apache.org/jira/browse/FLINK-34058.
> I have already completed the necessary development work for this task. It
> may still require 2-3 rounds of review before it is finalized.
> I anticipate that it will take another 2-3 days to complete.
>
>
> Therefore, I kindly request that we merge the pull request next Monday(Jan
> 29th). I apologize if this affects your related schedule.
>
> [1].  https://issues.apache.org/jira/browse/FLINK-34054
>
>
> Best regards,
> Feng Jin
>
> On Wed, Jan 24, 2024 at 10:00 PM Lincoln Lee 
> wrote:
>
>> Hi devs,
>>
>> I'd like to share some highlights from the release sync on 01/23/2024
>>
>>
>> *- Feature freeze*  *We plan to freeze the feature on Jan 26th. If there's
>> specific need for an extension, please confirm with RMs by replying this
>> mail.*
>>
>>
>> *- Features & issues tracking*  So far we've had 15 flips been marked
>> done(some documentation is still in progress), we also ask responsible
>> contributors to help update the status of the remaining items on the 1.19
>> wiki page [1], including *documentation and cross-team testing
>> requirements*,
>> this will help the release process.
>>
>>
>> *- Blockers*  There're performance regression and blocker issues are being
>> worked on:
>>   https://issues.apache.org/jira/browse/FLINK-34148
>>   https://issues.apache.org/jira/browse/FLINK-34007
>>   https://issues.apache.org/jira/browse/FLINK-34225
>>   Note that test instabilities will be upgraded to blocker if it is newly
>> introduced.
>>
>> *- Sync meeting* (https://meet.google.com/vcx-arzs-trv)
>>   The next release sync is *Jan 30th, 2024*. We'll switch to weekly
>> release
>> sync.
>>
>> [1] https://cwiki.apache.org/confluence/display/FLINK/1.19+Release
>>
>> Best,
>> Yun, Jing, Martijn and Lincoln
>>
>

[jira] [Created] (FLINK-34236) Evaluate strange unstable build after cleaning up CI machines

[Jing Ge (Jira)]

Jing Ge created FLINK-34236:
---

 Summary: Evaluate strange unstable build after cleaning up CI 
machines
 Key: FLINK-34236
 URL: https://issues.apache.org/jira/browse/FLINK-34236
 Project: Flink
  Issue Type: Improvement
  Components: Test Infrastructure
Reporter: Jing Ge


To check if it is one time issue because infra change or not.

 

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=56601=logs=e9d3d34f-3d15-59f4-0e3e-35067d100dfe=5d91035e-8022-55f2-2d4f-ab121508bf7e



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] Apache Flink 1.18.1 released

[Jing Ge]

Hi folks,

I am still working on the official images because of the issue
https://issues.apache.org/jira/browse/FLINK-34165. Images under
apache/flink are
available.

Best regards,
Jing

On Sun, Jan 21, 2024 at 11:06 PM Jing Ge  wrote:

> Thanks Leonard for the feedback! Also thanks @Jark Wu  
> @Chesnay
> Schepler  and each and everyone who worked closely
> with me for this release. We made it together!
>
> Best regards,
> Jing
>
> On Sun, Jan 21, 2024 at 9:25 AM Leonard Xu  wrote:
>
>> Thanks Jing for driving the release, nice work!
>>
>> Thanks all who involved this release!
>>
>> Best,
>> Leonard
>>
>> > 2024年1月20日 上午12:01，Jing Ge  写道：
>> >
>> > The Apache Flink community is very happy to announce the release of
>> Apache
>> > Flink 1.18.1, which is the first bugfix release for the Apache Flink
>> 1.18
>> > series.
>> >
>> > Apache Flink® is an open-source stream processing framework for
>> > distributed, high-performing, always-available, and accurate data
>> streaming
>> > applications.
>> >
>> > The release is available for download at:
>> > https://flink.apache.org/downloads.html
>> >
>> > Please check out the release blog post for an overview of the
>> improvements
>> > for this bugfix release:
>> >
>> https://flink.apache.org/2024/01/19/apache-flink-1.18.1-release-announcement/
>> >
>> > Please note: Users that have state compression should not migrate to
>> 1.18.1
>> > (nor 1.18.0) due to a critical bug that could lead to data loss. Please
>> > refer to FLINK-34063 for more information.
>> >
>> > The full release notes are available in Jira:
>> >
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353640
>> >
>> > We would like to thank all contributors of the Apache Flink community
>> who
>> > made this release possible! Special thanks to @Qingsheng Ren @Leonard Xu
>> > @Xintong Song @Matthias Pohl @Martijn Visser for the support during this
>> > release.
>> >
>> > A Jira task series based on the Flink release wiki has been created for
>> > 1.18.1 release. Tasks that need to be done by PMC have been explicitly
>> > created separately. It will be convenient for the release manager to
>> reach
>> > out to PMC for those tasks. Any future patch release could consider
>> cloning
>> > it and follow the standard release process.
>> > https://issues.apache.org/jira/browse/FLINK-33824
>> >
>> > Feel free to reach out to the release managers (or respond to this
>> thread)
>> > with feedback on the release process. Our goal is to constantly improve
>> the
>> > release process. Feedback on what could be improved or things that
>> didn't
>> > go so well are appreciated.
>> >
>> > Regards,
>> > Jing
>>
>>

Re: [ANNOUNCE] Apache Flink 1.18.1 released

[Jing Ge]

Thanks Leonard for the feedback! Also thanks @Jark Wu
 @Chesnay
Schepler  and each and everyone who worked closely with
me for this release. We made it together!

Best regards,
Jing

On Sun, Jan 21, 2024 at 9:25 AM Leonard Xu  wrote:

> Thanks Jing for driving the release, nice work!
>
> Thanks all who involved this release!
>
> Best,
> Leonard
>
> > 2024年1月20日 上午12:01，Jing Ge  写道：
> >
> > The Apache Flink community is very happy to announce the release of
> Apache
> > Flink 1.18.1, which is the first bugfix release for the Apache Flink 1.18
> > series.
> >
> > Apache Flink® is an open-source stream processing framework for
> > distributed, high-performing, always-available, and accurate data
> streaming
> > applications.
> >
> > The release is available for download at:
> > https://flink.apache.org/downloads.html
> >
> > Please check out the release blog post for an overview of the
> improvements
> > for this bugfix release:
> >
> https://flink.apache.org/2024/01/19/apache-flink-1.18.1-release-announcement/
> >
> > Please note: Users that have state compression should not migrate to
> 1.18.1
> > (nor 1.18.0) due to a critical bug that could lead to data loss. Please
> > refer to FLINK-34063 for more information.
> >
> > The full release notes are available in Jira:
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353640
> >
> > We would like to thank all contributors of the Apache Flink community who
> > made this release possible! Special thanks to @Qingsheng Ren @Leonard Xu
> > @Xintong Song @Matthias Pohl @Martijn Visser for the support during this
> > release.
> >
> > A Jira task series based on the Flink release wiki has been created for
> > 1.18.1 release. Tasks that need to be done by PMC have been explicitly
> > created separately. It will be convenient for the release manager to
> reach
> > out to PMC for those tasks. Any future patch release could consider
> cloning
> > it and follow the standard release process.
> > https://issues.apache.org/jira/browse/FLINK-33824
> >
> > Feel free to reach out to the release managers (or respond to this
> thread)
> > with feedback on the release process. Our goal is to constantly improve
> the
> > release process. Feedback on what could be improved or things that didn't
> > go so well are appreciated.
> >
> > Regards,
> > Jing
>
>

[ANNOUNCE] Apache Flink 1.18.1 released

[Jing Ge]

The Apache Flink community is very happy to announce the release of Apache
Flink 1.18.1, which is the first bugfix release for the Apache Flink 1.18
series.

Apache Flink® is an open-source stream processing framework for
distributed, high-performing, always-available, and accurate data streaming
applications.

The release is available for download at:
https://flink.apache.org/downloads.html

Please check out the release blog post for an overview of the improvements
for this bugfix release:
https://flink.apache.org/2024/01/19/apache-flink-1.18.1-release-announcement/

Please note: Users that have state compression should not migrate to 1.18.1
(nor 1.18.0) due to a critical bug that could lead to data loss. Please
refer to FLINK-34063 for more information.

The full release notes are available in Jira:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353640

We would like to thank all contributors of the Apache Flink community who
made this release possible! Special thanks to @Qingsheng Ren @Leonard Xu
 @Xintong Song @Matthias Pohl @Martijn Visser for the support during this
release.

A Jira task series based on the Flink release wiki has been created for
1.18.1 release. Tasks that need to be done by PMC have been explicitly
created separately. It will be convenient for the release manager to reach
out to PMC for those tasks. Any future patch release could consider cloning
it and follow the standard release process.
https://issues.apache.org/jira/browse/FLINK-33824

Feel free to reach out to the release managers (or respond to this thread)
with feedback on the release process. Our goal is to constantly improve the
release process. Feedback on what could be improved or things that didn't
go so well are appreciated.

Regards,
Jing

[jira] [Created] (FLINK-34165) It seems that Apache download link has been changed

[Jing Ge (Jira)]

Jing Ge created FLINK-34165:
---

 Summary: It seems that Apache download link has been changed
 Key: FLINK-34165
 URL: https://issues.apache.org/jira/browse/FLINK-34165
 Project: Flink
  Issue Type: Bug
  Components: flink-docker
Affects Versions: 1.18.1, 1.17.2, 1.16.3, 1.15.4
Reporter: Jing Ge
 Attachments: image-2024-01-19-07-55-07-775.png

The link 
[https://www.apache.org/dist/flink/flink-1.17.2/flink-1.17.2-bin-scala_2.12.tgz.asc][1]
 worked previously now redirect to a list page which leads to a wrong 
flink.tgz.asc with HTML instead of expected signature.

!image-2024-01-19-07-55-07-775.png!

The link should be replace with 
https://downloads.apache.org/flink/flink-1.17.2/flink-1.17.2-bin-scala_2.12.tgz.asc

 

[1] 
https://github.com/apache/flink-docker/blob/627987997ca7ec86bcc3d80b26df58aa595b91af/1.17/scala_2.12-java11-ubuntu/Dockerfile#L48C19-L48C101



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-34075) Mark version as released in Jira (need PMC role)

[Jing Ge (Jira)]

Jing Ge created FLINK-34075:
---

 Summary: Mark version as released in Jira (need PMC role)
 Key: FLINK-34075
 URL: https://issues.apache.org/jira/browse/FLINK-34075
 Project: Flink
  Issue Type: Sub-task
Reporter: Jing Ge


In JIRA, inside [version 
management|https://issues.apache.org/jira/plugins/servlet/project-config/FLINK/versions],
 hover over the current release and a settings menu will appear. Click Release, 
and select today’s date.

(Note: Only PMC members have access to the project administration. If you do 
not have access, ask on the mailing list for assistance.)

 

h3. Expectations
 * Release tagged in the source code repository
 * Release version finalized in JIRA. (Note: Not all committers have 
administrator access to JIRA. If you end up getting permissions errors ask on 
the mailing list for assistance)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-34074) Verify artifacts related expectations

[Jing Ge (Jira)]

Jing Ge created FLINK-34074:
---

 Summary: Verify artifacts related expectations
 Key: FLINK-34074
 URL: https://issues.apache.org/jira/browse/FLINK-34074
 Project: Flink
  Issue Type: Sub-task
Reporter: Jing Ge


h3. Expectations
 * Maven artifacts released and indexed in the [Maven Central 
Repository|https://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.apache.flink%22]
 (usually takes about a day to show up)
 * Source & binary distributions available in the release repository of 
[https://dist.apache.org/repos/dist/release/flink/]
 * Dev repository [https://dist.apache.org/repos/dist/dev/flink/] is empty
 * Website contains links to new release binaries and sources in download page
 * (for minor version updates) the front page references the correct new major 
release version and directs to the correct link



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-34073) Remove old release candidates from dist.apache.org

[Jing Ge (Jira)]

Jing Ge created FLINK-34073:
---

 Summary: Remove old release candidates from dist.apache.org
 Key: FLINK-34073
 URL: https://issues.apache.org/jira/browse/FLINK-34073
 Project: Flink
  Issue Type: Sub-task
Reporter: Jing Ge
Assignee: Jing Ge


h3. Remove old release candidates from [dist.apache.org|http://dist.apache.org/]

Remove the old release candidates from 
[https://dist.apache.org/repos/dist/dev/flink] using Subversion.
{code:java}
$ svn checkout https://dist.apache.org/repos/dist/dev/flink --depth=immediates
$ cd flink
$ svn remove flink-${RELEASE_VERSION}-rc*
$ svn commit -m "Remove old release candidates for Apache Flink 
${RELEASE_VERSION}
{code}
 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[RESULT] [VOTE] Release 1.18.1, release candidate #2

[Jing Ge]

I'm happy to announce that we have unanimously approved this release.

There are 10 approving votes, 4 of which are binding:

- Jiabao Sun
- Hang Ruan
- Gongzhong Qiang
- Martijn Visser (binding)
- Rui Fan
- Yun Tang
- Jark Wu (binding)
- Qingsheng Ren (binding)
- Leonard Xu (binding)
- Sergey Nuyanzin


There are no disapproving votes.

I'll work on the steps to finalize the release and will send out the
announcement as soon as that has been completed.

Thanks, everyone!

Best regards,
Jing

Re: [VOTE] Release 1.18.1, release candidate #2

[Jing Ge]

Thank you all! I will post the vote result soon.

Best regards,
Jing

On Wed, Jan 10, 2024 at 12:34 PM Sergey Nuyanzin 
wrote:

> +1 (non-binding)
>
> -verified checksums
> -verified signatures
> -checked release tag
> -verified that there is no binary in source
> -built from sources
>
> On Wed, Jan 10, 2024 at 12:30 PM Leonard Xu  wrote:
>
> > +1 (binding)
> >
> > - verified signatures
> > - verified hashsums
> > - checked Github release tag
> > - started SQL Client, used MySQL CDC connector to capture data change
> from
> > database , the result is expected
> > - reviewed the web PR
> > - reviewed the release notes PR
> >
> > Best,
> > Leonard
> >
> > > 2024年1月10日 上午10:48，Qingsheng Ren  写道：
> > >
> > > +1 (binding)
> > >
> > > - Built from source
> > > - Verified checksum and signature
> > > - Verified that no binary exist in source
> > > - Started a standalone cluster and submit Kafka consuming and producing
> > job
> > > with SQL client
> > > - Reviewed web PR
> > >
> > > Thanks for driving this, Jing!
> > >
> > > Best,
> > > Qingsheng
> > >
> > > On Mon, Jan 8, 2024 at 8:02 PM Jark Wu  wrote:
> > >
> > >> Thanks Jing for driving this.
> > >>
> > >> +1 (binding)
> > >>
> > >> - Build and compile the source code locally: *OK*
> > >> - Verified signatures and hashes: *OK*
> > >> - Checked no missing artifacts in the staging area: *OK*
> > >> - Reviewed the website release PR: *OK*
> > >> - Went through the quick start: *OK*
> > >>  * Started a cluster and ran the examples
> > >>  * Verified web ui and log output, nothing unexpected
> > >>
> > >> Best,
> > >> Jark
> > >>
> > >> On Thu, 28 Dec 2023 at 20:59, Yun Tang  wrote:
> > >>
> > >>> Thanks Jing for driving this release.
> > >>>
> > >>> +1 (non-binding)
> > >>>
> > >>>
> > >>>  *
> > >>> Download artifacts and verify the signatures.
> > >>>  *
> > >>> Verified the web PR
> > >>>  *
> > >>> Verified the number of Python packages is 11
> > >>>  *
> > >>> Started a local cluster and verified FLIP-291 to see the rescale
> > results.
> > >>>  *
> > >>> Verified the jar packages were built with JDK8
> > >>>
> > >>> Best
> > >>> Yun Tang
> > >>>
> > >>>
> > >>> 
> > >>> From: Rui Fan <1996fan...@gmail.com>
> > >>> Sent: Thursday, December 28, 2023 10:54
> > >>> To: dev@flink.apache.org 
> > >>> Subject: Re: [VOTE] Release 1.18.1, release candidate #2
> > >>>
> > >>> Thanks Jing for driving this release!
> > >>>
> > >>> +1(non-binding)
> > >>>
> > >>> - Downloaded artifacts
> > >>> - Verified signatures and sha512
> > >>> - The source archives do not contain any binaries
> > >>> - Verified web PR
> > >>> - Build the source with Maven 3 and java8 (Checked the license as
> well)
> > >>> - bin/start-cluster.sh with java8, it works fine and no any
> unexpected
> > >> LOG-
> > >>> Ran demo, it's fine:  bin/flink
> > >>> runexamples/streaming/StateMachineExample.jar
> > >>>
> > >>> Best,
> > >>> Rui
> > >>>
> > >>> On Wed, Dec 27, 2023 at 8:45 PM Martijn Visser <
> > martijnvis...@apache.org
> > >>>
> > >>> wrote:
> > >>>
> > >>>> Hi Jing,
> > >>>>
> > >>>> Thanks for driving this.
> > >>>>
> > >>>> +1 (binding)
> > >>>>
> > >>>> - Validated hashes
> > >>>> - Verified signature
> > >>>> - Verified that no binaries exist in the source archive
> > >>>> - Build the source with Maven via mvn clean install
> > >>>> -Pcheck-convergence -Dflink.version=1.18.1
> > >>>> - Verified licenses
> > >>>> - Verified web PR
> > >>>> - Started a cluster and the Flink SQL client, successfully read and
> > >>>> wrote with the Kafka connector to Confluent Cloud with AVRO and
> Schema
> > >>>> Registry enabled
> > >>>> - Started a cluster and submitted a job that checkpoints to GCS
> > without
> > >>>> problems
> > >>>>
> > >>>> Best regards,
> > >>>>
> > >>>> Martijn
> > >>>>
> > >>>> On Thu, Dec 21, 2023 at 4:55 AM gongzhongqiang
> > >>>>  wrote:
> > >>>>>
> > >>>>> Thanks Jing Ge for driving this release.
> > >>>>>
> > >>>>> +1 (non-binding), I have checked:
> > >>>>> [✓] The checksums and signatures are validated
> > >>>>> [✓] The tag checked is fine
> > >>>>> [✓] Built from source is passed
> > >>>>> [✓] The flink-web PR is reviewed and checked
> > >>>>>
> > >>>>>
> > >>>>> Best，
> > >>>>> Zhongqiang Gong
> > >>>>
> > >>>
> > >>
> >
> >
>
> --
> Best regards,
> Sergey
>

Re: [VOTE] Accept Flink CDC into Apache Flink

[Jing Ge]

+1 (not binding)

Best regards,
Jing

On Tue, Jan 9, 2024 at 8:46 AM Xintong Song  wrote:

> +1 (binding)
>
> Best,
>
> Xintong
>
>
>
> On Tue, Jan 9, 2024 at 3:31 PM Benchao Li  wrote:
>
> > +1 (non-binding)
> >
> > Feng Wang  于2024年1月9日周二 15:29写道：
> > >
> > > +1 non-binding
> > > Regards,
> > > Feng
> > >
> > > On Tue, Jan 9, 2024 at 3:05 PM Leonard Xu  wrote:
> > >
> > > > Hello all,
> > > >
> > > > This is the official vote whether to accept the Flink CDC code
> > contribution
> > > >  to Apache Flink.
> > > >
> > > > The current Flink CDC code, documentation, and website can be
> > > > found here:
> > > > code: https://github.com/ververica/flink-cdc-connectors <
> > > > https://github.com/ververica/flink-cdc-connectors>
> > > > docs: https://ververica.github.io/flink-cdc-connectors/ <
> > > > https://ververica.github.io/flink-cdc-connectors/>
> > > >
> > > > This vote should capture whether the Apache Flink community is
> > interested
> > > > in accepting, maintaining, and evolving Flink CDC.
> > > >
> > > > Regarding my original proposal[1] in the dev mailing list, I firmly
> > believe
> > > > that this initiative aligns perfectly with Flink. For the Flink
> > community,
> > > > it represents an opportunity to bolster Flink's competitive edge in
> > > > streaming
> > > > data integration, fostering the robust growth and prosperity of the
> > Apache
> > > > Flink
> > > > ecosystem. For the Flink CDC project, becoming a sub-project of
> Apache
> > > > Flink
> > > > means becoming an integral part of a neutral open-source community,
> > > > capable of
> > > > attracting a more diverse pool of contributors.
> > > >
> > > > All Flink CDC maintainers are dedicated to continuously contributing
> to
> > > > achieve
> > > > seamless integration with Flink. Additionally, PMC members like Jark,
> > > > Qingsheng,
> > > > and I are willing to infacilitate the expansion of contributors and
> > > > committers to
> > > > effectively maintain this new sub-project.
> > > >
> > > > This is a "Adoption of a new Codebase" vote as per the Flink bylaws
> > [2].
> > > > Only PMC votes are binding. The vote will be open at least 7 days
> > > > (excluding weekends), meaning until Thursday January 18 12:00 UTC, or
> > > > until we
> > > > achieve the 2/3rd majority. We will follow the instructions in the
> > Flink
> > > > Bylaws
> > > > in the case of insufficient active binding voters:
> > > >
> > > > > 1. Wait until the minimum length of the voting passes.
> > > > > 2. Publicly reach out via personal email to the remaining binding
> > voters
> > > > in the
> > > > voting mail thread for at least 2 attempts with at least 7 days
> between
> > > > two attempts.
> > > > > 3. If the binding voter being contacted still failed to respond
> after
> > > > all the attempts,
> > > > the binding voter will be considered as inactive for the purpose of
> > this
> > > > particular voting.
> > > >
> > > > Welcome voting !
> > > >
> > > > Best,
> > > > Leonard
> > > > [1] https://lists.apache.org/thread/o7klnbsotmmql999bnwmdgo56b6kxx9l
> > > > [2]
> > > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120731026
> >
> >
> >
> > --
> >
> > Best,
> > Benchao Li
> >
>

Re: [DISCUSS] Release Flink 1.18.1

[Jing Ge]

Hi everyone,

Because of some new incoming blockers and other tasks, the rc1 has been
created but the vote has been skipped. I just started the vote for
1.18.1-rc2 at [1]. It would be great if you might help check. Thanks!

Best regards,
Jing

[1] https://lists.apache.org/thread/zkgh6c9vhk9z3072vqlxxtbvrghllcp8

On Mon, Dec 18, 2023 at 5:34 PM Jing Ge  wrote:

> Hi folks,
>
> Found another blocker issue for 1.18:
> https://github.com/apache/flink/pull/23950. Thanks Sergey for the
> heads-up.
>
> Best regards,
> Jing
>
> On Mon, Dec 18, 2023 at 5:02 PM Jing Ge  wrote:
>
>> Hi,
>>
>> I am waiting for the CI of
>> https://issues.apache.org/jira/browse/FLINK-33872 @Hong. It would be
>> great if anyone would like to review the PR. Thanks.
>>
>> 1.18.1-rc1 release will be canceled because of
>> https://issues.apache.org/jira/browse/FLINK-33704, I will start with rc2
>> once FLINK-33872 has been merged into 1.18.
>>
>> Best regards,
>> Jing
>>
>> On Mon, Dec 18, 2023 at 11:52 AM Jing Ge  wrote:
>>
>>> Hi folks,
>>>
>>> Thanks Martijn for driving the
>>> https://issues.apache.org/jira/browse/FLINK-33704, which will solve the
>>> issue of https://issues.apache.org/jira/browse/FLINK-33793. I will move
>>> forward with the 1.18.1 release today.
>>>
>>> Best regards,
>>> Jing
>>>
>>> On Thu, Dec 14, 2023 at 11:17 AM Jing Ge  wrote:
>>>
>>>> Hi folks,
>>>>
>>>> What Martijn said makes sense. We should pay more attention to
>>>> https://issues.apache.org/jira/browse/FLINK-33793. I have also tried
>>>> to contribute and share my thoughts in the PR
>>>> https://github.com/apache/flink/pull/23489.
>>>>
>>>> For 1.18.1-rc1, I will wait until tomorrow. If there is no progress
>>>> with the PR, I will move forward with the release, unless we upgrade the
>>>> ticket to be a Blocker. Look forward to your feedback.
>>>>
>>>> Best regards,
>>>> Jing
>>>>
>>>> On Wed, Dec 13, 2023 at 9:57 AM Jing Ge  wrote:
>>>>
>>>>> Hi Martijn,
>>>>>
>>>>> Thanks for the heads-up! I already started 1.18.1-rc1[1]. But please
>>>>> feel free to ping me. I will either redo rc1 or start rc2. Thanks!
>>>>>
>>>>> Best regards,
>>>>> Jing
>>>>>
>>>>> [1] https://dist.apache.org/repos/dist/dev/flink/flink-1.18.1-rc1/
>>>>>
>>>>> On Tue, Dec 12, 2023 at 9:49 PM Martijn Visser <
>>>>> martijnvis...@apache.org> wrote:
>>>>>
>>>>>> Hi Jing,
>>>>>>
>>>>>> The only thing that was brought to my attention today was
>>>>>> https://issues.apache.org/jira/browse/FLINK-33793. I've marked it as
>>>>>> a
>>>>>> Critical (I think we should have working checkpointing with GCS), but
>>>>>> could also be considered a Blocker. There is a PR open for it, but I
>>>>>> don't think it's the right fix so I'm testing
>>>>>>
>>>>>> https://github.com/apache/flink/commit/2ebe7f6e090690486c1954099a2b57283c578192
>>>>>> at this moment. If you haven't started the 1.18.1 release yet, perhaps
>>>>>> we could include it (hopefully we can merge it tomorrow), else it
>>>>>> would have to wait for the next release.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Martijn
>>>>>>
>>>>>> On Tue, Dec 12, 2023 at 9:30 AM Jing Ge 
>>>>>> wrote:
>>>>>> >
>>>>>> > Hi All,
>>>>>> >
>>>>>> > Thank you for your feedback!
>>>>>> >
>>>>>> > Since FLINK-33523[1] is done(thanks Martijn for driving it) and
>>>>>> there are
>>>>>> > no other concerns or objections. Currently I am not aware of any
>>>>>> unresolved
>>>>>> > blockers. There is one critical task[2] whose PR still has some
>>>>>> checkstyle
>>>>>> > issues. I will try to include it into 1.18.1 with best effort,
>>>>>> since it
>>>>>> > depends on how quickly the author can finish it and cp to 1.18. If
>>>>>> anyone
>>>>>> > 

[VOTE] Release 1.18.1, release candidate #2

[Jing Ge]

Hi everyone,

The release candidate #1 has been skipped. Please review and vote on the
release candidate #2 for the version 1.18.1,

as follows:

[ ] +1, Approve the release

[ ] -1, Do not approve the release (please provide specific comments)


The complete staging area is available for your review, which includes:
* JIRA release notes [1],

* the official Apache source release and binary convenience releases to be
deployed to dist.apache.org [2], which are signed with the key with
fingerprint 96AE0E32CBE6E0753CE6 [3],

* all artifacts to be deployed to the Maven Central Repository [4],

* source code tag "release-1.18.1-rc2" [5],

* website pull request listing the new release and adding announcement blog
post [6].


The vote will be open for at least 72 hours. It is adopted by majority
approval, with at least 3 PMC affirmative votes.


[1]
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353640

[2] https://dist.apache.org/repos/dist/dev/flink/flink-1.18.1-rc2/

[3] https://dist.apache.org/repos/dist/release/flink/KEYS

[4] https://repository.apache.org/content/repositories/orgapacheflink-1689

[5] https://github.com/apache/flink/releases/tag/release-1.18.1-rc2

[6] https://github.com/apache/flink-web/pull/706

Thanks,
Release Manager

[jira] [Created] (FLINK-33886) CLONE - Build and stage Java and Python artifacts

[Jing Ge (Jira)]

Jing Ge created FLINK-33886:
---

 Summary: CLONE - Build and stage Java and Python artifacts
 Key: FLINK-33886
 URL: https://issues.apache.org/jira/browse/FLINK-33886
 Project: Flink
  Issue Type: Sub-task
Reporter: Jing Ge
Assignee: Jing Ge


# Create a local release branch ((!) this step can not be skipped for minor 
releases):
{code:bash}
$ cd ./tools
tools/ $ OLD_VERSION=$CURRENT_SNAPSHOT_VERSION NEW_VERSION=$RELEASE_VERSION 
RELEASE_CANDIDATE=$RC_NUM releasing/create_release_branch.sh
{code}
 # Tag the release commit:
{code:bash}
$ git tag -s ${TAG} -m "${TAG}"
{code}
 # We now need to do several things:
 ## Create the source release archive
 ## Deploy jar artefacts to the [Apache Nexus 
Repository|https://repository.apache.org/], which is the staging area for 
deploying the jars to Maven Central
 ## Build PyFlink wheel packages
You might want to create a directory on your local machine for collecting the 
various source and binary releases before uploading them. Creating the binary 
releases is a lengthy process but you can do this on another machine (for 
example, in the "cloud"). When doing this, you can skip signing the release 
files on the remote machine, download them to your local machine and sign them 
there.
 # Build the source release:
{code:bash}
tools $ RELEASE_VERSION=$RELEASE_VERSION releasing/create_source_release.sh
{code}
 # Stage the maven artifacts:
{code:bash}
tools $ releasing/deploy_staging_jars.sh
{code}
Review all staged artifacts ([https://repository.apache.org/]). They should 
contain all relevant parts for each module, including pom.xml, jar, test jar, 
source, test source, javadoc, etc. Carefully review any new artifacts.
 # Close the staging repository on Apache Nexus. When prompted for a 
description, enter “Apache Flink, version X, release candidate Y”.
Then, you need to build the PyFlink wheel packages (since 1.11):
 # Set up an azure pipeline in your own Azure account. You can refer to [Azure 
Pipelines|https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines#AzurePipelines-Tutorial:SettingupAzurePipelinesforaforkoftheFlinkrepository]
 for more details on how to set up azure pipeline for a fork of the Flink 
repository. Note that a google cloud mirror in Europe is used for downloading 
maven artifacts, therefore it is recommended to set your [Azure organization 
region|https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/change-organization-location]
 to Europe to speed up the downloads.
 # Push the release candidate branch to your forked personal Flink repository, 
e.g.
{code:bash}
tools $ git push  
refs/heads/release-${RELEASE_VERSION}-rc${RC_NUM}:release-${RELEASE_VERSION}-rc${RC_NUM}
{code}
 # Trigger the Azure Pipelines manually to build the PyFlink wheel packages
 ## Go to your Azure Pipelines Flink project → Pipelines
 ## Click the "New pipeline" button on the top right
 ## Select "GitHub" → your GitHub Flink repository → "Existing Azure Pipelines 
YAML file"
 ## Select your branch → Set path to "/azure-pipelines.yaml" → click on 
"Continue" → click on "Variables"
 ## Then click "New Variable" button, fill the name with "MODE", and the value 
with "release". Click "OK" to set the variable and the "Save" button to save 
the variables, then back on the "Review your pipeline" screen click "Run" to 
trigger the build.
 ## You should now see a build where only the "CI build (release)" is running
 # Download the PyFlink wheel packages from the build result page after the 
jobs of "build_wheels mac" and "build_wheels linux" have finished.
 ## Download the PyFlink wheel packages
 ### Open the build result page of the pipeline
 ### Go to the {{Artifacts}} page (build_wheels linux -> 1 artifact)
 ### Click {{wheel_Darwin_build_wheels mac}} and {{wheel_Linux_build_wheels 
linux}} separately to download the zip files
 ## Unzip these two zip files
{code:bash}
$ cd /path/to/downloaded_wheel_packages
$ unzip wheel_Linux_build_wheels\ linux.zip
$ unzip wheel_Darwin_build_wheels\ mac.zip{code}
 ## Create directory {{./dist}} under the directory of {{{}flink-python{}}}:
{code:bash}
$ cd 
$ mkdir flink-python/dist{code}
 ## Move the unzipped wheel packages to the directory of 
{{{}flink-python/dist{}}}:
{code:java}
$ mv /path/to/wheel_Darwin_build_wheels\ mac/* flink-python/dist/
$ mv /path/to/wheel_Linux_build_wheels\ linux/* flink-python/dist/
$ cd tools{code}

Finally, we create the binary convenience release files:
{code:bash}
tools $ RELEASE_VERSION=$RELEASE_VERSION releasing/create_binary_release.sh
{code}
If you want to run this step in parallel on a remote machine you have to make 
the release commit available there (for example by pushing to a reposi

[jira] [Created] (FLINK-33889) CLONE - Vote on the release candidate

[Jing Ge (Jira)]

Jing Ge created FLINK-33889:
---

 Summary: CLONE - Vote on the release candidate
 Key: FLINK-33889
 URL: https://issues.apache.org/jira/browse/FLINK-33889
 Project: Flink
  Issue Type: Sub-task
Affects Versions: 1.18.0
Reporter: Jing Ge
Assignee: Jing Ge


Once you have built and individually reviewed the release candidate, please 
share it for the community-wide review. Please review foundation-wide [voting 
guidelines|http://www.apache.org/foundation/voting.html] for more information.

Start the review-and-vote thread on the dev@ mailing list. Here’s an email 
template; please adjust as you see fit.
{quote}From: Release Manager
To: dev@flink.apache.org
Subject: [VOTE] Release 1.2.3, release candidate #3

Hi everyone,
Please review and vote on the release candidate #3 for the version 1.2.3, as 
follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)

The complete staging area is available for your review, which includes:
 * JIRA release notes [1],
 * the official Apache source release and binary convenience releases to be 
deployed to dist.apache.org [2], which are signed with the key with fingerprint 
 [3],
 * all artifacts to be deployed to the Maven Central Repository [4],
 * source code tag "release-1.2.3-rc3" [5],
 * website pull request listing the new release and adding announcement blog 
post [6].

The vote will be open for at least 72 hours. It is adopted by majority 
approval, with at least 3 PMC affirmative votes.

Thanks,
Release Manager

[1] link
[2] link
[3] [https://dist.apache.org/repos/dist/release/flink/KEYS]
[4] link
[5] link
[6] link
{quote}
*If there are any issues found in the release candidate, reply on the vote 
thread to cancel the vote.* There’s no need to wait 72 hours. Proceed to the 
Fix Issues step below and address the problem. However, some issues don’t 
require cancellation. For example, if an issue is found in the website pull 
request, just correct it on the spot and the vote can continue as-is.

For cancelling a release, the release manager needs to send an email to the 
release candidate thread, stating that the release candidate is officially 
cancelled. Next, all artifacts created specifically for the RC in the previous 
steps need to be removed:
 * Delete the staging repository in Nexus
 * Remove the source / binary RC files from dist.apache.org
 * Delete the source code tag in git

*If there are no issues, reply on the vote thread to close the voting.* Then, 
tally the votes in a separate email. Here’s an email template; please adjust as 
you see fit.
{quote}From: Release Manager
To: dev@flink.apache.org
Subject: [RESULT] [VOTE] Release 1.2.3, release candidate #3

I'm happy to announce that we have unanimously approved this release.

There are XXX approving votes, XXX of which are binding:
 * approver 1
 * approver 2
 * approver 3
 * approver 4

There are no disapproving votes.

Thanks everyone!
{quote}
 

h3. Expectations
 * Community votes to release the proposed candidate, with at least three 
approving PMC votes

Any issues that are raised till the vote is over should be either resolved or 
moved into the next release (if applicable).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33888) CLONE - Propose a pull request for website updates

[Jing Ge (Jira)]

Jing Ge created FLINK-33888:
---

 Summary: CLONE - Propose a pull request for website updates
 Key: FLINK-33888
 URL: https://issues.apache.org/jira/browse/FLINK-33888
 Project: Flink
  Issue Type: Sub-task
Affects Versions: 1.18.0
Reporter: Jing Ge
Assignee: Jing Ge


The final step of building the candidate is to propose a website pull request 
containing the following changes:
 * update docs/data/flink.yml

 ** Add a new major version or update minor version as required
 * update docs/data/release_archive.yml
 * update version references in quickstarts ({{{}q/{}}} directory) as required 
(outdated?)
 * add a blog post announcing the release in {{docs/content/posts}}

(!) Don’t merge the PRs before finalizing the release.

 

h3. Expectations
 * Website pull request proposed to list the 
[release|http://flink.apache.org/downloads.html]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33887) CLONE - Stage source and binary releases on dist.apache.org

[Jing Ge (Jira)]

Jing Ge created FLINK-33887:
---

 Summary: CLONE - Stage source and binary releases on 
dist.apache.org
 Key: FLINK-33887
 URL: https://issues.apache.org/jira/browse/FLINK-33887
 Project: Flink
  Issue Type: Sub-task
Reporter: Jing Ge
Assignee: Jing Ge


Copy the source release to the dev repository of dist.apache.org:
# If you have not already, check out the Flink section of the dev repository on 
dist.apache.org via Subversion. In a fresh directory:
{code:bash}
$ svn checkout https://dist.apache.org/repos/dist/dev/flink --depth=immediates
{code}
# Make a directory for the new release and copy all the artifacts (Flink 
source/binary distributions, hashes, GPG signatures and the python 
subdirectory) into that newly created directory:
{code:bash}
$ mkdir flink/flink-${RELEASE_VERSION}-rc${RC_NUM}
$ mv /tools/releasing/release/* 
flink/flink-${RELEASE_VERSION}-rc${RC_NUM}
{code}
# Add and commit all the files.
{code:bash}
$ cd flink
flink $ svn add flink-${RELEASE_VERSION}-rc${RC_NUM}
flink $ svn commit -m "Add flink-${RELEASE_VERSION}-rc${RC_NUM}"
{code}
# Verify that files are present under 
[https://dist.apache.org/repos/dist/dev/flink|https://dist.apache.org/repos/dist/dev/flink].
# Push the release tag if not done already (the following command assumes to be 
called from within the apache/flink checkout):
{code:bash}
$ git push  refs/tags/release-${RELEASE_VERSION}-rc${RC_NUM}
{code}

 

h3. Expectations
 * Maven artifacts deployed to the staging repository of 
[repository.apache.org|https://repository.apache.org/content/repositories/]
 * Source distribution deployed to the dev repository of 
[dist.apache.org|https://dist.apache.org/repos/dist/dev/flink/]
 * Check hashes (e.g. shasum -c *.sha512)
 * Check signatures (e.g. {{{}gpg --verify 
flink-1.2.3-source-release.tar.gz.asc flink-1.2.3-source-release.tar.gz{}}})
 * {{grep}} for legal headers in each file.
 * If time allows check the NOTICE files of the modules whose dependencies have 
been changed in this release in advance, since the license issues from time to 
time pop up during voting. See [Verifying a Flink 
Release|https://cwiki.apache.org/confluence/display/FLINK/Verifying+a+Flink+Release]
 "Checking License" section.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33885) Build Release Candidate: 1.18.1-rc2

[Jing Ge (Jira)]

Jing Ge created FLINK-33885:
---

 Summary: Build Release Candidate: 1.18.1-rc2
 Key: FLINK-33885
 URL: https://issues.apache.org/jira/browse/FLINK-33885
 Project: Flink
  Issue Type: New Feature
Affects Versions: 1.18.0
Reporter: Jing Ge
Assignee: Jing Ge


The core of the release process is the build-vote-fix cycle. Each cycle 
produces one release candidate. The Release Manager repeats this cycle until 
the community approves one release candidate, which is then finalized.

h4. Prerequisites
Set up a few environment variables to simplify Maven commands that follow. This 
identifies the release candidate being built. Start with {{RC_NUM}} equal to 1 
and increment it for each candidate:
{code}
RC_NUM="1"
TAG="release-${RELEASE_VERSION}-rc${RC_NUM}"
{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33882) UT/IT for checkpointing statistics

[Jing Ge (Jira)]

Jing Ge created FLINK-33882:
---

 Summary: UT/IT for checkpointing statistics
 Key: FLINK-33882
 URL: https://issues.apache.org/jira/browse/FLINK-33882
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Checkpointing
Reporter: Jing Ge


https://issues.apache.org/jira/browse/FLINK-33588

has been manually tested by [~zhutong66] as follows:
1. I will package the modified code, and the code modification will be done in 
the jar package of flink-dist-xxx.jar. Replace the jar package with the 
production Flink client.
2. Submit the Flink SQL task in the production environment to Yarn in 
application mode and check the Yarn logs
3. Check for any further errors in the Yarn log.
4. On the web interface of Flink web, check if the data displayed on the 
checkpoint information statistics page is normal.

It would be great to write UT or IT for this change



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Release Flink 1.18.1

[Jing Ge]

Hi folks,

Found another blocker issue for 1.18:
https://github.com/apache/flink/pull/23950. Thanks Sergey for the heads-up.

Best regards,
Jing

On Mon, Dec 18, 2023 at 5:02 PM Jing Ge  wrote:

> Hi,
>
> I am waiting for the CI of
> https://issues.apache.org/jira/browse/FLINK-33872 @Hong. It would be
> great if anyone would like to review the PR. Thanks.
>
> 1.18.1-rc1 release will be canceled because of
> https://issues.apache.org/jira/browse/FLINK-33704, I will start with rc2
> once FLINK-33872 has been merged into 1.18.
>
> Best regards,
> Jing
>
> On Mon, Dec 18, 2023 at 11:52 AM Jing Ge  wrote:
>
>> Hi folks,
>>
>> Thanks Martijn for driving the
>> https://issues.apache.org/jira/browse/FLINK-33704, which will solve the
>> issue of https://issues.apache.org/jira/browse/FLINK-33793. I will move
>> forward with the 1.18.1 release today.
>>
>> Best regards,
>> Jing
>>
>> On Thu, Dec 14, 2023 at 11:17 AM Jing Ge  wrote:
>>
>>> Hi folks,
>>>
>>> What Martijn said makes sense. We should pay more attention to
>>> https://issues.apache.org/jira/browse/FLINK-33793. I have also tried to
>>> contribute and share my thoughts in the PR
>>> https://github.com/apache/flink/pull/23489.
>>>
>>> For 1.18.1-rc1, I will wait until tomorrow. If there is no progress with
>>> the PR, I will move forward with the release, unless we upgrade the ticket
>>> to be a Blocker. Look forward to your feedback.
>>>
>>> Best regards,
>>> Jing
>>>
>>> On Wed, Dec 13, 2023 at 9:57 AM Jing Ge  wrote:
>>>
>>>> Hi Martijn,
>>>>
>>>> Thanks for the heads-up! I already started 1.18.1-rc1[1]. But please
>>>> feel free to ping me. I will either redo rc1 or start rc2. Thanks!
>>>>
>>>> Best regards,
>>>> Jing
>>>>
>>>> [1] https://dist.apache.org/repos/dist/dev/flink/flink-1.18.1-rc1/
>>>>
>>>> On Tue, Dec 12, 2023 at 9:49 PM Martijn Visser <
>>>> martijnvis...@apache.org> wrote:
>>>>
>>>>> Hi Jing,
>>>>>
>>>>> The only thing that was brought to my attention today was
>>>>> https://issues.apache.org/jira/browse/FLINK-33793. I've marked it as a
>>>>> Critical (I think we should have working checkpointing with GCS), but
>>>>> could also be considered a Blocker. There is a PR open for it, but I
>>>>> don't think it's the right fix so I'm testing
>>>>>
>>>>> https://github.com/apache/flink/commit/2ebe7f6e090690486c1954099a2b57283c578192
>>>>> at this moment. If you haven't started the 1.18.1 release yet, perhaps
>>>>> we could include it (hopefully we can merge it tomorrow), else it
>>>>> would have to wait for the next release.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Martijn
>>>>>
>>>>> On Tue, Dec 12, 2023 at 9:30 AM Jing Ge 
>>>>> wrote:
>>>>> >
>>>>> > Hi All,
>>>>> >
>>>>> > Thank you for your feedback!
>>>>> >
>>>>> > Since FLINK-33523[1] is done(thanks Martijn for driving it) and
>>>>> there are
>>>>> > no other concerns or objections. Currently I am not aware of any
>>>>> unresolved
>>>>> > blockers. There is one critical task[2] whose PR still has some
>>>>> checkstyle
>>>>> > issues. I will try to include it into 1.18.1 with best effort, since
>>>>> it
>>>>> > depends on how quickly the author can finish it and cp to 1.18. If
>>>>> anyone
>>>>> > considers FLINK-33588 as a must-have fix in 1.18.1, please let me
>>>>> know,
>>>>> > thanks!
>>>>> >
>>>>> > I will start working on the RC1 release today.
>>>>> >
>>>>> > @benchao: gotcha wrt FLINK-33313
>>>>> >
>>>>> > Best regards,
>>>>> > Jing
>>>>> >
>>>>> > [1] https://issues.apache.org/jira/browse/FLINK-33523
>>>>> > [2] https://issues.apache.org/jira/browse/FLINK-33588
>>>>> >
>>>>> > On Tue, Dec 12, 2023 at 7:03 AM weijie guo <
>>>>> guoweijieres...@gmail.com>
>>>>> > wrote:
>>>>> >
>>>>> >

Re: [DISCUSS] Release Flink 1.18.1

[Jing Ge]

Hi,

I am waiting for the CI of https://issues.apache.org/jira/browse/FLINK-33872
@Hong. It would be great if anyone would like to review the PR. Thanks.

1.18.1-rc1 release will be canceled because of
https://issues.apache.org/jira/browse/FLINK-33704, I will start with rc2
once FLINK-33872 has been merged into 1.18.

Best regards,
Jing

On Mon, Dec 18, 2023 at 11:52 AM Jing Ge  wrote:

> Hi folks,
>
> Thanks Martijn for driving the
> https://issues.apache.org/jira/browse/FLINK-33704, which will solve the
> issue of https://issues.apache.org/jira/browse/FLINK-33793. I will move
> forward with the 1.18.1 release today.
>
> Best regards,
> Jing
>
> On Thu, Dec 14, 2023 at 11:17 AM Jing Ge  wrote:
>
>> Hi folks,
>>
>> What Martijn said makes sense. We should pay more attention to
>> https://issues.apache.org/jira/browse/FLINK-33793. I have also tried to
>> contribute and share my thoughts in the PR
>> https://github.com/apache/flink/pull/23489.
>>
>> For 1.18.1-rc1, I will wait until tomorrow. If there is no progress with
>> the PR, I will move forward with the release, unless we upgrade the ticket
>> to be a Blocker. Look forward to your feedback.
>>
>> Best regards,
>> Jing
>>
>> On Wed, Dec 13, 2023 at 9:57 AM Jing Ge  wrote:
>>
>>> Hi Martijn,
>>>
>>> Thanks for the heads-up! I already started 1.18.1-rc1[1]. But please
>>> feel free to ping me. I will either redo rc1 or start rc2. Thanks!
>>>
>>> Best regards,
>>> Jing
>>>
>>> [1] https://dist.apache.org/repos/dist/dev/flink/flink-1.18.1-rc1/
>>>
>>> On Tue, Dec 12, 2023 at 9:49 PM Martijn Visser 
>>> wrote:
>>>
>>>> Hi Jing,
>>>>
>>>> The only thing that was brought to my attention today was
>>>> https://issues.apache.org/jira/browse/FLINK-33793. I've marked it as a
>>>> Critical (I think we should have working checkpointing with GCS), but
>>>> could also be considered a Blocker. There is a PR open for it, but I
>>>> don't think it's the right fix so I'm testing
>>>>
>>>> https://github.com/apache/flink/commit/2ebe7f6e090690486c1954099a2b57283c578192
>>>> at this moment. If you haven't started the 1.18.1 release yet, perhaps
>>>> we could include it (hopefully we can merge it tomorrow), else it
>>>> would have to wait for the next release.
>>>>
>>>> Thanks,
>>>>
>>>> Martijn
>>>>
>>>> On Tue, Dec 12, 2023 at 9:30 AM Jing Ge 
>>>> wrote:
>>>> >
>>>> > Hi All,
>>>> >
>>>> > Thank you for your feedback!
>>>> >
>>>> > Since FLINK-33523[1] is done(thanks Martijn for driving it) and there
>>>> are
>>>> > no other concerns or objections. Currently I am not aware of any
>>>> unresolved
>>>> > blockers. There is one critical task[2] whose PR still has some
>>>> checkstyle
>>>> > issues. I will try to include it into 1.18.1 with best effort, since
>>>> it
>>>> > depends on how quickly the author can finish it and cp to 1.18. If
>>>> anyone
>>>> > considers FLINK-33588 as a must-have fix in 1.18.1, please let me
>>>> know,
>>>> > thanks!
>>>> >
>>>> > I will start working on the RC1 release today.
>>>> >
>>>> > @benchao: gotcha wrt FLINK-33313
>>>> >
>>>> > Best regards,
>>>> > Jing
>>>> >
>>>> > [1] https://issues.apache.org/jira/browse/FLINK-33523
>>>> > [2] https://issues.apache.org/jira/browse/FLINK-33588
>>>> >
>>>> > On Tue, Dec 12, 2023 at 7:03 AM weijie guo >>> >
>>>> > wrote:
>>>> >
>>>> > > Thanks Jing for driving this bug-fix release.
>>>> > >
>>>> > > +1 from my side.
>>>> > >
>>>> > > Best regards,
>>>> > >
>>>> > > Weijie
>>>> > >
>>>> > >
>>>> > > Jark Wu  于2023年12月12日周二 12:17写道：
>>>> > >
>>>> > > > Thanks Jing for driving 1.18.1.
>>>> > > > +1 for this.
>>>> > > >
>>>> > > > Best,
>>>> > > > Jark
>>>> > > >
>>>> > > > On Mon, 11 Dec 2023 at 16:59, Hong Liang 
>>

Re: [DISCUSS] Release Flink 1.18.1

[Jing Ge]

Hi folks,

Thanks Martijn for driving the
https://issues.apache.org/jira/browse/FLINK-33704, which will solve the
issue of https://issues.apache.org/jira/browse/FLINK-33793. I will move
forward with the 1.18.1 release today.

Best regards,
Jing

On Thu, Dec 14, 2023 at 11:17 AM Jing Ge  wrote:

> Hi folks,
>
> What Martijn said makes sense. We should pay more attention to
> https://issues.apache.org/jira/browse/FLINK-33793. I have also tried to
> contribute and share my thoughts in the PR
> https://github.com/apache/flink/pull/23489.
>
> For 1.18.1-rc1, I will wait until tomorrow. If there is no progress with
> the PR, I will move forward with the release, unless we upgrade the ticket
> to be a Blocker. Look forward to your feedback.
>
> Best regards,
> Jing
>
> On Wed, Dec 13, 2023 at 9:57 AM Jing Ge  wrote:
>
>> Hi Martijn,
>>
>> Thanks for the heads-up! I already started 1.18.1-rc1[1]. But please feel
>> free to ping me. I will either redo rc1 or start rc2. Thanks!
>>
>> Best regards,
>> Jing
>>
>> [1] https://dist.apache.org/repos/dist/dev/flink/flink-1.18.1-rc1/
>>
>> On Tue, Dec 12, 2023 at 9:49 PM Martijn Visser 
>> wrote:
>>
>>> Hi Jing,
>>>
>>> The only thing that was brought to my attention today was
>>> https://issues.apache.org/jira/browse/FLINK-33793. I've marked it as a
>>> Critical (I think we should have working checkpointing with GCS), but
>>> could also be considered a Blocker. There is a PR open for it, but I
>>> don't think it's the right fix so I'm testing
>>>
>>> https://github.com/apache/flink/commit/2ebe7f6e090690486c1954099a2b57283c578192
>>> at this moment. If you haven't started the 1.18.1 release yet, perhaps
>>> we could include it (hopefully we can merge it tomorrow), else it
>>> would have to wait for the next release.
>>>
>>> Thanks,
>>>
>>> Martijn
>>>
>>> On Tue, Dec 12, 2023 at 9:30 AM Jing Ge 
>>> wrote:
>>> >
>>> > Hi All,
>>> >
>>> > Thank you for your feedback!
>>> >
>>> > Since FLINK-33523[1] is done(thanks Martijn for driving it) and there
>>> are
>>> > no other concerns or objections. Currently I am not aware of any
>>> unresolved
>>> > blockers. There is one critical task[2] whose PR still has some
>>> checkstyle
>>> > issues. I will try to include it into 1.18.1 with best effort, since it
>>> > depends on how quickly the author can finish it and cp to 1.18. If
>>> anyone
>>> > considers FLINK-33588 as a must-have fix in 1.18.1, please let me know,
>>> > thanks!
>>> >
>>> > I will start working on the RC1 release today.
>>> >
>>> > @benchao: gotcha wrt FLINK-33313
>>> >
>>> > Best regards,
>>> > Jing
>>> >
>>> > [1] https://issues.apache.org/jira/browse/FLINK-33523
>>> > [2] https://issues.apache.org/jira/browse/FLINK-33588
>>> >
>>> > On Tue, Dec 12, 2023 at 7:03 AM weijie guo 
>>> > wrote:
>>> >
>>> > > Thanks Jing for driving this bug-fix release.
>>> > >
>>> > > +1 from my side.
>>> > >
>>> > > Best regards,
>>> > >
>>> > > Weijie
>>> > >
>>> > >
>>> > > Jark Wu  于2023年12月12日周二 12:17写道：
>>> > >
>>> > > > Thanks Jing for driving 1.18.1.
>>> > > > +1 for this.
>>> > > >
>>> > > > Best,
>>> > > > Jark
>>> > > >
>>> > > > On Mon, 11 Dec 2023 at 16:59, Hong Liang 
>>> wrote:
>>> > > >
>>> > > > > +1. Thanks Jing for driving this.
>>> > > > >
>>> > > > > Hong
>>> > > > >
>>> > > > > On Mon, Dec 11, 2023 at 2:27 AM Yun Tang 
>>> wrote:
>>> > > > >
>>> > > > > > Thanks Jing for driving 1.18.1 release, +1 for this.
>>> > > > > >
>>> > > > > >
>>> > > > > > Best
>>> > > > > > Yun Tang
>>> > > > > > 
>>> > > > > > From: Rui Fan <1996fan...@gmail.com>
>>> > > > > > Sent: Saturday, December 9, 2023 21:46
>>> > > > > > To: dev@flink.apach

[jira] [Created] (FLINK-33851) CLONE - Start End of Life discussion thread for now outdated Flink minor version

[Jing Ge (Jira)]

Jing Ge created FLINK-33851:
---

 Summary: CLONE - Start End of Life discussion thread for now 
outdated Flink minor version
 Key: FLINK-33851
 URL: https://issues.apache.org/jira/browse/FLINK-33851
 Project: Flink
  Issue Type: Sub-task
Reporter: Jing Ge


The idea is to discuss whether we should do a final release for the now not 
supported minor version in the community. Such a minor release shouldn't be 
covered by the current minor version release managers. Their only 
responsibility is to trigger the discussion.

The intention of a final patch release for the now unsupported Flink minor 
version is to flush out all the fixes that didn't end up in the previous 
release.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33848) CLONE - Other announcements

[Jing Ge (Jira)]

Jing Ge created FLINK-33848:
---

 Summary: CLONE - Other announcements
 Key: FLINK-33848
 URL: https://issues.apache.org/jira/browse/FLINK-33848
 Project: Flink
  Issue Type: Sub-task
Reporter: Jing Ge


h3. Recordkeeping

Use [reporter.apache.org|https://reporter.apache.org/addrelease.html?flink] to 
seed the information about the release into future project reports.

(Note: Only PMC members have access report releases. If you do not have access, 
ask on the mailing list for assistance.)
h3. Flink blog

Major or otherwise important releases should have a blog post. Write one if 
needed for this particular release. Minor releases that don’t introduce new 
major functionality don’t necessarily need to be blogged (see [flink-web PR 
#581 for Flink 1.15.3|https://github.com/apache/flink-web/pull/581] as an 
example for a minor release blog post).

Please make sure that the release notes of the documentation (see section 
"Review and update documentation") are linked from the blog post of a major 
release.
We usually include the names of all contributors in the announcement blog post. 
Use the following command to get the list of contributors:
{code}
# first line is required to make sort first with uppercase and then lower
export LC_ALL=C
export FLINK_PREVIOUS_RELEASE_BRANCH=
export FLINK_CURRENT_RELEASE_BRANCH=
# e.g.
# export FLINK_PREVIOUS_RELEASE_BRANCH=release-1.17
# export FLINK_CURRENT_RELEASE_BRANCH=release-1.18
git log $(git merge-base master $FLINK_PREVIOUS_RELEASE_BRANCH)..$(git show-ref 
--hash ${FLINK_CURRENT_RELEASE_BRANCH}) --pretty=format:"%an%n%cn" | sort  -u | 
paste -sd, | sed "s/\,/\, /g"
{code}
h3. Social media

Tweet, post on Facebook, LinkedIn, and other platforms. Ask other contributors 
to do the same.
h3. Flink Release Wiki page

Add a summary of things that went well or that went not so well during the 
release process. This can include feedback from contributors but also more 
generic things like the release have taken longer than initially anticipated 
(and why) to give a bit of context to the release process.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33845) CLONE - Merge website pull request

[Jing Ge (Jira)]

Jing Ge created FLINK-33845:
---

 Summary: CLONE - Merge website pull request
 Key: FLINK-33845
 URL: https://issues.apache.org/jira/browse/FLINK-33845
 Project: Flink
  Issue Type: Sub-task
Reporter: Jing Ge


Merge the website pull request to [list the 
release|http://flink.apache.org/downloads.html]. Make sure to regenerate the 
website as well, as it isn't build automatically.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33844) CLONE - Update japicmp configuration

[Jing Ge (Jira)]

Jing Ge created FLINK-33844:
---

 Summary: CLONE - Update japicmp configuration
 Key: FLINK-33844
 URL: https://issues.apache.org/jira/browse/FLINK-33844
 Project: Flink
  Issue Type: Sub-task
Reporter: Jing Ge
Assignee: Sergey Nuyanzin
 Fix For: 1.19.0, 1.18.1


Update the japicmp reference version and wipe exclusions / enable API 
compatibility checks for {{@PublicEvolving}} APIs on the corresponding SNAPSHOT 
branch with the {{update_japicmp_configuration.sh}} script (see below).

For a new major release (x.y.0), run the same command also on the master branch 
for updating the japicmp reference version and removing out-dated exclusions in 
the japicmp configuration.

Make sure that all Maven artifacts are already pushed to Maven Central. 
Otherwise, there's a risk that CI fails due to missing reference artifacts.
{code:bash}
tools $ NEW_VERSION=$RELEASE_VERSION releasing/update_japicmp_configuration.sh
tools $ cd ..$ git add *$ git commit -m "Update japicmp configuration for 
$RELEASE_VERSION" {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33847) CLONE - Apache mailing lists announcements

[Jing Ge (Jira)]

Jing Ge created FLINK-33847:
---

 Summary: CLONE - Apache mailing lists announcements
 Key: FLINK-33847
 URL: https://issues.apache.org/jira/browse/FLINK-33847
 Project: Flink
  Issue Type: Sub-task
Reporter: Jing Ge


Announce on the {{dev@}} mailing list that the release has been finished.

Announce on the release on the {{user@}} mailing list, listing major 
improvements and contributions.

Announce the release on the [annou...@apache.org|mailto:annou...@apache.org] 
mailing list.
{panel}
{panel}
|{{From: Release Manager}}
{{To: dev@flink.apache.org, u...@flink.apache.org, user...@flink.apache.org, 
annou...@apache.org}}
{{Subject: [ANNOUNCE] Apache Flink 1.2.3 released}}
 
{{The Apache Flink community is very happy to announce the release of Apache 
Flink 1.2.3, which is the third bugfix release for the Apache Flink 1.2 
series.}}
 
{{Apache Flink® is an open-source stream processing framework for distributed, 
high-performing, always-available, and accurate data streaming applications.}}
 
{{The release is available for download at:}}
{{[https://flink.apache.org/downloads.html]}}
 
{{Please check out the release blog post for an overview of the improvements 
for this bugfix release:}}
{{}}
 
{{The full release notes are available in Jira:}}
{{}}
 
{{We would like to thank all contributors of the Apache Flink community who 
made this release possible!}}
 
{{Feel free to reach out to the release managers (or respond to this thread) 
with feedback on the release process. Our goal is to constantly improve the 
release process. Feedback on what could be improved or things that didn't go so 
well are appreciated.}}
 
{{Regards,}}
{{Release Manager}}|



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33850) CLONE - Updates the docs stable version

[Jing Ge (Jira)]

Jing Ge created FLINK-33850:
---

 Summary: CLONE - Updates the docs stable version
 Key: FLINK-33850
 URL: https://issues.apache.org/jira/browse/FLINK-33850
 Project: Flink
  Issue Type: Sub-task
Reporter: Jing Ge


Update docs to "stable" in {{docs/config.toml}} in the branch of the 
_just-released_ version:
 * Change V{{{}ersion{}}} from {{{}x.y-SNAPSHOT }}to \{{{}x.y.z{}}}, i.e. 
{{1.6-SNAPSHOT}} to {{1.6.0}}
 * Change V{{{}ersionTitle{}}} from {{x.y-SNAPSHOT}} to {{{}x.y{}}}, i.e. 
{{1.6-SNAPSHOT}} to {{1.6}}
 * Change Branch from {{master}} to {{{}release-x.y{}}}, i.e. {{master}} to 
{{release-1.6}}
 * Change {{baseURL}} from 
{{//[ci.apache.org/projects/flink/flink-docs-master|http://ci.apache.org/projects/flink/flink-docs-master]}}
 to 
{{//[ci.apache.org/projects/flink/flink-docs-release-x.y|http://ci.apache.org/projects/flink/flink-docs-release-x.y]}}
 * Change {{javadocs_baseurl}} from 
{{//[ci.apache.org/projects/flink/flink-docs-master|http://ci.apache.org/projects/flink/flink-docs-master]}}
 to 
{{//[ci.apache.org/projects/flink/flink-docs-release-x.y|http://ci.apache.org/projects/flink/flink-docs-release-x.y]}}
 * Change {{IsStable}} to {{true}}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33849) CLONE - Update reference data for Migration Tests

[Jing Ge (Jira)]

Jing Ge created FLINK-33849:
---

 Summary: CLONE - Update reference data for Migration Tests
 Key: FLINK-33849
 URL: https://issues.apache.org/jira/browse/FLINK-33849
 Project: Flink
  Issue Type: Sub-task
Reporter: Jing Ge
Assignee: Sergey Nuyanzin
 Fix For: 1.19.0, 1.18.1


Update migration tests in master to cover migration from new version. Since 
1.18, this step could be done automatically with the following steps. For more 
information please refer to [this 
page.|https://github.com/apache/flink/blob/master/flink-test-utils-parent/flink-migration-test-utils/README.md]
 # {*}On the published release tag (e.g., release-1.16.0){*}, run 
{panel}
{panel}
|{{$ mvn clean }}{{package}} {{{}-Pgenerate-migration-test-data 
-Dgenerate.version={}}}{{{}1.16{}}} {{-nsu -Dfast -DskipTests}}|

The version (1.16 in the command above) should be replaced with the target one.

 # Modify the content of the file 
[apache/flink:flink-test-utils-parent/flink-migration-test-utils/src/main/resources/most_recently_published_version|https://github.com/apache/flink/blob/master/flink-test-utils-parent/flink-migration-test-utils/src/main/resources/most_recently_published_version]
 to the latest version (it would be "v1_16" if sticking to the example where 
1.16.0 was released). 
 # Commit the modification in step a and b with "{_}[release] Generate 
reference data for state migration tests based on release-1.xx.0{_}" to the 
corresponding release branch (e.g. {{release-1.16}} in our example), replace 
"xx" with the actual version (in this example "16"). You should use the Jira 
issue ID in case of [release]  as the commit message's prefix if you have a 
dedicated Jira issue for this task.

 # Cherry-pick the commit to the master branch. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33843) Promote release 1.18.1

[Jing Ge (Jira)]

Jing Ge created FLINK-33843:
---

 Summary: Promote release 1.18.1
 Key: FLINK-33843
 URL: https://issues.apache.org/jira/browse/FLINK-33843
 Project: Flink
  Issue Type: New Feature
Affects Versions: 1.18.0
Reporter: Jing Ge
Assignee: Jing Ge


Once the release has been finalized (FLINK-32920), the last step of the process 
is to promote the release within the project and beyond. Please wait for 24h 
after finalizing the release in accordance with the [ASF release 
policy|http://www.apache.org/legal/release-policy.html#release-announcements].

*Final checklist to declare this issue resolved:*
 # Website pull request to [list the 
release|http://flink.apache.org/downloads.html] merged
 # Release announced on the user@ mailing list.
 # Blog post published, if applicable.
 # Release recorded in 
[reporter.apache.org|https://reporter.apache.org/addrelease.html?flink].
 # Release announced on social media.
 # Completion declared on the dev@ mailing list.
 # Update Homebrew: [https://docs.brew.sh/How-To-Open-a-Homebrew-Pull-Request] 
(seems to be done automatically - at least for minor releases  for both minor 
and major releases)
 # Updated the japicmp configuration
 ** corresponding SNAPSHOT branch japicmp reference version set to the just 
released version, and API compatibiltity checks for {{@PublicEvolving}}  was 
enabled
 ** (minor version release only) master branch japicmp reference version set to 
the just released version
 ** (minor version release only) master branch japicmp exclusions have been 
cleared
 # Update the list of previous version in {{docs/config.toml}} on the master 
branch.
 # Set {{show_outdated_warning: true}} in {{docs/config.toml}} in the branch of 
the _now deprecated_ Flink version (i.e. 1.16 if 1.18.0 is released)
 # Update stable and master alias in 
[https://github.com/apache/flink/blob/master/.github/workflows/docs.yml]
 # Open discussion thread for End of Life for Unsupported version (i.e. 1.16)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33846) CLONE - Remove outdated versions

[Jing Ge (Jira)]

Jing Ge created FLINK-33846:
---

 Summary: CLONE - Remove outdated versions
 Key: FLINK-33846
 URL: https://issues.apache.org/jira/browse/FLINK-33846
 Project: Flink
  Issue Type: Sub-task
Reporter: Jing Ge


h4. dist.apache.org

For a new major release remove all release files older than 2 versions, e.g., 
when releasing 1.7, remove all releases <= 1.5.

For a new bugfix version remove all release files for previous bugfix releases 
in the same series, e.g., when releasing 1.7.1, remove the 1.7.0 release.
# If you have not already, check out the Flink section of the {{release}} 
repository on {{[dist.apache.org|http://dist.apache.org/]}} via Subversion. In 
a fresh directory:
{code}
svn checkout https://dist.apache.org/repos/dist/release/flink --depth=immediates
cd flink
{code}
# Remove files for outdated releases and commit the changes.
{code}
svn remove flink-
svn commit
{code}
# Verify that files  are 
[removed|https://dist.apache.org/repos/dist/release/flink]
(!) Remember to remove the corresponding download links from the website.

h4. CI

Disable the cron job for the now-unsupported version from 
(tools/azure-pipelines/[build-apache-repo.yml|https://github.com/apache/flink/blob/master/tools/azure-pipelines/build-apache-repo.yml])
 in the respective branch.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33840) CLONE - Deploy artifacts to Maven Central Repository

[Jing Ge (Jira)]

Jing Ge created FLINK-33840:
---

 Summary: CLONE - Deploy artifacts to Maven Central Repository
 Key: FLINK-33840
 URL: https://issues.apache.org/jira/browse/FLINK-33840
 Project: Flink
  Issue Type: Sub-task
Reporter: Jing Ge
Assignee: Qingsheng Ren


Use the [Apache Nexus repository|https://repository.apache.org/] to release the 
staged binary artifacts to the Maven Central repository. In the Staging 
Repositories section, find the relevant release candidate orgapacheflink-XXX 
entry and click Release. Drop all other release candidates that are not being 
released.
h3. Deploy source and binary releases to dist.apache.org

Copy the source and binary releases from the dev repository to the release 
repository at [dist.apache.org|http://dist.apache.org/] using Subversion.
{code:java}
$ svn move -m "Release Flink ${RELEASE_VERSION}" 
https://dist.apache.org/repos/dist/dev/flink/flink-${RELEASE_VERSION}-rc${RC_NUM}
 https://dist.apache.org/repos/dist/release/flink/flink-${RELEASE_VERSION}
{code}
(Note: Only PMC members have access to the release repository. If you do not 
have access, ask on the mailing list for assistance.)
h3. Remove old release candidates from [dist.apache.org|http://dist.apache.org/]

Remove the old release candidates from 
[https://dist.apache.org/repos/dist/dev/flink] using Subversion.
{code:java}
$ svn checkout https://dist.apache.org/repos/dist/dev/flink --depth=immediates
$ cd flink
$ svn remove flink-${RELEASE_VERSION}-rc*
$ svn commit -m "Remove old release candidates for Apache Flink 
${RELEASE_VERSION}
{code}
 

h3. Expectations
 * Maven artifacts released and indexed in the [Maven Central 
Repository|https://search.maven.org/#search%7Cga%7C1%7Cg%3A%22org.apache.flink%22]
 (usually takes about a day to show up)
 * Source & binary distributions available in the release repository of 
[https://dist.apache.org/repos/dist/release/flink/]
 * Dev repository [https://dist.apache.org/repos/dist/dev/flink/] is empty
 * Website contains links to new release binaries and sources in download page
 * (for minor version updates) the front page references the correct new major 
release version and directs to the correct link



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33841) CLONE - Create Git tag and mark version as released in Jira

[Jing Ge (Jira)]

Jing Ge created FLINK-33841:
---

 Summary: CLONE - Create Git tag and mark version as released in 
Jira
 Key: FLINK-33841
 URL: https://issues.apache.org/jira/browse/FLINK-33841
 Project: Flink
  Issue Type: Sub-task
Reporter: Jing Ge
Assignee: Qingsheng Ren


Create and push a new Git tag for the released version by copying the tag for 
the final release candidate, as follows:
{code:java}
$ git tag -s "release-${RELEASE_VERSION}" refs/tags/${TAG}^{} -m "Release Flink 
${RELEASE_VERSION}"
$ git push  refs/tags/release-${RELEASE_VERSION}
{code}
In JIRA, inside [version 
management|https://issues.apache.org/jira/plugins/servlet/project-config/FLINK/versions],
 hover over the current release and a settings menu will appear. Click Release, 
and select today’s date.

(Note: Only PMC members have access to the project administration. If you do 
not have access, ask on the mailing list for assistance.)

If PRs have been merged to the release branch after the the last release 
candidate was tagged, make sure that the corresponding Jira tickets have the 
correct Fix Version set.

 

h3. Expectations
 * Release tagged in the source code repository
 * Release version finalized in JIRA. (Note: Not all committers have 
administrator access to JIRA. If you end up getting permissions errors ask on 
the mailing list for assistance)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33839) CLONE - Deploy Python artifacts to PyPI

[Jing Ge (Jira)]

Jing Ge created FLINK-33839:
---

 Summary: CLONE - Deploy Python artifacts to PyPI
 Key: FLINK-33839
 URL: https://issues.apache.org/jira/browse/FLINK-33839
 Project: Flink
  Issue Type: Sub-task
Reporter: Jing Ge
Assignee: Qingsheng Ren


Release manager should create a PyPI account and ask the PMC add this account 
to pyflink collaborator list with Maintainer role (The PyPI admin account info 
can be found here. NOTE, only visible to PMC members) to deploy the Python 
artifacts to PyPI. The artifacts could be uploaded using 
twine([https://pypi.org/project/twine/]). To install twine, just run:
{code:java}
pip install --upgrade twine==1.12.0
{code}
Download the python artifacts from dist.apache.org and upload it to pypi.org:
{code:java}
svn checkout 
https://dist.apache.org/repos/dist/dev/flink/flink-${RELEASE_VERSION}-rc${RC_NUM}
cd flink-${RELEASE_VERSION}-rc${RC_NUM}
 
cd python
 
#uploads wheels
for f in *.whl; do twine upload --repository-url 
https://upload.pypi.org/legacy/ $f $f.asc; done
 
#upload source packages
twine upload --repository-url https://upload.pypi.org/legacy/ 
apache-flink-libraries-${RELEASE_VERSION}.tar.gz 
apache-flink-libraries-${RELEASE_VERSION}.tar.gz.asc
 
twine upload --repository-url https://upload.pypi.org/legacy/ 
apache-flink-${RELEASE_VERSION}.tar.gz 
apache-flink-${RELEASE_VERSION}.tar.gz.asc
{code}
If upload failed or incorrect for some reason (e.g. network transmission 
problem), you need to delete the uploaded release package of the same version 
(if exists) and rename the artifact to 
{{{}apache-flink-${RELEASE_VERSION}.post0.tar.gz{}}}, then re-upload.

(!) Note: re-uploading to pypi.org must be avoided as much as possible because 
it will cause some irreparable problems. If that happens, users cannot install 
the apache-flink package by explicitly specifying the package version, i.e. the 
following command "pip install apache-flink==${RELEASE_VERSION}" will fail. 
Instead they have to run "pip install apache-flink" or "pip install 
apache-flink==${RELEASE_VERSION}.post0" to install the apache-flink package.

 

h3. Expectations
 * Python artifacts released and indexed in the 
[PyPI|https://pypi.org/project/apache-flink/] Repository



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33838) Finalize release 1.18.1

[Jing Ge (Jira)]

Jing Ge created FLINK-33838:
---

 Summary: Finalize release 1.18.1
 Key: FLINK-33838
 URL: https://issues.apache.org/jira/browse/FLINK-33838
 Project: Flink
  Issue Type: New Feature
Affects Versions: 1.17.0
Reporter: Jing Ge
Assignee: Qingsheng Ren


Once the release candidate has been reviewed and approved by the community, the 
release should be finalized. This involves the final deployment of the release 
candidate to the release repositories, merging of the website changes, etc.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33842) CLONE - Publish the Dockerfiles for the new release

[Jing Ge (Jira)]

Jing Ge created FLINK-33842:
---

 Summary: CLONE - Publish the Dockerfiles for the new release
 Key: FLINK-33842
 URL: https://issues.apache.org/jira/browse/FLINK-33842
 Project: Flink
  Issue Type: Sub-task
Reporter: Jing Ge
Assignee: Matthias Pohl


Note: the official Dockerfiles fetch the binary distribution of the target 
Flink version from an Apache mirror. After publishing the binary release 
artifacts, mirrors can take some hours to start serving the new artifacts, so 
you may want to wait to do this step until you are ready to continue with the 
"Promote the release" steps in the follow-up Jira.

Follow the [release instructions in the flink-docker 
repo|https://github.com/apache/flink-docker#release-workflow] to build the new 
Dockerfiles and send an updated manifest to Docker Hub so the new images are 
built and published.

 

h3. Expectations
 * Dockerfiles in [flink-docker|https://github.com/apache/flink-docker] updated 
for the new Flink release and pull request opened on the Docker official-images 
with an updated manifest



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33837) CLONE - Vote on the release candidate

[Jing Ge (Jira)]

Jing Ge created FLINK-33837:
---

 Summary: CLONE - Vote on the release candidate
 Key: FLINK-33837
 URL: https://issues.apache.org/jira/browse/FLINK-33837
 Project: Flink
  Issue Type: Sub-task
Affects Versions: 1.17.0
Reporter: Jing Ge
Assignee: Qingsheng Ren
 Fix For: 1.17.0


Once you have built and individually reviewed the release candidate, please 
share it for the community-wide review. Please review foundation-wide [voting 
guidelines|http://www.apache.org/foundation/voting.html] for more information.

Start the review-and-vote thread on the dev@ mailing list. Here’s an email 
template; please adjust as you see fit.
{quote}From: Release Manager
To: dev@flink.apache.org
Subject: [VOTE] Release 1.2.3, release candidate #3

Hi everyone,
Please review and vote on the release candidate #3 for the version 1.2.3, as 
follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)

The complete staging area is available for your review, which includes:
 * JIRA release notes [1],
 * the official Apache source release and binary convenience releases to be 
deployed to dist.apache.org [2], which are signed with the key with fingerprint 
 [3],
 * all artifacts to be deployed to the Maven Central Repository [4],
 * source code tag "release-1.2.3-rc3" [5],
 * website pull request listing the new release and adding announcement blog 
post [6].

The vote will be open for at least 72 hours. It is adopted by majority 
approval, with at least 3 PMC affirmative votes.

Thanks,
Release Manager

[1] link
[2] link
[3] [https://dist.apache.org/repos/dist/release/flink/KEYS]
[4] link
[5] link
[6] link
{quote}
*If there are any issues found in the release candidate, reply on the vote 
thread to cancel the vote.* There’s no need to wait 72 hours. Proceed to the 
Fix Issues step below and address the problem. However, some issues don’t 
require cancellation. For example, if an issue is found in the website pull 
request, just correct it on the spot and the vote can continue as-is.

For cancelling a release, the release manager needs to send an email to the 
release candidate thread, stating that the release candidate is officially 
cancelled. Next, all artifacts created specifically for the RC in the previous 
steps need to be removed:
 * Delete the staging repository in Nexus
 * Remove the source / binary RC files from dist.apache.org
 * Delete the source code tag in git

*If there are no issues, reply on the vote thread to close the voting.* Then, 
tally the votes in a separate email. Here’s an email template; please adjust as 
you see fit.
{quote}From: Release Manager
To: dev@flink.apache.org
Subject: [RESULT] [VOTE] Release 1.2.3, release candidate #3

I'm happy to announce that we have unanimously approved this release.

There are XXX approving votes, XXX of which are binding:
 * approver 1
 * approver 2
 * approver 3
 * approver 4

There are no disapproving votes.

Thanks everyone!
{quote}
 

h3. Expectations
 * Community votes to release the proposed candidate, with at least three 
approving PMC votes

Any issues that are raised till the vote is over should be either resolved or 
moved into the next release (if applicable).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33836) CLONE - Propose a pull request for website updates

[Jing Ge (Jira)]

Jing Ge created FLINK-33836:
---

 Summary: CLONE - Propose a pull request for website updates
 Key: FLINK-33836
 URL: https://issues.apache.org/jira/browse/FLINK-33836
 Project: Flink
  Issue Type: Sub-task
Affects Versions: 1.17.0
Reporter: Jing Ge
Assignee: Qingsheng Ren
 Fix For: 1.17.0


The final step of building the candidate is to propose a website pull request 
containing the following changes:
 # update 
[apache/flink-web:_config.yml|https://github.com/apache/flink-web/blob/asf-site/_config.yml]
 ## update {{FLINK_VERSION_STABLE}} and {{FLINK_VERSION_STABLE_SHORT}} as 
required
 ## update version references in quickstarts ({{{}q/{}}} directory) as required
 ## (major only) add a new entry to {{flink_releases}} for the release binaries 
and sources
 ## (minor only) update the entry for the previous release in the series in 
{{flink_releases}}
 ### Please pay notice to the ids assigned to the download entries. They should 
be unique and reflect their corresponding version number.
 ## add a new entry to {{release_archive.flink}}
 # add a blog post announcing the release in _posts
 # add a organized release notes page under docs/content/release-notes and 
docs/content.zh/release-notes (like 
[https://nightlies.apache.org/flink/flink-docs-release-1.15/release-notes/flink-1.15/]).
 The page is based on the non-empty release notes collected from the issues, 
and only the issues that affect existing users should be included (e.g., 
instead of new functionality). It should be in a separate PR since it would be 
merged to the flink project.

(!) Don’t merge the PRs before finalizing the release.

 

h3. Expectations
 * Website pull request proposed to list the 
[release|http://flink.apache.org/downloads.html]
 * (major only) Check {{docs/config.toml}} to ensure that
 ** the version constants refer to the new version
 ** the {{baseurl}} does not point to {{flink-docs-master}}  but 
{{flink-docs-release-X.Y}} instead



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33834) CLONE - Build and stage Java and Python artifacts

[Jing Ge (Jira)]

Jing Ge created FLINK-33834:
---

 Summary: CLONE - Build and stage Java and Python artifacts
 Key: FLINK-33834
 URL: https://issues.apache.org/jira/browse/FLINK-33834
 Project: Flink
  Issue Type: Sub-task
Reporter: Jing Ge
Assignee: Qingsheng Ren


# Create a local release branch ((!) this step can not be skipped for minor 
releases):
{code:bash}
$ cd ./tools
tools/ $ OLD_VERSION=$CURRENT_SNAPSHOT_VERSION NEW_VERSION=$RELEASE_VERSION 
RELEASE_CANDIDATE=$RC_NUM releasing/create_release_branch.sh
{code}
 # Tag the release commit:
{code:bash}
$ git tag -s ${TAG} -m "${TAG}"
{code}
 # We now need to do several things:
 ## Create the source release archive
 ## Deploy jar artefacts to the [Apache Nexus 
Repository|https://repository.apache.org/], which is the staging area for 
deploying the jars to Maven Central
 ## Build PyFlink wheel packages
You might want to create a directory on your local machine for collecting the 
various source and binary releases before uploading them. Creating the binary 
releases is a lengthy process but you can do this on another machine (for 
example, in the "cloud"). When doing this, you can skip signing the release 
files on the remote machine, download them to your local machine and sign them 
there.
 # Build the source release:
{code:bash}
tools $ RELEASE_VERSION=$RELEASE_VERSION releasing/create_source_release.sh
{code}
 # Stage the maven artifacts:
{code:bash}
tools $ releasing/deploy_staging_jars.sh
{code}
Review all staged artifacts ([https://repository.apache.org/]). They should 
contain all relevant parts for each module, including pom.xml, jar, test jar, 
source, test source, javadoc, etc. Carefully review any new artifacts.
 # Close the staging repository on Apache Nexus. When prompted for a 
description, enter “Apache Flink, version X, release candidate Y”.
Then, you need to build the PyFlink wheel packages (since 1.11):
 # Set up an azure pipeline in your own Azure account. You can refer to [Azure 
Pipelines|https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines#AzurePipelines-Tutorial:SettingupAzurePipelinesforaforkoftheFlinkrepository]
 for more details on how to set up azure pipeline for a fork of the Flink 
repository. Note that a google cloud mirror in Europe is used for downloading 
maven artifacts, therefore it is recommended to set your [Azure organization 
region|https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/change-organization-location]
 to Europe to speed up the downloads.
 # Push the release candidate branch to your forked personal Flink repository, 
e.g.
{code:bash}
tools $ git push  
refs/heads/release-${RELEASE_VERSION}-rc${RC_NUM}:release-${RELEASE_VERSION}-rc${RC_NUM}
{code}
 # Trigger the Azure Pipelines manually to build the PyFlink wheel packages
 ## Go to your Azure Pipelines Flink project → Pipelines
 ## Click the "New pipeline" button on the top right
 ## Select "GitHub" → your GitHub Flink repository → "Existing Azure Pipelines 
YAML file"
 ## Select your branch → Set path to "/azure-pipelines.yaml" → click on 
"Continue" → click on "Variables"
 ## Then click "New Variable" button, fill the name with "MODE", and the value 
with "release". Click "OK" to set the variable and the "Save" button to save 
the variables, then back on the "Review your pipeline" screen click "Run" to 
trigger the build.
 ## You should now see a build where only the "CI build (release)" is running
 # Download the PyFlink wheel packages from the build result page after the 
jobs of "build_wheels mac" and "build_wheels linux" have finished.
 ## Download the PyFlink wheel packages
 ### Open the build result page of the pipeline
 ### Go to the {{Artifacts}} page (build_wheels linux -> 1 artifact)
 ### Click {{wheel_Darwin_build_wheels mac}} and {{wheel_Linux_build_wheels 
linux}} separately to download the zip files
 ## Unzip these two zip files
{code:bash}
$ cd /path/to/downloaded_wheel_packages
$ unzip wheel_Linux_build_wheels\ linux.zip
$ unzip wheel_Darwin_build_wheels\ mac.zip{code}
 ## Create directory {{./dist}} under the directory of {{{}flink-python{}}}:
{code:bash}
$ cd 
$ mkdir flink-python/dist{code}
 ## Move the unzipped wheel packages to the directory of 
{{{}flink-python/dist{}}}:
{code:java}
$ mv /path/to/wheel_Darwin_build_wheels\ mac/* flink-python/dist/
$ mv /path/to/wheel_Linux_build_wheels\ linux/* flink-python/dist/
$ cd tools{code}

Finally, we create the binary convenience release files:
{code:bash}
tools $ RELEASE_VERSION=$RELEASE_VERSION releasing/create_binary_release.sh
{code}
If you want to run this step in parallel on a remote machine you have to make 
the release commit available there (for example by pushing to

[jira] [Created] (FLINK-33835) CLONE - Stage source and binary releases on dist.apache.org

[Jing Ge (Jira)]

Jing Ge created FLINK-33835:
---

 Summary: CLONE - Stage source and binary releases on 
dist.apache.org
 Key: FLINK-33835
 URL: https://issues.apache.org/jira/browse/FLINK-33835
 Project: Flink
  Issue Type: Sub-task
Reporter: Jing Ge
Assignee: Qingsheng Ren


Copy the source release to the dev repository of dist.apache.org:
# If you have not already, check out the Flink section of the dev repository on 
dist.apache.org via Subversion. In a fresh directory:
{code:bash}
$ svn checkout https://dist.apache.org/repos/dist/dev/flink --depth=immediates
{code}
# Make a directory for the new release and copy all the artifacts (Flink 
source/binary distributions, hashes, GPG signatures and the python 
subdirectory) into that newly created directory:
{code:bash}
$ mkdir flink/flink-${RELEASE_VERSION}-rc${RC_NUM}
$ mv /tools/releasing/release/* 
flink/flink-${RELEASE_VERSION}-rc${RC_NUM}
{code}
# Add and commit all the files.
{code:bash}
$ cd flink
flink $ svn add flink-${RELEASE_VERSION}-rc${RC_NUM}
flink $ svn commit -m "Add flink-${RELEASE_VERSION}-rc${RC_NUM}"
{code}
# Verify that files are present under 
[https://dist.apache.org/repos/dist/dev/flink|https://dist.apache.org/repos/dist/dev/flink].
# Push the release tag if not done already (the following command assumes to be 
called from within the apache/flink checkout):
{code:bash}
$ git push  refs/tags/release-${RELEASE_VERSION}-rc${RC_NUM}
{code}

 

h3. Expectations
 * Maven artifacts deployed to the staging repository of 
[repository.apache.org|https://repository.apache.org/content/repositories/]
 * Source distribution deployed to the dev repository of 
[dist.apache.org|https://dist.apache.org/repos/dist/dev/flink/]
 * Check hashes (e.g. shasum -c *.sha512)
 * Check signatures (e.g. {{{}gpg --verify 
flink-1.2.3-source-release.tar.gz.asc flink-1.2.3-source-release.tar.gz{}}})
 * {{grep}} for legal headers in each file.
 * If time allows check the NOTICE files of the modules whose dependencies have 
been changed in this release in advance, since the license issues from time to 
time pop up during voting. See [Verifying a Flink 
Release|https://cwiki.apache.org/confluence/display/FLINK/Verifying+a+Flink+Release]
 "Checking License" section.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33833) Build Release Candidate: 1.18.1-rc1

[Jing Ge (Jira)]

Jing Ge created FLINK-33833:
---

 Summary: Build Release Candidate: 1.18.1-rc1
 Key: FLINK-33833
 URL: https://issues.apache.org/jira/browse/FLINK-33833
 Project: Flink
  Issue Type: New Feature
Affects Versions: 1.17.0
Reporter: Jing Ge
Assignee: Jing Ge
 Fix For: 1.17.0


The core of the release process is the build-vote-fix cycle. Each cycle 
produces one release candidate. The Release Manager repeats this cycle until 
the community approves one release candidate, which is then finalized.

h4. Prerequisites
Set up a few environment variables to simplify Maven commands that follow. This 
identifies the release candidate being built. Start with {{RC_NUM}} equal to 1 
and increment it for each candidate:
{code}
RC_NUM="1"
TAG="release-${RELEASE_VERSION}-rc${RC_NUM}"
{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33829) CLONE - Review Release Notes in JIRA

[Jing Ge (Jira)]

Jing Ge created FLINK-33829:
---

 Summary: CLONE - Review Release Notes in JIRA
 Key: FLINK-33829
 URL: https://issues.apache.org/jira/browse/FLINK-33829
 Project: Flink
  Issue Type: Sub-task
Reporter: Jing Ge
Assignee: Qingsheng Ren


JIRA automatically generates Release Notes based on the {{Fix Version}} field 
applied to issues. Release Notes are intended for Flink users (not Flink 
committers/contributors). You should ensure that Release Notes are informative 
and useful.

Open the release notes from the version status page by choosing the release 
underway and clicking Release Notes.

You should verify that the issues listed automatically by JIRA are appropriate 
to appear in the Release Notes. Specifically, issues should:
 * Be appropriately classified as {{{}Bug{}}}, {{{}New Feature{}}}, 
{{{}Improvement{}}}, etc.
 * Represent noteworthy user-facing changes, such as new functionality, 
backward-incompatible API changes, or performance improvements.
 * Have occurred since the previous release; an issue that was introduced and 
fixed between releases should not appear in the Release Notes.
 * Have an issue title that makes sense when read on its own.

Adjust any of the above properties to the improve clarity and presentation of 
the Release Notes.

Ensure that the JIRA release notes are also included in the release notes of 
the documentation (see section "Review and update documentation").
h4. Content of Release Notes field from JIRA tickets 

To get the list of "release notes" field from JIRA, you can ran the following 
script using JIRA REST API (notes the maxResults limits the number of entries):
{code:bash}
curl -s 
https://issues.apache.org/jira//rest/api/2/search?maxResults=200=project%20%3D%20FLINK%20AND%20%22Release%20Note%22%20is%20not%20EMPTY%20and%20fixVersion%20%3D%20${RELEASE_VERSION}
 | jq '.issues[]|.key,.fields.summary,.fields.customfield_12310192' | paste - - 
-
{code}
{{jq}}  is present in most Linux distributions and on MacOS can be installed 
via brew.

 

h3. Expectations
 * Release Notes in JIRA have been audited and adjusted



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33827) CLONE - Review and update documentation

[Jing Ge (Jira)]

Jing Ge created FLINK-33827:
---

 Summary: CLONE - Review and update documentation
 Key: FLINK-33827
 URL: https://issues.apache.org/jira/browse/FLINK-33827
 Project: Flink
  Issue Type: Sub-task
Affects Versions: 1.17.0
Reporter: Jing Ge
Assignee: Qingsheng Ren
 Fix For: 1.17.0


There are a few pages in the documentation that need to be reviewed and updated 
for each release.
 * Ensure that there exists a release notes page for each non-bugfix release 
(e.g., 1.5.0) in {{{}./docs/release-notes/{}}}, that it is up-to-date, and 
linked from the start page of the documentation.
 * Upgrading Applications and Flink Versions: 
[https://ci.apache.org/projects/flink/flink-docs-master/ops/upgrading.html]
 * ...

 

h3. Expectations
 * Update upgrade compatibility table 
([apache-flink:./docs/content/docs/ops/upgrading.md|https://github.com/apache/flink/blob/master/docs/content/docs/ops/upgrading.md#compatibility-table]
 and 
[apache-flink:./docs/content.zh/docs/ops/upgrading.md|https://github.com/apache/flink/blob/master/docs/content.zh/docs/ops/upgrading.md#compatibility-table]).
 * Update [Release Overview in 
Confluence|https://cwiki.apache.org/confluence/display/FLINK/Release+Management+and+Feature+Plan]
 * (minor only) The documentation for the new major release is visible under 
[https://nightlies.apache.org/flink/flink-docs-release-$SHORT_RELEASE_VERSION] 
(after at least one [doc 
build|https://github.com/apache/flink/actions/workflows/docs.yml] succeeded).
 * (minor only) The documentation for the new major release does not contain 
"-SNAPSHOT" in its version title, and all links refer to the corresponding 
version docs instead of {{{}master{}}}.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33830) CLONE - Select executing Release Manager

[Jing Ge (Jira)]

Jing Ge created FLINK-33830:
---

 Summary: CLONE - Select executing Release Manager
 Key: FLINK-33830
 URL: https://issues.apache.org/jira/browse/FLINK-33830
 Project: Flink
  Issue Type: Sub-task
  Components: Release System
Affects Versions: 1.17.0
Reporter: Jing Ge
Assignee: Qingsheng Ren
 Fix For: 1.17.0


h4. GPG Key

You need to have a GPG key to sign the release artifacts. Please be aware of 
the ASF-wide [release signing 
guidelines|https://www.apache.org/dev/release-signing.html]. If you don’t have 
a GPG key associated with your Apache account, please create one according to 
the guidelines.

Determine your Apache GPG Key and Key ID, as follows:
{code:java}
$ gpg --list-keys
{code}
This will list your GPG keys. One of these should reflect your Apache account, 
for example:
{code:java}
--
pub   2048R/845E6689 2016-02-23
uid  Nomen Nescio 
sub   2048R/BA4D50BE 2016-02-23
{code}
In the example above, the key ID is the 8-digit hex string in the {{pub}} line: 
{{{}845E6689{}}}.

Now, add your Apache GPG key to the Flink’s {{KEYS}} file in the [Apache Flink 
release KEYS file|https://dist.apache.org/repos/dist/release/flink/KEYS] 
repository at [dist.apache.org|http://dist.apache.org/]. Follow the 
instructions listed at the top of these files. (Note: Only PMC members have 
write access to the release repository. If you end up getting 403 errors ask on 
the mailing list for assistance.)

Configure {{git}} to use this key when signing code by giving it your key ID, 
as follows:
{code:java}
$ git config --global user.signingkey 845E6689
{code}
You may drop the {{--global}} option if you’d prefer to use this key for the 
current repository only.

You may wish to start {{gpg-agent}} to unlock your GPG key only once using your 
passphrase. Otherwise, you may need to enter this passphrase hundreds of times. 
The setup for {{gpg-agent}} varies based on operating system, but may be 
something like this:
{code:bash}
$ eval $(gpg-agent --daemon --no-grab --write-env-file $HOME/.gpg-agent-info)
$ export GPG_TTY=$(tty)
$ export GPG_AGENT_INFO
{code}
h4. Access to Apache Nexus repository

Configure access to the [Apache Nexus 
repository|https://repository.apache.org/], which enables final deployment of 
releases to the Maven Central Repository.
 # You log in with your Apache account.
 # Confirm you have appropriate access by finding {{org.apache.flink}} under 
{{{}Staging Profiles{}}}.
 # Navigate to your {{Profile}} (top right drop-down menu of the page).
 # Choose {{User Token}} from the dropdown, then click {{{}Access User 
Token{}}}. Copy a snippet of the Maven XML configuration block.
 # Insert this snippet twice into your global Maven {{settings.xml}} file, 
typically {{{}${HOME}/.m2/settings.xml{}}}. The end result should look like 
this, where {{TOKEN_NAME}} and {{TOKEN_PASSWORD}} are your secret tokens:
{code:xml}

   
 
   apache.releases.https
   TOKEN_NAME
   TOKEN_PASSWORD
 
 
   apache.snapshots.https
   TOKEN_NAME
   TOKEN_PASSWORD
 
   
 
{code}

h4. Website development setup

Get ready for updating the Flink website by following the [website development 
instructions|https://flink.apache.org/contributing/improve-website.html].
h4. GNU Tar Setup for Mac (Skip this step if you are not using a Mac)

The default tar application on Mac does not support GNU archive format and 
defaults to Pax. This bloats the archive with unnecessary metadata that can 
result in additional files when decompressing (see [1.15.2-RC2 vote 
thread|https://lists.apache.org/thread/mzbgsb7y9vdp9bs00gsgscsjv2ygy58q]). 
Install gnu-tar and create a symbolic link to use in preference of the default 
tar program.
{code:bash}
$ brew install gnu-tar
$ ln -s /usr/local/bin/gtar /usr/local/bin/tar
$ which tar
{code}
 

h3. Expectations
 * Release Manager’s GPG key is published to 
[dist.apache.org|http://dist.apache.org/]
 * Release Manager’s GPG key is configured in git configuration
 * Release Manager's GPG key is configured as the default gpg key.
 * Release Manager has {{org.apache.flink}} listed under Staging Profiles in 
Nexus
 * Release Manager’s Nexus User Token is configured in settings.xml



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33832) CLONE - Verify that no exclusions were erroneously added to the japicmp plugin

[Jing Ge (Jira)]

Jing Ge created FLINK-33832:
---

 Summary: CLONE - Verify that no exclusions were erroneously added 
to the japicmp plugin
 Key: FLINK-33832
 URL: https://issues.apache.org/jira/browse/FLINK-33832
 Project: Flink
  Issue Type: Sub-task
Reporter: Jing Ge
Assignee: Matthias Pohl


Verify that no exclusions were erroneously added to the japicmp plugin that 
break compatibility guarantees. Check the exclusions for the 
japicmp-maven-plugin in the root pom (see 
[apache/flink:pom.xml:2175ff|https://github.com/apache/flink/blob/3856c49af77601cf7943a5072d8c932279ce46b4/pom.xml#L2175]
 for exclusions that:
* For minor releases: break source compatibility for {{@Public}} APIs
* For patch releases: break source/binary compatibility for 
{{@Public}}/{{@PublicEvolving}}  APIs
Any such exclusion must be properly justified, in advance.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33831) CLONE - Create a release branch

[Jing Ge (Jira)]

Jing Ge created FLINK-33831:
---

 Summary: CLONE - Create a release branch
 Key: FLINK-33831
 URL: https://issues.apache.org/jira/browse/FLINK-33831
 Project: Flink
  Issue Type: Sub-task
Affects Versions: 1.17.0
Reporter: Jing Ge
Assignee: Leonard Xu
 Fix For: 1.17.0


If you are doing a new minor release, you need to update Flink version in the 
following repositories and the [AzureCI project 
configuration|https://dev.azure.com/apache-flink/apache-flink/]:
 * [apache/flink|https://github.com/apache/flink]
 * [apache/flink-docker|https://github.com/apache/flink-docker]
 * [apache/flink-benchmarks|https://github.com/apache/flink-benchmarks]

Patch releases don't require the these repositories to be touched. Simply 
checkout the already existing branch for that version:
{code:java}
$ git checkout release-$SHORT_RELEASE_VERSION
{code}
h4. Flink repository

Create a branch for the new version that we want to release before updating the 
master branch to the next development version:
{code:bash}
$ cd ./tools
tools $ releasing/create_snapshot_branch.sh
tools $ git checkout master
tools $ OLD_VERSION=$CURRENT_SNAPSHOT_VERSION 
NEW_VERSION=$NEXT_SNAPSHOT_VERSION releasing/update_branch_version.sh
{code}
In the {{master}} branch, add a new value (e.g. {{v1_16("1.16")}}) to 
[apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
 as the last entry:
{code:java}
// ...
v1_12("1.12"),
v1_13("1.13"),
v1_14("1.14"),
v1_15("1.15"),
v1_16("1.16");
{code}
The newly created branch and updated {{master}} branch need to be pushed to the 
official repository.
h4. Flink Docker Repository

Afterwards fork off from {{dev-master}} a {{dev-x.y}} branch in the 
[apache/flink-docker|https://github.com/apache/flink-docker] repository. Make 
sure that 
[apache/flink-docker:.github/workflows/ci.yml|https://github.com/apache/flink-docker/blob/dev-master/.github/workflows/ci.yml]
 points to the correct snapshot version; for {{dev-x.y}} it should point to 
{{{}x.y-SNAPSHOT{}}}, while for {{dev-master}} it should point to the most 
recent snapshot version (\{[$NEXT_SNAPSHOT_VERSION}}).

After pushing the new minor release branch, as the last step you should also 
update the documentation workflow to also build the documentation for the new 
release branch. Check [Managing 
Documentation|https://cwiki.apache.org/confluence/display/FLINK/Managing+Documentation]
 on details on how to do that. You may also want to manually trigger a build to 
make the changes visible as soon as possible.

h4. Flink Benchmark Repository
First of all, checkout the {{master}} branch to {{dev-x.y}} branch in 
[apache/flink-benchmarks|https://github.com/apache/flink-benchmarks], so that 
we can have a branch named {{dev-x.y}} which could be built on top of 
(${{CURRENT_SNAPSHOT_VERSION}}).

Then, inside the repository you need to manually update the {{flink.version}} 
property inside the parent *pom.xml* file. It should be pointing to the most 
recent snapshot version ($NEXT_SNAPSHOT_VERSION). For example:
{code:xml}
1.18-SNAPSHOT
{code}

h4. AzureCI Project Configuration
The new release branch needs to be configured within AzureCI to make azure 
aware of the new release branch. This matter can only be handled by Ververica 
employees since they are owning the AzureCI setup.
 

h3. Expectations (Minor Version only if not stated otherwise)
 * Release branch has been created and pushed
 * Changes on the new release branch are picked up by [Azure 
CI|https://dev.azure.com/apache-flink/apache-flink/_build?definitionId=1&_a=summary]
 * {{master}} branch has the version information updated to the new version 
(check pom.xml files and 
 * 
[apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
 enum)
 * New version is added to the 
[apache-flink:flink-annotations/src/main/java/org/apache/flink/FlinkVersion|https://github.com/apache/flink/blob/master/flink-annotations/src/main/java/org/apache/flink/FlinkVersion.java]
 enum.
 * Make sure [flink-docker|https://github.com/apache/flink-docker/] has 
{{dev-x.y}} branch and docker e2e tests run against this branch in the 
corresponding Apache Flink release branch (see 
[apache/flink:flink-end-to-end-tests/test-scripts/common_docker.sh:51|https://github.com/apache/flink/blob/master/flink-end-to-end-tests/test-scripts/common_docker.sh#L51])
 * 
[apache-flink:docs/config.toml|https://github.com/apache/flink/blob/release-1.17/docs/config.toml]
 has been updated appropriately in the new Apache Flink release branch.
 * The {{flink.version}} property (see 
[apache/flink-be

[jira] [Created] (FLINK-33825) CLONE - Create a new version in JIRA

[Jing Ge (Jira)]

Jing Ge created FLINK-33825:
---

 Summary: CLONE - Create a new version in JIRA
 Key: FLINK-33825
 URL: https://issues.apache.org/jira/browse/FLINK-33825
 Project: Flink
  Issue Type: Sub-task
Reporter: Jing Ge
Assignee: Martijn Visser


When contributors resolve an issue in JIRA, they are tagging it with a release 
that will contain their changes. With the release currently underway, new 
issues should be resolved against a subsequent future release. Therefore, you 
should create a release item for this subsequent release, as follows:
 # In JIRA, navigate to the [Flink > Administration > 
Versions|https://issues.apache.org/jira/plugins/servlet/project-config/FLINK/versions].
 # Add a new release: choose the next minor version number compared to the one 
currently underway, select today’s date as the Start Date, and choose Add.
(Note: Only PMC members have access to the project administration. If you do 
not have access, ask on the mailing list for assistance.)

 

h3. Expectations
 * The new version should be listed in the dropdown menu of {{fixVersion}} or 
{{affectedVersion}} under "unreleased versions" when creating a new Jira issue.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33824) Prepare Flink 1.18.1 Release

[Jing Ge (Jira)]

Jing Ge created FLINK-33824:
---

 Summary: Prepare Flink 1.18.1 Release
 Key: FLINK-33824
 URL: https://issues.apache.org/jira/browse/FLINK-33824
 Project: Flink
  Issue Type: New Feature
  Components: Release System
Affects Versions: 1.17.0
Reporter: Jing Ge
Assignee: Leonard Xu
 Fix For: 1.17.0


This umbrella issue is meant as a test balloon for moving the [release 
documentation|https://cwiki.apache.org/confluence/display/FLINK/Creating+a+Flink+Release]
 into Jira.
h3. Prerequisites
h4. Environment Variables

Commands in the subtasks might expect some of the following enviroment 
variables to be set accordingly to the version that is about to be released:
{code:bash}
RELEASE_VERSION="1.5.0"
SHORT_RELEASE_VERSION="1.5"
CURRENT_SNAPSHOT_VERSION="$SHORT_RELEASE_VERSION-SNAPSHOT"
NEXT_SNAPSHOT_VERSION="1.6-SNAPSHOT"
SHORT_NEXT_SNAPSHOT_VERSION="1.6"
{code}
h4. Build Tools

All of the following steps require to use Maven 3.2.5 and Java 8. Modify your 
PATH environment variable accordingly if needed.
h4. Flink Source
 * Create a new directory for this release and clone the Flink repository from 
Github to ensure you have a clean workspace (this step is optional).
 * Run {{mvn -Prelease clean install}} to ensure that the build processes that 
are specific to that profile are in good shape (this step is optional).

The rest of this instructions assumes that commands are run in the root (or 
{{./tools}} directory) of a repository on the branch of the release version 
with the above environment variables set.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33828) CLONE - Cross team testing

[Jing Ge (Jira)]

Jing Ge created FLINK-33828:
---

 Summary: CLONE - Cross team testing
 Key: FLINK-33828
 URL: https://issues.apache.org/jira/browse/FLINK-33828
 Project: Flink
  Issue Type: Sub-task
Reporter: Jing Ge
Assignee: Qingsheng Ren


For user facing features that go into the release we'd like to ensure they can 
actually _be used_ by Flink users. To achieve this the release managers ensure 
that an issue for cross team testing is created in the Apache Flink Jira. This 
can and should be picked up by other community members to verify the 
functionality and usability of the feature.
The issue should contain some entry points which enables other community 
members to test it. It should not contain documentation on how to use the 
feature as this should be part of the actual documentation. The cross team 
tests are performed after the feature freeze. Documentation should be in place 
before that. Those tests are manual tests, so do not confuse them with 
automated tests.
To sum that up:
 * User facing features should be tested by other contributors
 * The scope is usability and sanity of the feature
 * The feature needs to be already documented
 * The contributor creates an issue containing some pointers on how to get 
started (e.g. link to the documentation, suggested targets of verification)
 * Other community members pick those issues up and provide feedback
 * Cross team testing happens right after the feature freeze

 

h3. Expectations
 * Jira issues for each expected release task according to the release plan is 
created and labeled as {{{}release-testing{}}}.
 * All the created release-testing-related Jira issues are resolved and the 
corresponding blocker issues are fixed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33826) CLONE - Triage release-blocking issues in JIRA

[Jing Ge (Jira)]

Jing Ge created FLINK-33826:
---

 Summary: CLONE - Triage release-blocking issues in JIRA
 Key: FLINK-33826
 URL: https://issues.apache.org/jira/browse/FLINK-33826
 Project: Flink
  Issue Type: Sub-task
Reporter: Jing Ge
Assignee: Qingsheng Ren


There could be outstanding release-blocking issues, which should be triaged 
before proceeding to build a release candidate. We track them by assigning a 
specific Fix version field even before the issue resolved.

The list of release-blocking issues is available at the version status page. 
Triage each unresolved issue with one of the following resolutions:
 * If the issue has been resolved and JIRA was not updated, resolve it 
accordingly.
 * If the issue has not been resolved and it is acceptable to defer this until 
the next release, update the Fix Version field to the new version you just 
created. Please consider discussing this with stakeholders and the dev@ mailing 
list, as appropriate.
 ** When using "Bulk Change" functionality of Jira
 *** First, add the newly created version to Fix Version for all unresolved 
tickets that have old the old version among its Fix Versions.
 *** Afterwards, remove the old version from the Fix Version.
 * If the issue has not been resolved and it is not acceptable to release until 
it is fixed, the release cannot proceed. Instead, work with the Flink community 
to resolve the issue.

 

h3. Expectations
 * There are no release blocking JIRA issues



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Release Flink 1.18.1

[Jing Ge]

Hi folks,

What Martijn said makes sense. We should pay more attention to
https://issues.apache.org/jira/browse/FLINK-33793. I have also tried to
contribute and share my thoughts in the PR
https://github.com/apache/flink/pull/23489.

For 1.18.1-rc1, I will wait until tomorrow. If there is no progress with
the PR, I will move forward with the release, unless we upgrade the ticket
to be a Blocker. Look forward to your feedback.

Best regards,
Jing

On Wed, Dec 13, 2023 at 9:57 AM Jing Ge  wrote:

> Hi Martijn,
>
> Thanks for the heads-up! I already started 1.18.1-rc1[1]. But please feel
> free to ping me. I will either redo rc1 or start rc2. Thanks!
>
> Best regards,
> Jing
>
> [1] https://dist.apache.org/repos/dist/dev/flink/flink-1.18.1-rc1/
>
> On Tue, Dec 12, 2023 at 9:49 PM Martijn Visser 
> wrote:
>
>> Hi Jing,
>>
>> The only thing that was brought to my attention today was
>> https://issues.apache.org/jira/browse/FLINK-33793. I've marked it as a
>> Critical (I think we should have working checkpointing with GCS), but
>> could also be considered a Blocker. There is a PR open for it, but I
>> don't think it's the right fix so I'm testing
>>
>> https://github.com/apache/flink/commit/2ebe7f6e090690486c1954099a2b57283c578192
>> at this moment. If you haven't started the 1.18.1 release yet, perhaps
>> we could include it (hopefully we can merge it tomorrow), else it
>> would have to wait for the next release.
>>
>> Thanks,
>>
>> Martijn
>>
>> On Tue, Dec 12, 2023 at 9:30 AM Jing Ge 
>> wrote:
>> >
>> > Hi All,
>> >
>> > Thank you for your feedback!
>> >
>> > Since FLINK-33523[1] is done(thanks Martijn for driving it) and there
>> are
>> > no other concerns or objections. Currently I am not aware of any
>> unresolved
>> > blockers. There is one critical task[2] whose PR still has some
>> checkstyle
>> > issues. I will try to include it into 1.18.1 with best effort, since it
>> > depends on how quickly the author can finish it and cp to 1.18. If
>> anyone
>> > considers FLINK-33588 as a must-have fix in 1.18.1, please let me know,
>> > thanks!
>> >
>> > I will start working on the RC1 release today.
>> >
>> > @benchao: gotcha wrt FLINK-33313
>> >
>> > Best regards,
>> > Jing
>> >
>> > [1] https://issues.apache.org/jira/browse/FLINK-33523
>> > [2] https://issues.apache.org/jira/browse/FLINK-33588
>> >
>> > On Tue, Dec 12, 2023 at 7:03 AM weijie guo 
>> > wrote:
>> >
>> > > Thanks Jing for driving this bug-fix release.
>> > >
>> > > +1 from my side.
>> > >
>> > > Best regards,
>> > >
>> > > Weijie
>> > >
>> > >
>> > > Jark Wu  于2023年12月12日周二 12:17写道：
>> > >
>> > > > Thanks Jing for driving 1.18.1.
>> > > > +1 for this.
>> > > >
>> > > > Best,
>> > > > Jark
>> > > >
>> > > > On Mon, 11 Dec 2023 at 16:59, Hong Liang 
>> wrote:
>> > > >
>> > > > > +1. Thanks Jing for driving this.
>> > > > >
>> > > > > Hong
>> > > > >
>> > > > > On Mon, Dec 11, 2023 at 2:27 AM Yun Tang 
>> wrote:
>> > > > >
>> > > > > > Thanks Jing for driving 1.18.1 release, +1 for this.
>> > > > > >
>> > > > > >
>> > > > > > Best
>> > > > > > Yun Tang
>> > > > > > 
>> > > > > > From: Rui Fan <1996fan...@gmail.com>
>> > > > > > Sent: Saturday, December 9, 2023 21:46
>> > > > > > To: dev@flink.apache.org 
>> > > > > > Subject: Re: [DISCUSS] Release Flink 1.18.1
>> > > > > >
>> > > > > > Thanks Jing for driving this release, +1
>> > > > > >
>> > > > > > Best,
>> > > > > > Rui
>> > > > > >
>> > > > > > On Sat, Dec 9, 2023 at 7:33 AM Leonard Xu 
>> wrote:
>> > > > > >
>> > > > > > > Thanks Jing for driving this release, +1
>> > > > > > >
>> > > > > > > Best,
>> > > > > > > Leonard
>> > > > > > >
>> > > > > > > >

Re: [DISCUSS] Release Flink 1.18.1

[Jing Ge]

Hi Martijn,

Thanks for the heads-up! I already started 1.18.1-rc1[1]. But please feel
free to ping me. I will either redo rc1 or start rc2. Thanks!

Best regards,
Jing

[1] https://dist.apache.org/repos/dist/dev/flink/flink-1.18.1-rc1/

On Tue, Dec 12, 2023 at 9:49 PM Martijn Visser 
wrote:

> Hi Jing,
>
> The only thing that was brought to my attention today was
> https://issues.apache.org/jira/browse/FLINK-33793. I've marked it as a
> Critical (I think we should have working checkpointing with GCS), but
> could also be considered a Blocker. There is a PR open for it, but I
> don't think it's the right fix so I'm testing
>
> https://github.com/apache/flink/commit/2ebe7f6e090690486c1954099a2b57283c578192
> at this moment. If you haven't started the 1.18.1 release yet, perhaps
> we could include it (hopefully we can merge it tomorrow), else it
> would have to wait for the next release.
>
> Thanks,
>
> Martijn
>
> On Tue, Dec 12, 2023 at 9:30 AM Jing Ge 
> wrote:
> >
> > Hi All,
> >
> > Thank you for your feedback!
> >
> > Since FLINK-33523[1] is done(thanks Martijn for driving it) and there are
> > no other concerns or objections. Currently I am not aware of any
> unresolved
> > blockers. There is one critical task[2] whose PR still has some
> checkstyle
> > issues. I will try to include it into 1.18.1 with best effort, since it
> > depends on how quickly the author can finish it and cp to 1.18. If anyone
> > considers FLINK-33588 as a must-have fix in 1.18.1, please let me know,
> > thanks!
> >
> > I will start working on the RC1 release today.
> >
> > @benchao: gotcha wrt FLINK-33313
> >
> > Best regards,
> > Jing
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-33523
> > [2] https://issues.apache.org/jira/browse/FLINK-33588
> >
> > On Tue, Dec 12, 2023 at 7:03 AM weijie guo 
> > wrote:
> >
> > > Thanks Jing for driving this bug-fix release.
> > >
> > > +1 from my side.
> > >
> > > Best regards,
> > >
> > > Weijie
> > >
> > >
> > > Jark Wu  于2023年12月12日周二 12:17写道：
> > >
> > > > Thanks Jing for driving 1.18.1.
> > > > +1 for this.
> > > >
> > > > Best,
> > > > Jark
> > > >
> > > > On Mon, 11 Dec 2023 at 16:59, Hong Liang  wrote:
> > > >
> > > > > +1. Thanks Jing for driving this.
> > > > >
> > > > > Hong
> > > > >
> > > > > On Mon, Dec 11, 2023 at 2:27 AM Yun Tang  wrote:
> > > > >
> > > > > > Thanks Jing for driving 1.18.1 release, +1 for this.
> > > > > >
> > > > > >
> > > > > > Best
> > > > > > Yun Tang
> > > > > > 
> > > > > > From: Rui Fan <1996fan...@gmail.com>
> > > > > > Sent: Saturday, December 9, 2023 21:46
> > > > > > To: dev@flink.apache.org 
> > > > > > Subject: Re: [DISCUSS] Release Flink 1.18.1
> > > > > >
> > > > > > Thanks Jing for driving this release, +1
> > > > > >
> > > > > > Best,
> > > > > > Rui
> > > > > >
> > > > > > On Sat, Dec 9, 2023 at 7:33 AM Leonard Xu 
> wrote:
> > > > > >
> > > > > > > Thanks Jing for driving this release, +1
> > > > > > >
> > > > > > > Best,
> > > > > > > Leonard
> > > > > > >
> > > > > > > > 2023年12月9日 上午1:23，Danny Cranmer 
> 写道：
> > > > > > > >
> > > > > > > > +1
> > > > > > > >
> > > > > > > > Thanks for driving this
> > > > > > > >
> > > > > > > > On Fri, 8 Dec 2023, 12:05 Timo Walther, 
> > > > wrote:
> > > > > > > >
> > > > > > > >> Thanks for taking care of this Jing.
> > > > > > > >>
> > > > > > > >> +1 to release 1.18.1 for this.
> > > > > > > >>
> > > > > > > >> Cheers,
> > > > > > > >> Timo
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> On 08.12.23 10:00, Benchao Li wrote:
> > > > > > > >>> I've merged

Re: [VOTE] FLIP-335: Deprecation and removal of Flink's Time classes in favor of Java's Duration

[Jing Ge]

+1(binding)

Thanks.

Best regards,
Jing

On Tue, Dec 12, 2023 at 1:12 PM Zhu Zhu  wrote:

> +1 (binding)
>
> Thanks,
> Zhu
>
> Matthias Pohl  于2023年12月12日周二 01:32写道：
>
> > Hi everyone,
> > I'd like to start a vote on FLIP-335 [1]. It covers the deprecation (and
> > Flink 2.0-related removal) of Flink's Time classes in favor of Java's
> > Duration class.
> >
> > The related discussion thread can be found in [2].
> >
> > The vote will remain open for at least 72 hours and only concluded if
> there
> > are no objections and enough (i.e. at least 3) binding votes.
> >
> > Matthias
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-335%3A+Removing+Flink%27s+Time+classes
> > [2] https://lists.apache.org/thread/48ysrg1rrtl8s1twg9wmx35l201hnc2w
> >
> > --
> >
> > [image: Aiven] 
> >
> > *Matthias Pohl*
> > Opensource Software Engineer, *Aiven*
> > matthias.p...@aiven.io|  +49 170 9869525
> > aiven.io    |   <
> https://www.facebook.com/aivencloud
> > >
> >      <
> > https://twitter.com/aiven_io>
> > *Aiven Deutschland GmbH*
> > Alexanderufer 3-7, 10117 Berlin
> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> > Amtsgericht Charlottenburg, HRB 209739 B
> >
>

Re: [DISCUSS] Release Flink 1.18.1

[Jing Ge]

Hi All,

Thank you for your feedback!

Since FLINK-33523[1] is done(thanks Martijn for driving it) and there are
no other concerns or objections. Currently I am not aware of any unresolved
blockers. There is one critical task[2] whose PR still has some checkstyle
issues. I will try to include it into 1.18.1 with best effort, since it
depends on how quickly the author can finish it and cp to 1.18. If anyone
considers FLINK-33588 as a must-have fix in 1.18.1, please let me know,
thanks!

I will start working on the RC1 release today.

@benchao: gotcha wrt FLINK-33313

Best regards,
Jing

[1] https://issues.apache.org/jira/browse/FLINK-33523
[2] https://issues.apache.org/jira/browse/FLINK-33588

On Tue, Dec 12, 2023 at 7:03 AM weijie guo 
wrote:

> Thanks Jing for driving this bug-fix release.
>
> +1 from my side.
>
> Best regards,
>
> Weijie
>
>
> Jark Wu  于2023年12月12日周二 12:17写道：
>
> > Thanks Jing for driving 1.18.1.
> > +1 for this.
> >
> > Best,
> > Jark
> >
> > On Mon, 11 Dec 2023 at 16:59, Hong Liang  wrote:
> >
> > > +1. Thanks Jing for driving this.
> > >
> > > Hong
> > >
> > > On Mon, Dec 11, 2023 at 2:27 AM Yun Tang  wrote:
> > >
> > > > Thanks Jing for driving 1.18.1 release, +1 for this.
> > > >
> > > >
> > > > Best
> > > > Yun Tang
> > > > 
> > > > From: Rui Fan <1996fan...@gmail.com>
> > > > Sent: Saturday, December 9, 2023 21:46
> > > > To: dev@flink.apache.org 
> > > > Subject: Re: [DISCUSS] Release Flink 1.18.1
> > > >
> > > > Thanks Jing for driving this release, +1
> > > >
> > > > Best,
> > > > Rui
> > > >
> > > > On Sat, Dec 9, 2023 at 7:33 AM Leonard Xu  wrote:
> > > >
> > > > > Thanks Jing for driving this release, +1
> > > > >
> > > > > Best,
> > > > > Leonard
> > > > >
> > > > > > 2023年12月9日 上午1:23，Danny Cranmer  写道：
> > > > > >
> > > > > > +1
> > > > > >
> > > > > > Thanks for driving this
> > > > > >
> > > > > > On Fri, 8 Dec 2023, 12:05 Timo Walther, 
> > wrote:
> > > > > >
> > > > > >> Thanks for taking care of this Jing.
> > > > > >>
> > > > > >> +1 to release 1.18.1 for this.
> > > > > >>
> > > > > >> Cheers,
> > > > > >> Timo
> > > > > >>
> > > > > >>
> > > > > >> On 08.12.23 10:00, Benchao Li wrote:
> > > > > >>> I've merged FLINK-33313 to release-1.18 branch.
> > > > > >>>
> > > > > >>> Péter Váry  于2023年12月8日周五
> 16:56写道：
> > > > > >>>>
> > > > > >>>> Hi Jing,
> > > > > >>>> Thanks for taking care of this!
> > > > > >>>> +1 (non-binding)
> > > > > >>>> Peter
> > > > > >>>>
> > > > > >>>> Sergey Nuyanzin  ezt írta (időpont:
> 2023.
> > > dec.
> > > > > >> 8., P,
> > > > > >>>> 9:36):
> > > > > >>>>
> > > > > >>>>> Thanks Jing driving it
> > > > > >>>>> +1
> > > > > >>>>>
> > > > > >>>>> also +1 to include FLINK-33313 mentioned by Benchao Li
> > > > > >>>>>
> > > > > >>>>> On Fri, Dec 8, 2023 at 9:17 AM Benchao Li <
> > libenc...@apache.org>
> > > > > >> wrote:
> > > > > >>>>>
> > > > > >>>>>> Thanks Jing for driving 1.18.1 releasing.
> > > > > >>>>>>
> > > > > >>>>>> I would like to include FLINK-33313[1] in 1.18.1, it's just
> a
> > > > > bugfix,
> > > > > >>>>>> not a blocker, but it's already merged into master, I plan
> to
> > > > merge
> > > > > it
> > > > > >>>>>> to 1.8/1.7 branches today after the CI passes.
> > > > > >>>>>>
> > > > > >>>>>> [1] https://issues.apache.org/jira/browse/FLINK-33313
> > > > > &g

Re: SQL return type change from 1.17 to 1.18

[Jing Ge]

Hi all,

FYI: the 1.18.1 release discussion thread:
https://lists.apache.org/thread/1sjmn3lj32rrqmdmqom43hr2c34qczvw
Look forward to your feedback. Thanks!

Best regards,
Jing

On Fri, Dec 8, 2023 at 8:51 AM Jing Ge  wrote:

> Hi all,
>
> +1 for reverting it.
>
> I'd like to volunteer as the release manager of 1.18.1 and will send the
> discussion email for the 1.18 release asap.
>
> Best regards,
> Jing
>
> On Fri, Dec 8, 2023 at 1:26 AM Rui Fan <1996fan...@gmail.com> wrote:
>
>> Hi all,
>>
>> +1 for reverting it.
>>
>> Also, I’m not sure should we release 1.18.1 as soon as possible after it’s
>> merged. If I understand correctly, users cannot upgrade flink job from
>> 1.18.0 to 1.18.1 directly if we reverting it in 1.18.1, right?
>>
>> If so, it’s better to release 1.18.1 asap, and try to avoid users to use a
>> wrong Data type.
>> Looking forward to hearing from you, thank you.
>>
>> Best,
>> Rui
>>
>> On Fri, 8 Dec 2023 at 00:28, Xingcan Cui  wrote:
>>
>> > Hi,
>> >
>> > Thanks for bringing this up, Peter. I'm +1 for reverting the change.
>> >
>> > Best,
>> > Xingcan
>> >
>> > On Thu, Dec 7, 2023 at 10:40 AM Martijn Visser <
>> martijnvis...@apache.org>
>> > wrote:
>> >
>> > > Hi all,
>> > >
>> > > Agree with what has been said already. I've marked
>> > > https://issues.apache.org/jira/browse/FLINK-33523 as a blocker ticket
>> > > for 1.18.1
>> > >
>> > > Best regards,
>> > >
>> > > Martijn
>> > >
>> > > On Thu, Dec 7, 2023 at 4:25 PM Márton Balassi <
>> balassi.mar...@gmail.com>
>> > > wrote:
>> > > >
>> > > > Thanks, for raising this Peter. +1 for reverting the change.
>> > > >
>> > > > Given the response from Timo and Aitozi, I believe it would be best
>> if
>> > we
>> > > > could ship reverting the change in 1.18.1.
>> > > >
>> > > > On Thu, Dec 7, 2023 at 2:47 PM Aitozi  wrote:
>> > > >
>> > > > > Hi Peter, Timo
>> > > > > Sorry for this breaking change, I didn't notice that was a
>> > breaking
>> > > > > change.
>> > > > > I'm +1 to revert the FLINK-33523
>> > > > >
>> > > > > Regards,
>> > > > > aitozi
>> > > > >
>> > > > > Timo Walther  于2023年12月7日周四 20:41写道：
>> > > > >
>> > > > > > Hi Peter,
>> > > > > >
>> > > > > > thanks for reaching out to the Flink community. This is indeed a
>> > > serious
>> > > > > > issue. As the author of the Flink type system, DataType and many
>> > > related
>> > > > > > utilities I strongly vote for reverting FLINK-33523:
>> > > > > >
>> > > > > > - It changes the Flink type system without a FLIP.
>> > > > > > - It breaks backwards compatibility with UDFs and connectors.
>> > > > > >
>> > > > > > Regards,
>> > > > > > Timo
>> > > > > >
>> > > > > > On 07.12.23 07:38, Péter Váry wrote:
>> > > > > > > Hi Team,
>> > > > > > >
>> > > > > > > We are working on upgrading the Iceberg-Flink connector from
>> 1.17
>> > > to
>> > > > > > 1.18,
>> > > > > > > and found that some of our tests are failing. Prabhu Joseph
>> > > created a
>> > > > > > jira
>> > > > > > > [1] to discuss this issue, along with short example code.
>> > > > > > >
>> > > > > > > In a nutshell:
>> > > > > > > - Create a table with an 'ARRAY' column
>> > > > > > > - Run a select which returns this column
>> > > > > > > - The return type changes:
>> > > > > > >  - From 'Object[]' - in 1.17
>> > > > > > >  - To 'int[]' - in 1.18
>> > > > > > >
>> > > > > > > The change is introduced by this jira [2].
>> > > > > > >
>> > > > > > > While I understand the reasoning behind this change, this will
>> > > break
>> > > > > some
>> > > > > > > users existing workflow as evidenced by Xingcan Cui finding
>> this
>> > > > > > > independently [3].
>> > > > > > >
>> > > > > > > What is the opinion of the community about this change?
>> > > > > > > - Do we want to revert the change?
>> > > > > > > - Do we ask the owners of the change to make this behavior
>> > > > > configurable?
>> > > > > > > - Do we accept this behavior change in a minor release?
>> > > > > > >
>> > > > > > > Thanks,
>> > > > > > > Peter
>> > > > > > >
>> > > > > > > [1] - https://issues.apache.org/jira/browse/FLINK-33523 -
>> > DataType
>> > > > > > > ARRAY fails to cast into Object[]
>> > > > > > > [2] - https://issues.apache.org/jira/browse/FLINK-31835 -
>> > > DataTypeHint
>> > > > > > > don't support Row>
>> > > > > > > [3] - https://issues.apache.org/jira/browse/FLINK-33547 - SQL
>> > > > > primitive
>> > > > > > > array type after upgrading to Flink 1.18.0
>> > > > > > >
>> > > > > >
>> > > > > >
>> > > > >
>> > >
>> >
>>
>

[DISCUSS] Release Flink 1.18.1

[Jing Ge]

Hi all,

I would like to discuss creating a new 1.18 patch release (1.18.1). The
last 1.18 release is nearly two months old, and since then, 37 tickets have
been closed [1], of which 6 are blocker/critical [2].  Some of them are
quite important, such as FLINK-33598 [3]

Most urgent and important one is FLINK-33523 [4] and according to the
discussion thread[5] on the ML, 1.18.1 should/must be released asap after
the breaking change commit has been reverted.

I am not aware of any other unresolved blockers and there are no in-progress
tickets [6].
Please let me know if there are any issues you'd like to be included in
this release but still not merged.

If the community agrees to create this new patch release, I could
volunteer as the release manager.

Best regards,
Jing

[1]
https://issues.apache.org/jira/browse/FLINK-33567?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%201.18.1%20%20and%20resolution%20%20!%3D%20%20Unresolved%20order%20by%20priority%20DESC
[2]
https://issues.apache.org/jira/browse/FLINK-33693?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%201.18.1%20and%20resolution%20%20!%3D%20Unresolved%20%20and%20priority%20in%20(Blocker%2C%20Critical)%20ORDER%20by%20priority%20%20DESC
[3] https://issues.apache.org/jira/browse/FLINK-33598
[4] https://issues.apache.org/jira/browse/FLINK-33523
[5] https://lists.apache.org/thread/m4c879y8mb7hbn2kkjh9h3d8g1jphh3j
[6] https://issues.apache.org/jira/projects/FLINK/versions/12353640
Thanks,

Re: SQL return type change from 1.17 to 1.18

[Jing Ge]

Hi all,

+1 for reverting it.

I'd like to volunteer as the release manager of 1.18.1 and will send the
discussion email for the 1.18 release asap.

Best regards,
Jing

On Fri, Dec 8, 2023 at 1:26 AM Rui Fan <1996fan...@gmail.com> wrote:

> Hi all,
>
> +1 for reverting it.
>
> Also, I’m not sure should we release 1.18.1 as soon as possible after it’s
> merged. If I understand correctly, users cannot upgrade flink job from
> 1.18.0 to 1.18.1 directly if we reverting it in 1.18.1, right?
>
> If so, it’s better to release 1.18.1 asap, and try to avoid users to use a
> wrong Data type.
> Looking forward to hearing from you, thank you.
>
> Best,
> Rui
>
> On Fri, 8 Dec 2023 at 00:28, Xingcan Cui  wrote:
>
> > Hi,
> >
> > Thanks for bringing this up, Peter. I'm +1 for reverting the change.
> >
> > Best,
> > Xingcan
> >
> > On Thu, Dec 7, 2023 at 10:40 AM Martijn Visser  >
> > wrote:
> >
> > > Hi all,
> > >
> > > Agree with what has been said already. I've marked
> > > https://issues.apache.org/jira/browse/FLINK-33523 as a blocker ticket
> > > for 1.18.1
> > >
> > > Best regards,
> > >
> > > Martijn
> > >
> > > On Thu, Dec 7, 2023 at 4:25 PM Márton Balassi <
> balassi.mar...@gmail.com>
> > > wrote:
> > > >
> > > > Thanks, for raising this Peter. +1 for reverting the change.
> > > >
> > > > Given the response from Timo and Aitozi, I believe it would be best
> if
> > we
> > > > could ship reverting the change in 1.18.1.
> > > >
> > > > On Thu, Dec 7, 2023 at 2:47 PM Aitozi  wrote:
> > > >
> > > > > Hi Peter, Timo
> > > > > Sorry for this breaking change, I didn't notice that was a
> > breaking
> > > > > change.
> > > > > I'm +1 to revert the FLINK-33523
> > > > >
> > > > > Regards,
> > > > > aitozi
> > > > >
> > > > > Timo Walther  于2023年12月7日周四 20:41写道：
> > > > >
> > > > > > Hi Peter,
> > > > > >
> > > > > > thanks for reaching out to the Flink community. This is indeed a
> > > serious
> > > > > > issue. As the author of the Flink type system, DataType and many
> > > related
> > > > > > utilities I strongly vote for reverting FLINK-33523:
> > > > > >
> > > > > > - It changes the Flink type system without a FLIP.
> > > > > > - It breaks backwards compatibility with UDFs and connectors.
> > > > > >
> > > > > > Regards,
> > > > > > Timo
> > > > > >
> > > > > > On 07.12.23 07:38, Péter Váry wrote:
> > > > > > > Hi Team,
> > > > > > >
> > > > > > > We are working on upgrading the Iceberg-Flink connector from
> 1.17
> > > to
> > > > > > 1.18,
> > > > > > > and found that some of our tests are failing. Prabhu Joseph
> > > created a
> > > > > > jira
> > > > > > > [1] to discuss this issue, along with short example code.
> > > > > > >
> > > > > > > In a nutshell:
> > > > > > > - Create a table with an 'ARRAY' column
> > > > > > > - Run a select which returns this column
> > > > > > > - The return type changes:
> > > > > > >  - From 'Object[]' - in 1.17
> > > > > > >  - To 'int[]' - in 1.18
> > > > > > >
> > > > > > > The change is introduced by this jira [2].
> > > > > > >
> > > > > > > While I understand the reasoning behind this change, this will
> > > break
> > > > > some
> > > > > > > users existing workflow as evidenced by Xingcan Cui finding
> this
> > > > > > > independently [3].
> > > > > > >
> > > > > > > What is the opinion of the community about this change?
> > > > > > > - Do we want to revert the change?
> > > > > > > - Do we ask the owners of the change to make this behavior
> > > > > configurable?
> > > > > > > - Do we accept this behavior change in a minor release?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Peter
> > > > > > >
> > > > > > > [1] - https://issues.apache.org/jira/browse/FLINK-33523 -
> > DataType
> > > > > > > ARRAY fails to cast into Object[]
> > > > > > > [2] - https://issues.apache.org/jira/browse/FLINK-31835 -
> > > DataTypeHint
> > > > > > > don't support Row>
> > > > > > > [3] - https://issues.apache.org/jira/browse/FLINK-33547 - SQL
> > > > > primitive
> > > > > > > array type after upgrading to Flink 1.18.0
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > >
> >
>

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

[Jing Ge]

Awesome! +1

Best regards,
Jing

On Thu, Dec 7, 2023 at 8:34 AM Sergey Nuyanzin  wrote:

> thanks for working on this and driving it
>
> +1
>
> On Thu, Dec 7, 2023 at 7:26 AM Feng Jin  wrote:
>
> > This is incredibly exciting news, a big +1 for this.
> >
> > Thank you for the fantastic work on Flink CDC. We have created thousands
> of
> > real-time integration jobs using Flink CDC connectors.
> >
> >
> > Best,
> > Feng
> >
> > On Thu, Dec 7, 2023 at 1:45 PM gongzhongqiang  >
> > wrote:
> >
> > > It's very exciting to hear the news.
> > > +1 for adding CDC Connectors  to Apache Flink !
> > >
> > >
> > > Best,
> > > Zhongqiang
> > >
> > > Leonard Xu  于2023年12月7日周四 11:25写道：
> > >
> > > > Dear Flink devs,
> > > >
> > > >
> > > > As you may have heard, we at Alibaba (Ververica) are planning to
> donate
> > > CDC Connectors for the Apache Flink project
> > > > *[1]* to the Apache Flink community.
> > > >
> > > >
> > > >
> > > > CDC Connectors for Apache Flink comprise a collection of source
> > > connectors designed specifically for Apache Flink. These connectors
> > > > *[2]*
> > > >  enable the ingestion of changes from various databases using Change
> > > Data Capture (CDC), most of these CDC connectors are powered by
> Debezium
> > > > *[3]*
> > > > . They support both the DataStream API and the Table/SQL API,
> > > facilitating the reading of database snapshots and continuous reading
> of
> > > transaction logs with exactly-once processing, even in the event of
> > > failures.
> > > >
> > > >
> > > >
> > > > Additionally, in the latest version 3.0, we have introduced many
> > > long-awaited features. Starting from CDC version 3.0, we've built a
> > > Streaming ELT Framework available for streaming data integration. This
> > > framework allows users to write their data synchronization logic in a
> > > simple YAML file, which will automatically be translated into a Flink
> > > DataStreaming job. It emphasizes optimizing the task submission process
> > and
> > > offers advanced functionalities such as whole database synchronization,
> > > merging sharded tables, and schema evolution
> > > > *[4]*.
> > > >
> > > >
> > > >
> > > >
> > > > I believe this initiative is a perfect match for both sides. For the
> > > Flink community, it presents an opportunity to enhance Flink's
> > competitive
> > > advantage in streaming data integration, promoting the healthy growth
> and
> > > prosperity of the Apache Flink ecosystem. For the CDC Connectors
> project,
> > > becoming a sub-project of Apache Flink means being part of a neutral
> > > open-source community, which can attract a more diverse pool of
> > > contributors.
> > > >
> > > >
> > > > Please note that the aforementioned points represent only some of our
> > > motivations and vision for this donation. Specific future operations
> need
> > > to be further discussed in this thread. For example, the sub-project
> name
> > > after the donation; we hope to name it Flink-CDC
> > > > aiming to streaming data intergration through Apache Flink,
> > > > following the naming convention of Flink-ML; And this project is
> > managed
> > > by a total of 8 maintainers, including 3 Flink PMC members and 1 Flink
> > > Committer. The remaining 4 maintainers are also highly active
> > contributors
> > > to the Flink community, donating this project to the Flink community
> > > implies that their permissions might be reduced. Therefore, we may need
> > to
> > > bring up this topic for further discussion within the Flink PMC.
> > > Additionally, we need to discuss how to migrate existing users and
> > > documents. We have a user group of nearly 10,000 people and a
> > multi-version
> > > documentation site need to migrate. We also need to plan for the
> > migration
> > > of CI/CD processes and other specifics.
> > > >
> > > >
> > > >
> > > > While there are many intricate details that require implementation,
> we
> > > are committed to progressing and finalizing this donation process.
> > > >
> > > >
> > > >
> > > > Despite being Flink’s most active ecological project (as evaluated by
> > > GitHub metrics), it also boasts a significant user base. However, I
> > believe
> > > it's essential to commence discussions on future operations only after
> > the
> > > community reaches a consensus on whether they desire this donation.
> > > >
> > > >
> > > > Really looking forward to hear what you think!
> > > >
> > > >
> > > >
> > > > Best,
> > > > Leonard (on behalf of the Flink CDC Connectors project maintainers)
> > > >
> > > > [1] https://github.com/ververica/flink-cdc-connectors
> > > > [2]
> > > >
> > >
> >
> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html
> > > > [3] https://debezium.io
> > > > [4]
> > > >
> > >
> >
> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html
> > > >
> > >
> >
>
>
> --
> Best regards,
> Sergey
>

[jira] [Created] (FLINK-33730) Update the compatibility table to only include last three versions

[Jing Ge (Jira)]

Jing Ge created FLINK-33730:
---

 Summary: Update the compatibility table to only include last three 
versions
 Key: FLINK-33730
 URL: https://issues.apache.org/jira/browse/FLINK-33730
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Reporter: Jing Ge
Assignee: Jing Ge


Update the compatibility table 
([apache-flink:./docs/content/docs/ops/upgrading.md|https://github.com/apache/flink/blob/master/docs/content/docs/ops/upgrading.md#compatibility-table]
 and 
[apache-flink:./docs/content.zh/docs/ops/upgrading.md|https://github.com/apache/flink/blob/master/docs/content.zh/docs/ops/upgrading.md#compatibility-table])
 according to the discussion[1].

 

[1] https://lists.apache.org/thread/7yx396x5lmtws0s4t0sf9f2psgny11d6

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] FLIP-364: Improve the restart-strategy

[Jing Ge]

+1(binding)
Thanks!

Best regards,
Jing

On Fri, Dec 1, 2023 at 9:29 AM Matthias Pohl 
wrote:

> +1 (binding)
>
> On Fri, Dec 1, 2023 at 3:40 AM Zhu Zhu  wrote:
>
> > +1 (binding)
> >
> > Thanks,
> > Zhu
> >
> > Zhanghao Chen  于2023年11月30日周四 23:31写道：
> >
> > > +1 (non-binding)
> > >
> > > Best,
> > > Zhanghao Chen
> > > 
> > > From: Rui Fan <1996fan...@gmail.com>
> > > Sent: Monday, November 13, 2023 11:01
> > > To: dev 
> > > Subject: [VOTE] FLIP-364: Improve the restart-strategy
> > >
> > > Hi everyone,
> > >
> > > Thank you to everyone for the feedback on FLIP-364: Improve the
> > > restart-strategy[1]
> > > which has been discussed in this thread [2].
> > >
> > > I would like to start a vote for it. The vote will be open for at least
> > 72
> > > hours unless there is an objection or not enough votes.
> > >
> > > [1] https://cwiki.apache.org/confluence/x/uJqzDw
> > > [2] https://lists.apache.org/thread/5cgrft73kgkzkgjozf9zfk0w2oj7rjym
> > >
> > > Best,
> > > Rui
> > >
> >
>

[jira] [Created] (FLINK-33726) Print cost time for stream queries in SQL Client

[Jing Ge (Jira)]

Jing Ge created FLINK-33726:
---

 Summary: Print cost time for stream queries in SQL Client
 Key: FLINK-33726
 URL: https://issues.apache.org/jira/browse/FLINK-33726
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Client
Reporter: Jing Ge
Assignee: Jing Ge






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Release flink-shaded 18.0, release candidate #1

[Jing Ge]

+1(not binding)

- validate checksum
- validate hash
- checked the release notes
- verified that no binaries exist in the source archive
- build the source with Maven 3.8.6 and jdk11
- checked repo
- checked tag
- verified web PR

Best regards,
Jing

On Thu, Nov 30, 2023 at 11:39 AM Sergey Nuyanzin 
wrote:

> +1 (non-binding)
>
> - Downloaded all the resources
> - Validated checksum hash
> - Build the source with Maven and jdk8
> - Build Flink master with new flink-shaded and check that all the tests are
> passing
>
> one minor thing that I noticed during releasing: for ci it uses maven 3.8.6
> at the same time for release profile there is an enforcement plugin to
> check that maven version is less than 3.3
> I created a jira issue[1] for that
> i made the release with 3.2.5 maven version (I suppose previous version was
> also done with 3.2.5 because of same issue)
>
> [1] https://issues.apache.org/jira/browse/FLINK-33703
>
> On Wed, Nov 29, 2023 at 11:41 AM Matthias Pohl 
> wrote:
>
> > +1 (binding)
> >
> > * Downloaded all resources
> > * Extracts sources and compilation on these sources
> > * Diff of git tag checkout with downloaded sources
> > * Verifies SHA512 checksums & GPG certification
> > * Checks that all POMs have the right expected version
> > * Generated diffs to compare pom file changes with NOTICE files: Nothing
> > suspicious found except for a minor (non-blocking) typo [1]
> >
> > Thanks for driving this effort, Sergey. :)
> >
> > [1] https://github.com/apache/flink-shaded/pull/126/files#r1409080162
> >
> > On Wed, Nov 29, 2023 at 10:25 AM Rui Fan <1996fan...@gmail.com> wrote:
> >
> >> Sorry, it's non-binding.
> >>
> >> On Wed, Nov 29, 2023 at 5:19 PM Rui Fan <1996fan...@gmail.com> wrote:
> >>
> >> > Thanks Matthias for the clarification!
> >> >
> >> > After I import the latest KEYS, it works fine.
> >> >
> >> > +1(binding)
> >> >
> >> > - Validated checksum hash
> >> > - Verified signature
> >> > - Verified that no binaries exist in the source archive
> >> > - Build the source with Maven and jdk8
> >> > - Verified licenses
> >> > - Verified web PRs, and left a comment
> >> >
> >> > Best,
> >> > Rui
> >> >
> >> > On Wed, Nov 29, 2023 at 5:05 PM Matthias Pohl
> >> >  wrote:
> >> >
> >> >> The key is the last key in the KEYS file. It's just having a
> different
> >> >> format with spaces being added (due to different gpg versions?): F752
> >> 9FAE
> >> >> 2481 1A5C 0DF3  CA74 1596 BBF0 7268 35D8
> >> >>
> >> >> On Wed, Nov 29, 2023 at 9:41 AM Rui Fan <1996fan...@gmail.com>
> wrote:
> >> >>
> >> >> > Hey Sergey,
> >> >> >
> >> >> > Thank you for driving this release.
> >> >> >
> >> >> > I try to check this signature, the whole key is
> >> >> > F7529FAE24811A5C0DF3CA741596BBF0726835D8,
> >> >> > it matches your 1596BBF0726835D8, but I cannot
> >> >> > find it from the Flink KEYS[1].
> >> >> >
> >> >> > Please correct me if my operation is wrong, thanks~
> >> >> >
> >> >> > [1] https://dist.apache.org/repos/dist/release/flink/KEYS
> >> >> >
> >> >> > Best,
> >> >> > Rui
> >> >> >
> >> >> >
> >> >> > On Wed, Nov 29, 2023 at 6:09 AM Sergey Nuyanzin <
> snuyan...@gmail.com
> >> >
> >> >> > wrote:
> >> >> >
> >> >> > > Hi everyone,
> >> >> > > Please review and vote on the release candidate #1 for the
> version
> >> >> 18.0,
> >> >> > as
> >> >> > > follows:
> >> >> > > [ ] +1, Approve the release
> >> >> > > [ ] -1, Do not approve the release (please provide specific
> >> comments)
> >> >> > >
> >> >> > >
> >> >> > > The complete staging area is available for your review, which
> >> >> includes:
> >> >> > > * JIRA release notes [1],
> >> >> > > * the official Apache source release to be deployed to
> >> >> dist.apache.org
> >> >> > > [2],
> >> >> > > which are signed with the key with fingerprint 1596BBF0726835D8
> >> [3],
> >> >> > > * all artifacts to be deployed to the Maven Central Repository
> [4],
> >> >> > > * source code tag "release-18.0-rc1" [5],
> >> >> > > * website pull request listing the new release [6].
> >> >> > >
> >> >> > > The vote will be open for at least 72 hours. It is adopted by
> >> majority
> >> >> > > approval, with at least 3 PMC affirmative votes.
> >> >> > >
> >> >> > > Thanks,
> >> >> > > Sergey
> >> >> > >
> >> >> > > [1]
> >> https://issues.apache.org/jira/projects/FLINK/versions/12353081
> >> >> > > [2]
> >> >> https://dist.apache.org/repos/dist/dev/flink/flink-shaded-18.0-rc1
> >> >> > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> >> >> > > [4]
> >> >> > >
> >> >>
> >> https://repository.apache.org/content/repositories/orgapacheflink-1676/
> >> >> > > [5]
> >> >> https://github.com/apache/flink-shaded/releases/tag/release-18.0-rc1
> >> >> > > [6] https://github.com/apache/flink-web/pull/701
> >> >> > >
> >> >> >
> >> >>
> >> >
> >>
> >
>
> --
> Best regards,
> Sergey
>

Re: [VOTE] FLIP-391: Deprecate RuntimeContext#getExecutionConfig

[Jing Ge]

+1(binding)
Thanks!

Best regards,
Jing

On Tue, Nov 28, 2023 at 1:00 PM weijie guo 
wrote:

> Thanks Junrui for driving this!
>
> +1(binding)
>
> Best regards,
>
> Weijie
>
>
> Zhanghao Chen  于2023年11月28日周二 17:29写道：
>
> > +1 (non-binding)
> >
> > Best,
> > Zhanghao Chen
> > 
> > From: Junrui Lee 
> > Sent: Tuesday, November 28, 2023 12:34
> > To: dev@flink.apache.org 
> > Subject: [VOTE] FLIP-391: Deprecate RuntimeContext#getExecutionConfig
> >
> > Hi everyone,
> >
> > Thank you to everyone for the feedback on FLIP-391: Deprecate
> > RuntimeContext#getExecutionConfig[1] which has been discussed in this
> > thread [2].
> >
> > I would like to start a vote for it. The vote will be open for at least
> 72
> > hours unless there is an objection or not enough votes.
> >
> > [1]
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=278465937
> > [2]https://lists.apache.org/thread/cv4x3372g5txw1j20r5l1vwlqrvjoqq5
> >
>

Re: [DISCUSS] Contribute Flink Doris Connector to the Flink community

[Jing Ge]

That sounds great! +1

Best regards
Jing

On Mon, Nov 27, 2023 at 3:38 AM Leonard Xu  wrote:

> Thanks wudi for kicking off the discussion,
>
> +1 for the idea from my side.
>
> A FLIP like Yun posted is required if no other objections.
>
> Best,
> Leonard
>
> > 2023年11月26日 下午6:22，wudi <676366...@qq.com.INVALID> 写道：
> >
> > Hi all,
> >
> > At present, Flink Connector and Flink's repository have been
> decoupled[1].
> > At the same time, the Flink-Doris-Connector[3] has been maintained based
> on the Apache Doris[2] community.
> > I think the Flink Doris Connector can be migrated to the Flink community
> because it It is part of Flink Connectors and can also expand the ecosystem
> of Flink Connectors.
> >
> > I volunteer to move this forward if I can.
> >
> > [1]
> https://cwiki.apache.org/confluence/display/FLINK/Externalized+Connector+development
> > [2] https://doris.apache.org/
> > [3] https://github.com/apache/doris-flink-connector
> >
> > --
> >
> > Brs,
> > di.wu
>
>

[jira] [Created] (FLINK-33654) deprecate DummyStreamExecutionEnvironment

[Jing Ge (Jira)]

Jing Ge created FLINK-33654:
---

 Summary: deprecate DummyStreamExecutionEnvironment
 Key: FLINK-33654
 URL: https://issues.apache.org/jira/browse/FLINK-33654
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Planner
Reporter: Jing Ge


Deprecate DummyStreamExecutionEnvironment first since it will take time to 
remove it.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Apache Flink Kafka Connectors v3.0.2, RC #1

[Jing Ge]

+1 (non-binding)

- verified signature and hash
- checked repo
- checked tag, BTW, the tag link at [5] should be
https://github.com/apache/flink-connector-kafka/releases/tag/v3.0.2-rc1
- verified source archives do not contains any binaries
- build source maven 3.8.6 and jdk11
- verified web PR

Best regards,
Jing

On Sat, Nov 25, 2023 at 6:44 AM Rui Fan <1996fan...@gmail.com> wrote:

> +1 (non-binding)
>
> - Validated checksum hash
> - Verified signature
> - Verified that no binaries exist in the source archive
> - Build the source with Maven and jdk8
> - Verified licenses
> - Verified web PRs
>
> Best,
> Rui
>
> On Sat, Nov 25, 2023 at 2:05 AM Tzu-Li (Gordon) Tai 
> wrote:
>
> > +1 (binding)
> >
> > - Verified signature and hashes
> > - Verified mvn dependency:tree for a typical user job jar [1]. When using
> > Flink 1.18.0, flink-connector-base is no longer getting bundled, and all
> > Flink dependencies resolve as 1.18.0 / provided.
> > - Submitting user job jar to local Flink 1.18.0 cluster works and job
> runs
> >
> > note: If running in the IDE, the flink-connector-base dependency is
> > explicitly required when using KafkaSource. Otherwise, if submitting an
> > uber jar, the flink-connector-base dependency should not be bundled as
> > it'll be provided by the Flink distribution and will already be on the
> > classpath.
> >
> > [1] mvn dependency:tree output
> > ```
> > [INFO] com.tzulitai:testing-kafka:jar:1.0-SNAPSHOT
> > [INFO] +- org.apache.flink:flink-streaming-java:jar:1.18.0:provided
> > [INFO] |  +- org.apache.flink:flink-core:jar:1.18.0:provided
> > [INFO] |  |  +- org.apache.flink:flink-annotations:jar:1.18.0:provided
> > [INFO] |  |  +- org.apache.flink:flink-metrics-core:jar:1.18.0:provided
> > [INFO] |  |  +- org.apache.flink:flink-shaded-asm-9:jar:9.5-17.0:provided
> > [INFO] |  |  +-
> > org.apache.flink:flink-shaded-jackson:jar:2.14.2-17.0:provided
> > [INFO] |  |  +- org.apache.commons:commons-lang3:jar:3.12.0:provided
> > [INFO] |  |  +- org.apache.commons:commons-text:jar:1.10.0:provided
> > [INFO] |  |  +- com.esotericsoftware.kryo:kryo:jar:2.24.0:provided
> > [INFO] |  |  |  +- com.esotericsoftware.minlog:minlog:jar:1.2:provided
> > [INFO] |  |  |  \- org.objenesis:objenesis:jar:2.1:provided
> > [INFO] |  |  +-
> commons-collections:commons-collections:jar:3.2.2:provided
> > [INFO] |  |  \- org.apache.commons:commons-compress:jar:1.21:provided
> > [INFO] |  +- org.apache.flink:flink-file-sink-common:jar:1.18.0:provided
> > [INFO] |  +- org.apache.flink:flink-runtime:jar:1.18.0:provided
> > [INFO] |  |  +- org.apache.flink:flink-rpc-core:jar:1.18.0:provided
> > [INFO] |  |  +-
> org.apache.flink:flink-rpc-akka-loader:jar:1.18.0:provided
> > [INFO] |  |  +-
> > org.apache.flink:flink-queryable-state-client-java:jar:1.18.0:provided
> > [INFO] |  |  +- org.apache.flink:flink-hadoop-fs:jar:1.18.0:provided
> > [INFO] |  |  +- commons-io:commons-io:jar:2.11.0:provided
> > [INFO] |  |  +-
> > org.apache.flink:flink-shaded-netty:jar:4.1.91.Final-17.0:provided
> > [INFO] |  |  +-
> > org.apache.flink:flink-shaded-zookeeper-3:jar:3.7.1-17.0:provided
> > [INFO] |  |  +- org.javassist:javassist:jar:3.24.0-GA:provided
> > [INFO] |  |  +- org.xerial.snappy:snappy-java:jar:1.1.10.4:runtime
> > [INFO] |  |  \- org.lz4:lz4-java:jar:1.8.0:runtime
> > [INFO] |  +- org.apache.flink:flink-java:jar:1.18.0:provided
> > [INFO] |  |  \- com.twitter:chill-java:jar:0.7.6:provided
> > [INFO] |  +-
> org.apache.flink:flink-shaded-guava:jar:31.1-jre-17.0:provided
> > [INFO] |  +- org.apache.commons:commons-math3:jar:3.6.1:provided
> > [INFO] |  +- org.slf4j:slf4j-api:jar:1.7.36:runtime
> > [INFO] |  \- com.google.code.findbugs:jsr305:jar:1.3.9:provided
> > [INFO] +- org.apache.flink:flink-clients:jar:1.18.0:provided
> > [INFO] |  +- org.apache.flink:flink-optimizer:jar:1.18.0:provided
> > [INFO] |  \- commons-cli:commons-cli:jar:1.5.0:provided
> > [INFO] +- org.apache.flink:flink-connector-kafka:jar:3.0.2-1.18:compile
> > [INFO] |  +- org.apache.kafka:kafka-clients:jar:3.2.3:compile
> > [INFO] |  |  \- com.github.luben:zstd-jni:jar:1.5.2-1:runtime
> > [INFO] |  +- com.fasterxml.jackson.core:jackson-core:jar:2.15.2:compile
> > [INFO] |  +-
> com.fasterxml.jackson.core:jackson-databind:jar:2.15.2:compile
> > [INFO] |  |  \-
> > com.fasterxml.jackson.core:jackson-annotations:jar:2.15.2:compile
> > [INFO] |  +-
> > com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.15.2:compile
> > [INFO] |  \-
> > com.fasterxml.jackson.datatype:jackson-datatype-jdk8:jar:2.15.2:compile
> > [INFO] +- org.apache.logging.log4j:log4j-slf4j-impl:jar:2.17.1:runtime
> > [INFO] +- org.apache.logging.log4j:log4j-api:jar:2.17.1:runtime
> > [INFO] \- org.apache.logging.log4j:log4j-core:jar:2.17.1:runtime
> > ```
> >
> > On Fri, Nov 24, 2023 at 9:19 AM Tzu-Li (Gordon) Tai  >
> > wrote:
> >
> > > Hi everyone,
> > >
> > > Please review and vote on release candidate #1 for version 3.0.2 of the
> > > Apache Flink Kafka Connector, as 

[jira] [Created] (FLINK-33651) Update the Chinese version checkpointing doc in fault tolerance

[Jing Ge (Jira)]

Jing Ge created FLINK-33651:
---

 Summary: Update the Chinese version checkpointing doc in fault 
tolerance
 Key: FLINK-33651
 URL: https://issues.apache.org/jira/browse/FLINK-33651
 Project: Flink
  Issue Type: Improvement
  Components: Documentation, Runtime / Checkpointing
Reporter: Jing Ge
Assignee: Jing Ge


the Chinese doc missed some items compared to the English doc[1], would you 
mind adding them together? Such as:
 * checkpoint storage
 * unaligned checkpoints:
 * checkpoints with finished tasks:

[1] 
[https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/checkpointing/#enabling-and-configuring-checkpointing]

 

Reference: https://github.com/apache/flink/pull/23795#discussion_r1404855432



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33648) table usfs doc in Chinese is out-of-date

[Jing Ge (Jira)]

Jing Ge created FLINK-33648:
---

 Summary: table usfs doc in Chinese is out-of-date
 Key: FLINK-33648
 URL: https://issues.apache.org/jira/browse/FLINK-33648
 Project: Flink
  Issue Type: Improvement
  Components: Documentation, Table SQL / API
Affects Versions: 1.18.0
Reporter: Jing Ge


The English version 
[https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/functions/udfs/]
 is very different to the Chinese version 
[https://nightlies.apache.org/flink/flink-docs-master/zh/docs/dev/table/functions/udfs/]

 

Not only the content but also the code examples.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Release 1.16.3, release candidate #1

[Jing Ge]

+1(not-binding)

- verified signatures hash
- verified hash
- checked the tag
- checked the repo
- verified there are no binaries in the source archive
- built from sources

Best regards,
Jing

On Thu, Nov 23, 2023 at 7:44 PM Feng Jin  wrote:

> +1(non-binding)
>
> - verified signatures and hashsums
> - Verified there are no binaries in the source archive
> - Built Flink from sources
> - Started local standalone cluster, submitted sql job using sql-client
>
> Best,
> Feng
>
> On Thu, Nov 23, 2023 at 11:00 PM Hang Ruan  wrote:
>
> > +1 (non-binding)
> >
> > - verified signatures
> > - verified hashsums
> > - Verified there are no binaries in the source archive
> > - reviewed the Release Note
> > - reviewed the web PR
> >
> > Best,
> > Hang
> >
> > Jiabao Sun  于2023年11月23日周四 22:15写道：
> >
> > > Thanks for driving this release.
> > >
> > > +1 (non-binding)
> > >
> > > - Checked the tag in git
> > > - Verified signatures and hashsums
> > > - Verified there are no binaries in the source archive
> > > - Built Flink from sources
> > >
> > > Best,
> > > Jiabao
> > >
> > > > 2023年11月23日 21:55，Sergey Nuyanzin  写道：
> > > >
> > > > +1 (non-binding)
> > > >
> > > > - Downloaded artifacts
> > > > - Built Flink from sources
> > > > - Verified checksums & signatures
> > > > - Verified pom/NOTICE files
> > > > - reviewed the web PR
> > > >
> > > > On Thu, Nov 23, 2023 at 1:28 PM Leonard Xu 
> wrote:
> > > >
> > > >> +1 (binding)
> > > >>
> > > >> - verified signatures
> > > >> - verified hashsums
> > > >> - checked that all POM files point to the same version 1.16.3
> > > >> - started SQL Client, used MySQL CDC connector to read changelog
> from
> > > >> database , the result is expected
> > > >> - reviewed the web PR, left minor comment
> > > >>
> > > >> Best,
> > > >> Leonard
> > > >>
> > > >>> 2023年11月21日 下午8:56，Matthias Pohl 
> > 写道：
> > > >>>
> > > >>> +1 (binding)
> > > >>>
> > > >>> * Downloaded artifacts
> > > >>> * Built Flink from sources
> > > >>> * Verified SHA512 checksums & GPG signatures
> > > >>> * Compared checkout with provided sources
> > > >>> * Verified pom file versions
> > > >>> * Went over NOTICE/pom file changes without finding anything
> > suspicious
> > > >>> * Deployed standalone session cluster and ran WordCount example in
> > > batch
> > > >>> and streaming: Nothing suspicious in log files found
> > > >>> * Verified Java version of uploaded binaries
> > > >>>
> > > >>> Thanks for wrapping 1.16 up. :)
> > > >>>
> > > >>> On Tue, Nov 21, 2023 at 4:55 AM Rui Fan <1996fan...@gmail.com>
> > wrote:
> > > >>>
> > >  +1 (non-binding)
> > > 
> > >  Verified based on this wiki[1].
> > > 
> > >  - Verified signatures and sha512
> > >  - The source archives do not contain any binaries
> > >  - Build the source with Maven 3 and java8 (Checked the license as
> > > well)
> > >  - bin/start-cluster.sh with java8, it works fine and no any
> > unexpected
> > > >> LOG
> > >  - Ran demo, it's fine:  bin/flink run
> > >  examples/streaming/StateMachineExample.jar
> > > 
> > >  [1]
> > > 
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/Verifying+a+Flink+Release
> > > 
> > >  Best,
> > >  Rui
> > > 
> > >  On Fri, Nov 17, 2023 at 11:52 AM Yun Tang 
> wrote:
> > > 
> > > > +1 (non-binding)
> > > >
> > > >
> > > > *   Verified signatures
> > > > *   Build from source code, and it looks good
> > > > *   Verified that jar packages are built with maven-3.2.5 and
> JDK8
> > > > *   Reviewed the flink-web PR
> > > > *   Start a local standalone cluster and submit examples
> > > >
> > > > Best
> > > > Yun Tang
> > > > 
> > > > From: Rui Fan <1996fan...@gmail.com>
> > > > Sent: Monday, November 13, 2023 18:20
> > > > To: dev 
> > > > Subject: [VOTE] Release 1.16.3, release candidate #1
> > > >
> > > > Hi everyone,
> > > >
> > > > Please review and vote on the release candidate #1 for the
> version
> > >  1.16.3,
> > > >
> > > > as follows:
> > > >
> > > > [ ] +1, Approve the release
> > > >
> > > > [ ] -1, Do not approve the release (please provide specific
> > comments)
> > > >
> > > >
> > > > The complete staging area is available for your review, which
> > > includes:
> > > > * JIRA release notes [1],
> > > >
> > > > * the official Apache source release and binary convenience
> > releases
> > > to
> > >  be
> > > > deployed to dist.apache.org [2], which are signed with the key
> > with
> > > > fingerprint B2D64016B940A7E0B9B72E0D7D0528B28037D8BC [3],
> > > >
> > > > * all artifacts to be deployed to the Maven Central Repository
> [4],
> > > >
> > > > * source code tag "release-1.16.3-rc1" [5],
> > > >
> > > > * website pull request listing the new release and adding
> > > announcement
> > >  blog
> > > > post [6].
> > >