Re: [VOTE] FLIP-379: Dynamic source parallelism inference for batch jobs

[Lijie Wang]

+1 (binding)

Best,
Lijie

Zhu Zhu  于2023年11月30日周四 13:13写道：

> +1
>
> Thanks,
> Zhu
>
> Xia Sun  于2023年11月30日周四 11:41写道：
>
> > Hi everyone,
> >
> > I'd like to start a vote on FLIP-379: Dynamic source parallelism
> inference
> > for batch jobs[1] which has been discussed in this thread [2].
> >
> > The vote will be open for at least 72 hours unless there is an objection
> or
> > not enough votes.
> >
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-379%3A+Dynamic+source+parallelism+inference+for+batch+jobs
> > [2] https://lists.apache.org/thread/ocftkqy5d2x4n58wzprgm5qqrzzkbmb8
> >
> >
> > Best Regards,
> > Xia
> >
>

[DISCUSS][2.0] FLIP-382: Unify the Provision of Diverse Metadata for Context-like APIs

[Wencong Liu]

Hi devs,

I would like to start a discussion on FLIP-382: Unify the Provision
of Diverse Metadata for Context-like APIs [1].

In the Flink project, the context-like APIs are interfaces annotated by
@Public and supply runtime metadata and functionalities to its modules and
components. RuntimeContext is such an interface with 27 methods for
accessing metadata and framework functionalities. Currently, any
addition of metadata requires updating the RuntimeContext interface
and all 12 of its implementation classes, leading to high code
maintenance costs. To improve this, we propose to a categorize all
metadata into some metadata classes and provide it through dedicated
methods in context-like APIs. The newly provided metadata in context-like
API will only require to update the metadata classes, not context-like API
itself or it's implementations.

Looking forward to your feedback.

[1] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-382%3A+Unify+the+Provision+of+Diverse+Metadata+for+Context-like+APIs

Best regards,
Wencong Liu

Re: [ANNOUNCE] Experimental Java 21 support now available on master

[Sergey Nuyanzin]

Thanks Yun Tang

One question to clarify: since the scala version was also bumped for java
17, shouldn't there be a similar task for java 17?

On Thu, Nov 30, 2023 at 3:43 AM Yun Tang  wrote:

> Hi Sergey,
>
> You can leverage all tests extending SnapshotMigrationTestBase[1] to
> verify the logic. I believe all binary _metadata existing in the resources
> folder[2] were built by JDK8.
>
> I also create a ticket FLINK-33699[3] to track this.
>
> [1]
> https://github.com/apache/flink/blob/master/flink-tests/src/test/java/org/apache/flink/test/checkpointing/utils/SnapshotMigrationTestBase.java
> [2]
> https://github.com/apache/flink/tree/master/flink-tests/src/test/resources
> [3] https://issues.apache.org/jira/browse/FLINK-33699
>
> Best
> Yun Tang
> 
> From: Sergey Nuyanzin 
> Sent: Wednesday, November 29, 2023 22:56
> To: dev@flink.apache.org 
> Subject: Re: [ANNOUNCE] Experimental Java 21 support now available on
> master
>
> thanks for the response
>
>
> >I feel doubt about the conclusion that "don't try to load a savepoint from
> a Java 8/11/17 build due to bumping to scala-2.12.18", since the
> snapshotted state (operator/keyed state-backend),  and most key/value
> serializer snapshots are generated by pure-java code.
> >The only left part is that the developer uses scala UDF or scala types for
> key/value types. However, since all user-facing scala APIs have been
> deprecated, I don't think we have so many cases. Maybe we can give
> descriptions without such strong suggestions.
>
> That is the area where I feel I lack the knowledge to answer this
> precisely.
> My assumption was that statement about Java 21 regarding this should be
> similar to Java 17 which is almost same [1]
> Sorry for the inaccuracy
> Based on your statements I agree that the conclusion could be more relaxed.
>
> I'm curious whether there are some tests or anything which could clarify
> this?
>
> [1] https://lists.apache.org/thread/mz0m6wqjmqy8htob3w4469pjbg9305do
>
> On Wed, Nov 29, 2023 at 12:25 PM Yun Tang  wrote:
>
> > Thanks Sergey for the great work.
> >
> > I feel doubt about the conclusion that "don't try to load a savepoint
> from
> > a Java 8/11/17 build due to bummping to scala-2.12.18", since the
> > snapshotted state (operator/keyed state-backend),  and most key/value
> > serializer snapshots are generated by pure-java code. The only left part
> is
> > that the developer uses scala UDF or scala types for key/value types.
> > However, since all user-facing scala APIs have been deprecated [1], I
> don't
> > think we have so many cases. Maybe we can give descriptions without such
> > strong suggestions.
> >
> > Please correct me if I am wrong.
> >
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-29740
> >
> > Best
> > Yun Tang
> >
> > 
> > From: Rui Fan <1996fan...@gmail.com>
> > Sent: Wednesday, November 29, 2023 16:43
> > To: dev@flink.apache.org 
> > Subject: Re: [ANNOUNCE] Experimental Java 21 support now available on
> > master
> >
> > Thanks Sergey for the great work!
> >
> > Best,
> > Rui
> >
> > On Wed, Nov 29, 2023 at 4:42 PM Leonard Xu  wrote:
> >
> > > Cool !
> > >
> > > Thanks Sergey for the great effort and all involved.
> > >
> > >
> > > Best,
> > > Leonard
> > >
> > > > 2023年11月29日 下午4:31，Swapnal Varma  写道：
> > > >
> > > > Congratulations Sergey, and everyone involved!
> > > >
> > > > Excited to work with and on this!
> > > >
> > > > Best,
> > > > Swapnal
> > > >
> > > >
> > > > On Wed, 29 Nov 2023, 13:58 Sergey Nuyanzin, 
> > wrote:
> > > >
> > > >> The master branch now builds and runs with Java 21 out-of-the-box.
> > > >>
> > > >> Notes:
> > > >> - a nightly cron build was set up.
> > > >> - In Java 21 builds, Scala is being bumped to 2.12.18
> > > >> which causes incompatibilities within Flink;
> > > >> i.e. don't try to load a savepoint from a Java 8/11/17 build
> > > >> - All the tests that are being skipped on Java 11/17
> > > >> are also skipped on Java 21.
> > > >>
> > > >> Huge shout-out to everyone participating
> > > >> in review of my Java 21 related PRs
> > > >>
> > > >> If you run into any issues, please report it in FLINK-33163
> > > >>  .
> > > >>
> > > >> --
> > > >> Best regards,
> > > >> Sergey
> > > >>
> > >
> > >
> >
>
>
> --
> Best regards,
> Sergey
>


-- 
Best regards,
Sergey

Re: [VOTE] Release flink-connector-aws, v4.2.0 release candidate #1

[Samrat Deb]

Hi danny ,

+1 (non binding)

Release notes look good
- Signatures and checksums verified
- pom versions checked
- Source build and tests verified

Bests,
Samrat

On Wed, 29 Nov 2023 at 12:11 PM, Leonard Xu  wrote:

> Thanks Danny for driving this, sorry for late verification.
>
> +1 (binding)
>
> - built from source code succeeded
> - verified signatures
> - verified hashsums
> - checked the contents contains jar and pom files in apache repo
> - checked Github release tag
> - reviewed the web PR with minor comment
>
> Best,
> Leonard
>
>
>
> > 2023年11月27日 上午1:13，Ahmed Hamdy  写道：
> >
> > Hi Danny
> > +1 (non-binding)
> >
> >
> > - Verified signatures and checksums
> > - verified no binaries exists in archive
> > - built source
> > - reviewed web PR
> > - Ran E2E example with Kinesis, Firehose & DynamoDB datastream
> connectors.
> >
> > Best Regards
> > Ahmed Hamdy
> >
> >
> > On Fri, 24 Nov 2023 at 08:38, Martijn Visser 
> > wrote:
> >
> >> Hi Danny,
> >>
> >> Thanks for driving this.
> >>
> >> +1 (binding)
> >>
> >> - Validated hashes
> >> - Verified signature
> >> - Verified that no binaries exist in the source archive
> >> - Build the source with Maven
> >> - Verified licenses
> >> - Verified web PRs
> >>
> >> On Mon, Nov 20, 2023 at 12:29 PM Jiabao Sun
> >>  wrote:
> >>>
> >>> Thanks Danny for driving the release,
> >>>
> >>> +1 (non-binding)
> >>>
> >>> - built from source code succeeded
> >>> - verified signatures
> >>> - verified hashsums
> >>> - checked release notes
> >>>
> >>> Best,
> >>> Jiabao
> >>>
> >>>
>  2023年11月20日 19:11，Danny Cranmer  写道：
> 
>  Hello all,
> 
>  +1 (binding).
> 
>  - Release notes look good
>  - Signatures and checksums match
>  - There are no binaries in the source archive
>  - pom versions are correct
>  - Tag is present in Github
>  - CI passes against FLink 1.17 and 1.18
>  - Source build and tests pass
> 
>  Thanks,
>  Danny
> 
>  On Wed, Nov 1, 2023 at 1:15 AM mystic lama 
> >> wrote:
> 
> > +1 (non-binding)
> >
> > - validated shasum
> > - verified build
> >  - Java 8   - build good and all test cases pass
> >  - Java 11 - build good and all test cases pass
> >
> > Observations: got test failures with Java 17, something to look for
> in
> > future
> >
> > On Tue, 31 Oct 2023 at 08:42, Danny Cranmer  >
> > wrote:
> >
> >> Hi everyone,
> >>
> >> Please review and vote on release candidate #1 for the version
> >> 4.2.0, as
> >> follows:
> >> [ ] +1, Approve the release
> >> [ ] -1, Do not approve the release (please provide specific
> comments)
> >>
> >> The complete staging area is available for your review, which
> >> includes:
> >> * JIRA release notes [1],
> >> * the official Apache source release to be deployed to
> >> dist.apache.org
> >> [2],
> >> which are signed with the key with fingerprint
> >> 0F79F2AFB2351BC29678544591F9C1EC125FD8DB [3],
> >> * all artifacts to be deployed to the Maven Central Repository [4],
> >> * source code tag v4.2.0-rc1 [5],
> >> * website pull request listing the new release [6].
> >> * A link to the CI run on the release tag [7]
> >>
> >> The vote will be open for at least 72 hours. It is adopted by
> >> majority
> >> approval, with at least 3 PMC affirmative votes.
> >>
> >> Thanks,
> >> Danny
> >>
> >> [1]
> >>
> >>
> >
> >>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353011
> >> [2]
> >>
> >
> >>
> https://dist.apache.org/repos/dist/dev/flink/flink-connector-aws-4.2.0-rc1
> >> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> >> [4]
> >>
> >> https://repository.apache.org/content/repositories/orgapacheflink-1665/
> >> [5]
> >
> https://github.com/apache/flink-connector-aws/releases/tag/v4.2.0-rc1
> >> [6] https://github.com/apache/flink-web/pull/693
> >> [7]
> >
> https://github.com/apache/flink-connector-aws/actions/runs/6707962074
> >>
> >
> >>>
> >>
>
>

Re: [VOTE] FLIP-379: Dynamic source parallelism inference for batch jobs

[Zhu Zhu]

+1

Thanks,
Zhu

Xia Sun  于2023年11月30日周四 11:41写道：

> Hi everyone,
>
> I'd like to start a vote on FLIP-379: Dynamic source parallelism inference
> for batch jobs[1] which has been discussed in this thread [2].
>
> The vote will be open for at least 72 hours unless there is an objection or
> not enough votes.
>
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-379%3A+Dynamic+source+parallelism+inference+for+batch+jobs
> [2] https://lists.apache.org/thread/ocftkqy5d2x4n58wzprgm5qqrzzkbmb8
>
>
> Best Regards,
> Xia
>

[VOTE] FLIP-379: Dynamic source parallelism inference for batch jobs

[Xia Sun]

Hi everyone,

I'd like to start a vote on FLIP-379: Dynamic source parallelism inference
for batch jobs[1] which has been discussed in this thread [2].

The vote will be open for at least 72 hours unless there is an objection or
not enough votes.


[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-379%3A+Dynamic+source+parallelism+inference+for+batch+jobs
[2] https://lists.apache.org/thread/ocftkqy5d2x4n58wzprgm5qqrzzkbmb8


Best Regards,
Xia

[jira] [Created] (FLINK-33700) CustomSSLEngineProvider supports SHA-256

[Bo Cui (Jira)]

Bo Cui created FLINK-33700:
--

 Summary: CustomSSLEngineProvider supports SHA-256
 Key: FLINK-33700
 URL: https://issues.apache.org/jira/browse/FLINK-33700
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / RPC
Affects Versions: 1.19.0
Reporter: Bo Cui


The algorithm of CustomSSLEngineProvider supports only SHA1.
https://github.com/apache/flink/blob/72654384686d127172b48b0071ea7656b16e9134/flink-rpc/flink-rpc-akka/src/main/java/org/apache/flink/runtime/rpc/pekko/CustomSSLEngineProvider.java#L58C1-L59C1



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33699) Verify the snapshot migration on Java21

[Yun Tang (Jira)]

Yun Tang created FLINK-33699:


 Summary: Verify the snapshot migration on Java21
 Key: FLINK-33699
 URL: https://issues.apache.org/jira/browse/FLINK-33699
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Checkpointing
Reporter: Yun Tang


In Java 21 builds, Scala is being bumped to 2.12.18, which causes 
incompatibilities within Flink.

This could affect loading savepoints from a Java 8/11/17 build. We already have 
tests extending {{SnapshotMigrationTestBase}} to verify the logic of migrating 
snapshots generated by the older Flink version. I think we can also introduce 
similar tests to verify the logic across different Java versions.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] Experimental Java 21 support now available on master

[Yun Tang]

Hi Sergey,

You can leverage all tests extending SnapshotMigrationTestBase[1] to verify the 
logic. I believe all binary _metadata existing in the resources folder[2] were 
built by JDK8.

I also create a ticket FLINK-33699[3] to track this.

[1] 
https://github.com/apache/flink/blob/master/flink-tests/src/test/java/org/apache/flink/test/checkpointing/utils/SnapshotMigrationTestBase.java
[2] https://github.com/apache/flink/tree/master/flink-tests/src/test/resources
[3] https://issues.apache.org/jira/browse/FLINK-33699

Best
Yun Tang

From: Sergey Nuyanzin 
Sent: Wednesday, November 29, 2023 22:56
To: dev@flink.apache.org 
Subject: Re: [ANNOUNCE] Experimental Java 21 support now available on master

thanks for the response


>I feel doubt about the conclusion that "don't try to load a savepoint from
a Java 8/11/17 build due to bumping to scala-2.12.18", since the
snapshotted state (operator/keyed state-backend),  and most key/value
serializer snapshots are generated by pure-java code.
>The only left part is that the developer uses scala UDF or scala types for
key/value types. However, since all user-facing scala APIs have been
deprecated, I don't think we have so many cases. Maybe we can give
descriptions without such strong suggestions.

That is the area where I feel I lack the knowledge to answer this precisely.
My assumption was that statement about Java 21 regarding this should be
similar to Java 17 which is almost same [1]
Sorry for the inaccuracy
Based on your statements I agree that the conclusion could be more relaxed.

I'm curious whether there are some tests or anything which could clarify
this?

[1] https://lists.apache.org/thread/mz0m6wqjmqy8htob3w4469pjbg9305do

On Wed, Nov 29, 2023 at 12:25 PM Yun Tang  wrote:

> Thanks Sergey for the great work.
>
> I feel doubt about the conclusion that "don't try to load a savepoint from
> a Java 8/11/17 build due to bummping to scala-2.12.18", since the
> snapshotted state (operator/keyed state-backend),  and most key/value
> serializer snapshots are generated by pure-java code. The only left part is
> that the developer uses scala UDF or scala types for key/value types.
> However, since all user-facing scala APIs have been deprecated [1], I don't
> think we have so many cases. Maybe we can give descriptions without such
> strong suggestions.
>
> Please correct me if I am wrong.
>
>
> [1] https://issues.apache.org/jira/browse/FLINK-29740
>
> Best
> Yun Tang
>
> 
> From: Rui Fan <1996fan...@gmail.com>
> Sent: Wednesday, November 29, 2023 16:43
> To: dev@flink.apache.org 
> Subject: Re: [ANNOUNCE] Experimental Java 21 support now available on
> master
>
> Thanks Sergey for the great work!
>
> Best,
> Rui
>
> On Wed, Nov 29, 2023 at 4:42 PM Leonard Xu  wrote:
>
> > Cool !
> >
> > Thanks Sergey for the great effort and all involved.
> >
> >
> > Best,
> > Leonard
> >
> > > 2023年11月29日 下午4:31，Swapnal Varma  写道：
> > >
> > > Congratulations Sergey, and everyone involved!
> > >
> > > Excited to work with and on this!
> > >
> > > Best,
> > > Swapnal
> > >
> > >
> > > On Wed, 29 Nov 2023, 13:58 Sergey Nuyanzin, 
> wrote:
> > >
> > >> The master branch now builds and runs with Java 21 out-of-the-box.
> > >>
> > >> Notes:
> > >> - a nightly cron build was set up.
> > >> - In Java 21 builds, Scala is being bumped to 2.12.18
> > >> which causes incompatibilities within Flink;
> > >> i.e. don't try to load a savepoint from a Java 8/11/17 build
> > >> - All the tests that are being skipped on Java 11/17
> > >> are also skipped on Java 21.
> > >>
> > >> Huge shout-out to everyone participating
> > >> in review of my Java 21 related PRs
> > >>
> > >> If you run into any issues, please report it in FLINK-33163
> > >>  .
> > >>
> > >> --
> > >> Best regards,
> > >> Sergey
> > >>
> >
> >
>


--
Best regards,
Sergey

[ANNOUNCE] Apache Flink 1.16.3 released

[Rui Fan]

The Apache Flink community is very happy to announce the release of
Apache Flink 1.16.3, which is the
third bugfix release for the Apache Flink 1.16 series.



Apache Flink® is an open-source stream processing framework for
distributed, high-performing, always-available, and accurate data
streaming applications.



The release is available for download at:

https://flink.apache.org/downloads.html



Please check out the release blog post for an overview of the
improvements for this bugfix release:

https://flink.apache.org/2023/11/29/apache-flink-1.16.3-release-announcement/



The full release notes are available in Jira:

https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12353259



We would like to thank all contributors of the Apache Flink community
who made this release possible!



Feel free to reach out to the release managers (or respond to this
thread) with feedback on the release process. Our goal is to
constantly improve the release process. Feedback on what could be
improved or things that didn't go so well are appreciated.



Regards,

Release Manager

[jira] [Created] (FLINK-33698) Fix the backoff time calculation in ExponentialBackoffDelayRetryStrategy

[xiangyu feng (Jira)]

xiangyu feng created FLINK-33698:


 Summary: Fix the backoff time calculation in 
ExponentialBackoffDelayRetryStrategy
 Key: FLINK-33698
 URL: https://issues.apache.org/jira/browse/FLINK-33698
 Project: Flink
  Issue Type: Bug
  Components: API / DataStream
Reporter: xiangyu feng


The backoff time calculation in `ExponentialBackoffDelayRetryStrategy` should 
consider currentAttempts. 

 

Current Version:
{code:java}
@Override
public long getBackoffTimeMillis(int currentAttempts) {
if (currentAttempts <= 1) {
// equivalent to initial delay
return lastRetryDelay;
}
long backoff = Math.min((long) (lastRetryDelay * multiplier), 
maxRetryDelay);
this.lastRetryDelay = backoff;
return backoff;
} {code}
Fixed Version:
{code:java}
@Override
public long getBackoffTimeMillis(int currentAttempts) {
if (currentAttempts <= 1) {
// equivalent to initial delay
return initialDelay;
}
long backoff =
Math.min(
(long) (initialDelay * Math.pow(multiplier, currentAttempts 
- 1)),
maxRetryDelay);
return backoff;
} {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33696) FLIP-385: Add OpenTelemetryTraceReporter and OpenTelemetryMetricReporter

[Piotr Nowojski (Jira)]

Piotr Nowojski created FLINK-33696:
--

 Summary: FLIP-385: Add OpenTelemetryTraceReporter and 
OpenTelemetryMetricReporter
 Key: FLINK-33696
 URL: https://issues.apache.org/jira/browse/FLINK-33696
 Project: Flink
  Issue Type: New Feature
  Components: Runtime / Metrics
Reporter: Piotr Nowojski
Assignee: Piotr Nowojski
 Fix For: 1.19.0


h1. Motivation

[FLIP-384|https://cwiki.apache.org/confluence/display/FLINK/FLIP-384%3A+Introduce+TraceReporter+and+use+it+to+create+checkpointing+and+recovery+traces]
 is adding TraceReporter interface. However with 
[FLIP-384|https://cwiki.apache.org/confluence/display/FLINK/FLIP-384%3A+Introduce+TraceReporter+and+use+it+to+create+checkpointing+and+recovery+traces]
 alone, Log4jTraceReporter would be the only available implementation of 
TraceReporter interface, which is not very helpful.

In this FLIP I’m proposing to contribute both MetricExporter and TraceReporter 
implementation using OpenTelemetry.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33697) FLIP-386: Support adding custom metrics in Recovery Spans

[Piotr Nowojski (Jira)]

Piotr Nowojski created FLINK-33697:
--

 Summary: FLIP-386: Support adding custom metrics in Recovery Spans
 Key: FLINK-33697
 URL: https://issues.apache.org/jira/browse/FLINK-33697
 Project: Flink
  Issue Type: New Feature
  Components: Runtime / Metrics, Runtime / State Backends
Reporter: Piotr Nowojski
Assignee: Piotr Nowojski
 Fix For: 1.19.0


h1. Motivation

FLIP-386 is building on top of 
[FLIP-384|https://cwiki.apache.org/confluence/display/FLINK/FLIP-384%3A+Introduce+TraceReporter+and+use+it+to+create+checkpointing+and+recovery+traces].
 The intention here is to add a capability for state backends to attach custom 
attributes during recovery to recovery spans. For example 
RocksDBIncrementalRestoreOperation could report both remote download time and 
time to actually clip/ingest the RocksDB instances after rescaling.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-33695) FLIP-384: Introduce TraceReporter and use it to create checkpointing and recovery traces

[Piotr Nowojski (Jira)]

Piotr Nowojski created FLINK-33695:
--

 Summary: FLIP-384: Introduce TraceReporter and use it to create 
checkpointing and recovery traces
 Key: FLINK-33695
 URL: https://issues.apache.org/jira/browse/FLINK-33695
 Project: Flink
  Issue Type: New Feature
  Components: Runtime / Checkpointing, Runtime / Metrics
Reporter: Piotr Nowojski
Assignee: Piotr Nowojski
 Fix For: 1.19.0






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] FLIP-386: Support adding custom metrics in Recovery Spans

[Piotr Nowojski]

Thank you all for voting, I will post the results soon!

Best,
Piotrek

śr., 29 lis 2023 o 10:47 Stefan Richter 
napisał(a):

> +1 (binding)
>
> Best,
> Stefan
>
>
> > On 23. Nov 2023, at 09:49, Roman Khachatryan  wrote:
> >
> > +1 (binding)
> >
> > Regards,
> > Roman
> >
> >
> > On Wed, Nov 22, 2023 at 12:55 PM Rui Fan <1996fan...@gmail.com  1996fan...@gmail.com>> wrote:
> >
> >> +1(binding)
> >>
> >> Thanks for driving this  proposal!
> >>
> >> Best,
> >> Rui
> >>
> >> On Wed, Nov 22, 2023 at 7:44 PM Piotr Nowojski 
> >> wrote:
> >>
> >>> Hi All,
> >>>
> >>> I'd like to start a vote on the FLIP-386: Support adding custom metrics
> >> in
> >>> Recovery Spans [1]. The discussion thread is here [2].
> >>>
> >>> The vote will be open for at least 72 hours unless there is an
> objection
> >> or
> >>> not enough votes.
> >>>
> >>> [1]
> https://www.google.com/url?q=https://cwiki.apache.org/confluence/x/VAuZE=gmail-imap=170133425800=AOvVaw3oQ0TtRfjerwf5nJ0hQkPA
> >>> [2]
> https://www.google.com/url?q=https://lists.apache.org/thread/zt4ykyhv6cco83j9hjngn52b1oprj1tv=gmail-imap=170133425800=AOvVaw0LqLxSwykk0rLpBTXAB0WA
>
>

[RESULT][VOTE] FLIP-386: Support adding custom metrics in Recovery Spans

[Piotr Nowojski]

I'm pleased to announce that FLIP-386 has been accepted.

Binding votes +1:
- Rui Fan
- Roman Khachatryan
- Stefan Richter

There were no votes against.

Best,
Piotrek

[RESULT][VOTE] FLIP-385: Add OpenTelemetryTraceReporter and OpenTelemetryMetricReporter

[Piotr Nowojski]

I'm pleased to announce that FLIP-385 has been accepted.

Binding votes +1:
- Rui Fan
- Hangxiang Yu
- Roman Khachatryan
- Jing Ge
- Stefan Richter

There were no votes against.

Best,
Piotrek

Re: [VOTE] FLIP-385: Add OpenTelemetryTraceReporter and OpenTelemetryMetricReporter

[Piotr Nowojski]

Thank you all for voting, I will post the results soon.

Best,
Piotrek

śr., 29 lis 2023 o 10:47 Stefan Richter 
napisał(a):

> +1 (binding)
>
> Best,
> Stefan
>
>
> > On 22. Nov 2023, at 12:34, Jing Ge  wrote:
> >
> > +1(binding)
> > Thanks!
> >
> > Best Regards,
> > Jing
> >
> > On Wed, Nov 22, 2023 at 11:21 AM Roman Khachatryan  > wrote:
> >
> >> +1 (binding)
> >>
> >> Regards,
> >> Roman
> >>
> >> On Wed, Nov 22, 2023, 7:30 AM Hangxiang Yu  wrote:
> >>
> >>> +1(binding)
> >>>
> >>> On Wed, Nov 22, 2023 at 10:29 AM Rui Fan <1996fan...@gmail.com> wrote:
> >>>
>  +1(binding)
> 
>  Best,
>  Rui
> 
>  On Wed, Nov 22, 2023 at 1:20 AM Piotr Nowojski 
>  wrote:
> 
> > Hi All,
> >
> > I'd like to start a vote on the FLIP-385: Add
> >>> OpenTelemetryTraceReporter
> > and OpenTelemetryMetricReporter [1]. The discussion thread is here
> >> [2].
> >
> > The vote will be open for at least 72 hours unless there is an
> >>> objection
>  or
> > not enough votes.
> >
> > [1]
> https://www.google.com/url?q=https://cwiki.apache.org/confluence/x/UAuZE=gmail-imap=170125778900=AOvVaw2pEEbI5Qe9cEDMKKf77bb-
> > [2]
> https://www.google.com/url?q=https://lists.apache.org/thread/1rqp8czz8wnplpzgn8m4qmzvf14lyx0k=gmail-imap=170125778900=AOvVaw1prt9Abiq4GEXW4zLSIEaW
> >
> >
> > Best,
> > Piotrek
> >
> 
> >>>
> >>>
> >>> --
> >>> Best,
> >>> Hangxiang.
>
>

[RESULT][VOTE] FLIP-384: Introduce TraceReporter and use it to create checkpointing and recovery traces

[Piotr Nowojski]

I'm pleased to announce that FLIP-384 has been accepted.

Binding votes +1:
- Jing Ge
- Rui Fan
- Hangxiang Yu
- Roman Khachatryan
- David Morávek
- Stefan Richter

Non-binding votes +1:
- Zakelly Lan
- David Radley

There were no votes against.

Best,
Piotrek

Re: [VOTE] FLIP-384: Introduce TraceReporter and use it to create checkpointing and recovery traces

[Piotr Nowojski]

Thank you all for voting. I will post the results soon.

Best,
Piotrek


śr., 29 lis 2023 o 11:10 David Radley  napisał(a):

> +1(non-binding)
>
> From: Stefan Richter 
> Date: Wednesday, 29 November 2023 at 09:44
> To: dev@flink.apache.org 
> Subject: [EXTERNAL] Re: [VOTE] FLIP-384: Introduce TraceReporter and use
> it to create checkpointing and recovery traces
> +1 (binding)
>
> Best,
> Stefan
>
>
> > On 22. Nov 2023, at 11:20, Roman Khachatryan  wrote:
> >
> > +1 (binding)
> >
> > Regards,
> > Roman
> >
> > On Wed, Nov 22, 2023, 7:08 AM Zakelly Lan  > wrote:
> >
> >> +1(non-binding)
> >>
> >> Best,
> >> Zakelly
> >>
> >> On Wed, Nov 22, 2023 at 3:04 PM Hangxiang Yu 
> wrote:
> >>
> >>> +1 (binding)
> >>> Thanks for driving this again!
> >>>
> >>> On Wed, Nov 22, 2023 at 10:30 AM Rui Fan <1996fan...@gmail.com> wrote:
> >>>
>  +1(binding)
> 
>  Best,
>  Rui
> 
>  On Wed, Nov 22, 2023 at 6:43 AM Jing Ge 
>  wrote:
> 
> > +1(binding) Thanks!
> >
> > Best regards,
> > Jing
> >
> > On Tue, Nov 21, 2023 at 6:17 PM Piotr Nowojski  >>>
> > wrote:
> >
> >> Hi All,
> >>
> >> I'd like to start a vote on the FLIP-384: Introduce TraceReporter
> >> and
>  use
> >> it to create checkpointing and recovery traces [1]. The discussion
>  thread
> >> is here [2].
> >>
> >> The vote will be open for at least 72 hours unless there is an
>  objection
> > or
> >> not enough votes.
> >>
> >> [1]
> https://www.google.com/url?q=https://cwiki.apache.org/confluence/x/TguZE=gmail-imap=170125329000=AOvVaw3QR8LNFApod9Cz_gw2y64w
> >> [2]
> >>
> https://www.google.com/url?q=https://lists.apache.org/thread/7lql5f5q1np68fw1wc9trq3d9l2ox8f4=gmail-imap=170125329000=AOvVaw28yzl2wfrtrnoPLsdLW-7q
> >>
> >>
> >> Best,
> >> Piotrek
> >>
> >
> 
> >>>
> >>>
> >>> --
> >>> Best,
> >>> Hangxiang.
>
> Unless otherwise stated above:
>
> IBM United Kingdom Limited
> Registered in England and Wales with number 741598
> Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU
>

Re: Python connector question

[Gabor Somogyi]

Hi Peter,

Thanks for picking this up! Please find my thoughts inline.

BR,
G


On Mon, Nov 27, 2023 at 4:05 PM Péter Váry 
wrote:

> Hi Team,
>
> During some, mostly unrelated work we come to realize that during the
> externalization of the connector's python modules and the related tests are
> not moved to the respective connectors repository.
> We created the jiras [1] to create the infra, and to move the python code
> and the tests to the respective repos.
>
> When working on this code, I have found several oddities, which I would
> like to hear the community's opinion on:

- Does anyone know what the
> site-packages/pyflink/opt/flink-sql-client-1.18.0.jar supposed to do? We
> specifically put it there [2], but then we ignore it when we check the
> directory of jars [3]. If there is no objection, I would remove it.
>
+1 on removing that unless there are objections. As I see  the jar is not
included in the classpath so not used.

> - I would like to use the `opt` directory, to contain optional jars created
> by the connectors, like flink-sql-connector-kafka-3.1.x.jar.
>
+1 on that. The other options would be:
 * plugins -> Kafka connector is not used as plugin w/ separate classloader
so this solution is not preferred
 * lib -> here we store our core jars which are part of the main Flink repo so
this solution is not preferred
All in all I'm fine to use opt.

> Also, the lint-python.sh [4], and install_command.sh [5] provides the base
> of the testing infra. Would there be any objections to mark these as a
> public apis for the connectors?

I fully agree that we should avoid code duplications and re-using the
existing code parts but making them API
is not what I can imagine. The reason behind is simple. That would just
block development in mid/long term in Flink main repo.
 * lint-python.sh is far from a stable version from my understanding + that
way Flink will contain the superset of maybe 10+ connector needs
 * install_command.sh contains generic and Flink specific parts. When we
extract the Flink specific parts we can declare it logically API
What I can imagine is to make the mentioned scripts more configurable and
make them @Experimental.

When a development team of a connector is not happy to use it as-is then a
deep-copy and/or rewrite is still an option.

> Thanks,
> Peter
>
> [1] - https://issues.apache.org/jira/browse/FLINK-33528
> [2] -
>
> https://github.com/apache/flink/blob/2da9a9639216b8c48850ee714065f090a80dcd65/flink-python/apache-flink-libraries/setup.py#L129-L130
> [3] -
>
> https://github.com/apache/flink/blob/2da9a9639216b8c48850ee714065f090a80dcd65/flink-python/pyflink/pyflink_gateway_server.py#L183-L190
> [4] -
> https://github.com/apache/flink/blob/master/flink-python/dev/lint-python.sh
> [5] -
>
> https://github.com/apache/flink/blob/master/flink-python/dev/install_command.sh
>

Re: [DISCUSS] FLIP-395: Migration to GitHub Actions

[Matthias Pohl]

With regards to Alex' concerns on hardware disparity: I did a bit more
digging on that one. I added my findings in a hardware section to FLIP-396
[1]. It appears that the hardware is more or less the same between the
different hosts. Apache INFRA's runners have more disk space (1TB in
comparison to 14GB), though.

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-396%3A+Trial+during+Flink+1.19+Cycle+to+test+migrating+to+GitHub+Actions#FLIP396:TrialduringFlink1.19CycletotestmigratingtoGitHubActions-HardwareSpecifications

On Wed, Nov 29, 2023 at 4:01 PM Matthias Pohl 
wrote:

> Thanks for your feedback Alex. I responded to your comments below:
>
> This is mentioned in the "Limitations of GitHub Actions in the past"
>> section of the FLIP. Does this also apply to the Apache INFRA setup or
>> can we expect contributors' runs executed there too?
>
>
> Workflow runs on Flink forks (independent of PRs that would merge to
> Apache Flink's core repo) will be executed with runners provided by GitHub
> with their own limitations. Secrets are not set in these runs (similar to
> what we have right now with PR runs).
>
> If we allow the PR CI to run on Apache INFRA-hosted ephemeral runners we
> might have the same freedom because of their ephemeral nature (the VMs are
> discarded leaving).
>
> We only have to start thinking about self-hosted customized runners if we
> decide/need to have dedicated VMs for Flink's CI (similar to what we have
> right now with Azure CI and Alibaba's VMs). This might happen if the
> waiting times for acquiring a runner are too long. In that case, we might
> give a certain group of people (e.g. committers) or certain types of events
> (for PRs,  nightly builds, PR merges) the ability to use the self-hosted
> runners.
>
> As you mentioned in the FLIP, there are some timeout-related test
>> discrepancies between different setups. Similar discrepancies could
>> manifest themselves between the Github runners and the Apache INFRA
>> runners. It would be great if we should have a uniform setup, where if
>> tests pass in the individual CI, they also pass in the main runner and vice
>> versa.
>
>
> I agree. So far, what we've seen is that the timeout instability is coming
> from too optimistic timeout configurations in some tests (they eventually
> also fail in Azure CI; but the GitHub-provided runners seem to be more
> sensitive in this regard). Fixing the tests if such a flakiness is observed
> should bring us to a stage where the test behavior is matching between
> different runners.
>
> We had a similar issue in the Azure CI setup: Certain tests were more
> stable on the Alibaba machines than on Azure VMs. That is why we introduced
> a dedicated stage for Azure CI VMs as part of the nightly runs (see
> FLINK-18370 [1]). We could do the same for GitHub Actions if necessary.
>
> Currently we have such memory limits-related issues in individual vs main
>> Azure CI pipelines.
>
>
> I'm not sure I understand what you mean by memory limit-related issues.
> The GitHub-provided runners do not seem to run into memory-related issues.
> We have to see whether this also applies to Apache INFRA-provided runners.
> My hope is that they have even better hardware than what GitHub offers. But
> GitHub-provided runners seem to be a good fallback to rely on (see the
> workflows I shared in my previous response to Xintong's message).
>
> [1] https://issues.apache.org/jira/browse/FLINK-18370
>
> On Wed, Nov 29, 2023 at 3:17 PM Matthias Pohl 
> wrote:
>
>> Thanks for your comments, Xintong. See my answers below.
>>
>>
>>> I think it would be helpful if we can at the end migrate the CI to an
>>> ASF-managed Github Action, as long as it provides us a similar
>>> computation capacity and stability.
>>
>>
>> The current test runs in my Flink fork (using the GitHub-provided
>> runners) suggest that even with using generic GitHub runners we get decent
>> performance and stability. In this way I'm confident that we wouldn't lose
>> much.
>>
>> Here's a comparison of the pipelines once more:
>> * Nightly workflow: GitHub Actions [1] vs Azure CI [2]
>> * PR workflow: GitHub Actions [3] vs Azure CI [4]
>>
>> [1]
>> https://github.com/XComp/flink/actions/workflows/flink-ci-extended.yml
>> [2]
>> https://dev.azure.com/apache-flink/apache-flink/_build?definitionId=1&_a=summary
>> [3] https://github.com/XComp/flink/actions/workflows/flink-ci-basic.yml
>> [4] https://dev.azure.com/apache-flink/apache-flink/_build?definitionId=2
>>
>> Regarding the migration plan, I wonder if we should not disable the CIbot
>>> until we fully decide to migrate to Github Actions? In case the nightly
>>> runs don't really work well, it might be debatable whether we should
>>> maintain the CI in two places (i.e. PRs on Github Actions and cron builds
>>> on Azure).
>>
>>
>> The CIbot handles the PR CI. Disabling it would mean that users would
>> fully rely on the GitHub Actions workflow right away. I like the fact that
>> for PRs we actually 

[jira] [Created] (FLINK-33694) GCS filesystem does not respect gs.storage.root.url config option

[Patrick Lucas (Jira)]

Patrick Lucas created FLINK-33694:
-

 Summary: GCS filesystem does not respect gs.storage.root.url 
config option
 Key: FLINK-33694
 URL: https://issues.apache.org/jira/browse/FLINK-33694
 Project: Flink
  Issue Type: Bug
  Components: FileSystems
Affects Versions: 1.17.2, 1.18.0
Reporter: Patrick Lucas


The GCS FileSystem's RecoverableWriter implementation uses the GCS SDK directly 
rather than going through Hadoop. While support has been added to configure 
credentials correctly based on the standard Hadoop implementation 
configuration, no other options are passed through to the underlying client.

Because this only affects the RecoverableWriter-related codepaths, it can 
result in very surprising differing behavior whether the FileSystem is being 
used as a source or a sink—while a {{{}gs://{}}}-URI FileSource may work fine, 
a {{{}gs://{}}}-URI FileSink may not work at all.

We use [fake-gcs-server|https://github.com/fsouza/fake-gcs-server] in testing, 
and so we override the Hadoop GCS FileSystem config option 
{{{}gs.storage.root.url{}}}. However, because this option is not considered 
when creating the GCS client for the RecoverableWriter codepath, in a FileSink 
the GCS FileSystem attempts to write to the real GCS service rather than 
fake-gcs-server. At the same time, a FileSource works as expected, reading from 
fake-gcs-server.

The fix should be fairly straightforward, reading the {{gs.storage.root.url}} 
config option from the Hadoop FileSystem config in 
[{{GSFileSystemOptions}}|https://github.com/apache/flink/blob/release-1.18.0/flink-filesystems/flink-gs-fs-hadoop/src/main/java/org/apache/flink/fs/gs/GSFileSystemOptions.java#L30]
 and, if set, passing it to {{storageOptionsBuilder}} in 
[{{GSFileSystemFactory}}|https://github.com/apache/flink/blob/release-1.18.0/flink-filesystems/flink-gs-fs-hadoop/src/main/java/org/apache/flink/fs/gs/GSFileSystemFactory.java].

The only workaround for this is to build a custom flink-gs-fs-hadoop JAR with a 
patch and use it as a plugin.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Sending a hi!

[Matthias Pohl]

Welcome to the community. :)

Some good starting points aside from reading the mailing lists [1] are the
documentation about contributing [2] and setting up your IDE [3]. You can
look for Jira issues labeled as starter [4].

Matthias

[1] https://flink.apache.org/what-is-flink/community/#mailing-lists
[2] https://flink.apache.org/how-to-contribute/overview/
[3]
https://nightlies.apache.org/flink/flink-docs-master/docs/flinkdev/ide_setup/
[4]
https://issues.apache.org/jira/browse/FLINK-33383?jql=project%20%3D%20FLINK%20AND%20labels%20%3D%20starter%20AND%20status%20NOT%20IN%20(Closed%2C%20Fixed%2C%20Resolved)%20ORDER%20BY%20created%20DESC

On Mon, Nov 27, 2023 at 12:46 PM Etienne Chauchot 
wrote:

> Welcome to the community Pranav !
>
> Best
>
> Etienne
>
> Le 25/11/2023 à 17:59, Pranav Sharma a écrit :
> > Hi everyone,
> >
> > I am Pranav, and I am getting started to contribute to the Apache Flink
> > project. I have been previously contributing to another Apache project,
> > Allura.
> >
> > I came across flink during my day job as a data engineer, and hoping to
> > contribute to the codebase as I learn more about the internal workings of
> > the framework.
> >
> > Regards,
> > Pranav S
> >

Re: [DISCUSS] FLIP-395: Migration to GitHub Actions

[Matthias Pohl]

Thanks for your feedback Alex. I responded to your comments below:

This is mentioned in the "Limitations of GitHub Actions in the past"
> section of the FLIP. Does this also apply to the Apache INFRA setup or can
> we expect contributors' runs executed there too?


Workflow runs on Flink forks (independent of PRs that would merge to Apache
Flink's core repo) will be executed with runners provided by GitHub with
their own limitations. Secrets are not set in these runs (similar to what
we have right now with PR runs).

If we allow the PR CI to run on Apache INFRA-hosted ephemeral runners we
might have the same freedom because of their ephemeral nature (the VMs are
discarded leaving).

We only have to start thinking about self-hosted customized runners if we
decide/need to have dedicated VMs for Flink's CI (similar to what we have
right now with Azure CI and Alibaba's VMs). This might happen if the
waiting times for acquiring a runner are too long. In that case, we might
give a certain group of people (e.g. committers) or certain types of events
(for PRs,  nightly builds, PR merges) the ability to use the self-hosted
runners.

As you mentioned in the FLIP, there are some timeout-related test
> discrepancies between different setups. Similar discrepancies could
> manifest themselves between the Github runners and the Apache INFRA
> runners. It would be great if we should have a uniform setup, where if
> tests pass in the individual CI, they also pass in the main runner and vice
> versa.


I agree. So far, what we've seen is that the timeout instability is coming
from too optimistic timeout configurations in some tests (they eventually
also fail in Azure CI; but the GitHub-provided runners seem to be more
sensitive in this regard). Fixing the tests if such a flakiness is observed
should bring us to a stage where the test behavior is matching between
different runners.

We had a similar issue in the Azure CI setup: Certain tests were more
stable on the Alibaba machines than on Azure VMs. That is why we introduced
a dedicated stage for Azure CI VMs as part of the nightly runs (see
FLINK-18370 [1]). We could do the same for GitHub Actions if necessary.

Currently we have such memory limits-related issues in individual vs main
> Azure CI pipelines.


I'm not sure I understand what you mean by memory limit-related issues. The
GitHub-provided runners do not seem to run into memory-related issues. We
have to see whether this also applies to Apache INFRA-provided runners. My
hope is that they have even better hardware than what GitHub offers. But
GitHub-provided runners seem to be a good fallback to rely on (see the
workflows I shared in my previous response to Xintong's message).

[1] https://issues.apache.org/jira/browse/FLINK-18370

On Wed, Nov 29, 2023 at 3:17 PM Matthias Pohl 
wrote:

> Thanks for your comments, Xintong. See my answers below.
>
>
>> I think it would be helpful if we can at the end migrate the CI to an
>> ASF-managed Github Action, as long as it provides us a similar
>> computation capacity and stability.
>
>
> The current test runs in my Flink fork (using the GitHub-provided runners)
> suggest that even with using generic GitHub runners we get decent
> performance and stability. In this way I'm confident that we wouldn't lose
> much.
>
> Here's a comparison of the pipelines once more:
> * Nightly workflow: GitHub Actions [1] vs Azure CI [2]
> * PR workflow: GitHub Actions [3] vs Azure CI [4]
>
> [1] https://github.com/XComp/flink/actions/workflows/flink-ci-extended.yml
> [2]
> https://dev.azure.com/apache-flink/apache-flink/_build?definitionId=1&_a=summary
> [3] https://github.com/XComp/flink/actions/workflows/flink-ci-basic.yml
> [4] https://dev.azure.com/apache-flink/apache-flink/_build?definitionId=2
>
> Regarding the migration plan, I wonder if we should not disable the CIbot
>> until we fully decide to migrate to Github Actions? In case the nightly
>> runs don't really work well, it might be debatable whether we should
>> maintain the CI in two places (i.e. PRs on Github Actions and cron builds
>> on Azure).
>
>
> The CIbot handles the PR CI. Disabling it would mean that users would
> fully rely on the GitHub Actions workflow right away. I like the fact that
> for PRs we actually have both. That makes it more obvious if CI is not on
> par.
> For the nightly builds, I'm not too worried because they are not exposed
> to the contributors that much. That's more a question for the release
> managers who are monitoring the nightly runs how they want to handle it.
> But even there I see benefits of having both CIs running for some time to
> see how much they differ from each other in terms of stability
>
> - What exactly are the changes that would affect contributors during the
>> trial period? Is it only an additional CI report that you can potentially
>> just ignore? Or would there be some larger impacts, e.g. you cannot merge a
>> PR if the Github Action CI is not passed (I don't know, I just 

Re: [ANNOUNCE] Experimental Java 21 support now available on master

[Sergey Nuyanzin]

thanks for the response


>I feel doubt about the conclusion that "don't try to load a savepoint from
a Java 8/11/17 build due to bumping to scala-2.12.18", since the
snapshotted state (operator/keyed state-backend),  and most key/value
serializer snapshots are generated by pure-java code.
>The only left part is that the developer uses scala UDF or scala types for
key/value types. However, since all user-facing scala APIs have been
deprecated, I don't think we have so many cases. Maybe we can give
descriptions without such strong suggestions.

That is the area where I feel I lack the knowledge to answer this precisely.
My assumption was that statement about Java 21 regarding this should be
similar to Java 17 which is almost same [1]
Sorry for the inaccuracy
Based on your statements I agree that the conclusion could be more relaxed.

I'm curious whether there are some tests or anything which could clarify
this?

[1] https://lists.apache.org/thread/mz0m6wqjmqy8htob3w4469pjbg9305do

On Wed, Nov 29, 2023 at 12:25 PM Yun Tang  wrote:

> Thanks Sergey for the great work.
>
> I feel doubt about the conclusion that "don't try to load a savepoint from
> a Java 8/11/17 build due to bummping to scala-2.12.18", since the
> snapshotted state (operator/keyed state-backend),  and most key/value
> serializer snapshots are generated by pure-java code. The only left part is
> that the developer uses scala UDF or scala types for key/value types.
> However, since all user-facing scala APIs have been deprecated [1], I don't
> think we have so many cases. Maybe we can give descriptions without such
> strong suggestions.
>
> Please correct me if I am wrong.
>
>
> [1] https://issues.apache.org/jira/browse/FLINK-29740
>
> Best
> Yun Tang
>
> 
> From: Rui Fan <1996fan...@gmail.com>
> Sent: Wednesday, November 29, 2023 16:43
> To: dev@flink.apache.org 
> Subject: Re: [ANNOUNCE] Experimental Java 21 support now available on
> master
>
> Thanks Sergey for the great work!
>
> Best,
> Rui
>
> On Wed, Nov 29, 2023 at 4:42 PM Leonard Xu  wrote:
>
> > Cool !
> >
> > Thanks Sergey for the great effort and all involved.
> >
> >
> > Best,
> > Leonard
> >
> > > 2023年11月29日 下午4:31，Swapnal Varma  写道：
> > >
> > > Congratulations Sergey, and everyone involved!
> > >
> > > Excited to work with and on this!
> > >
> > > Best,
> > > Swapnal
> > >
> > >
> > > On Wed, 29 Nov 2023, 13:58 Sergey Nuyanzin, 
> wrote:
> > >
> > >> The master branch now builds and runs with Java 21 out-of-the-box.
> > >>
> > >> Notes:
> > >> - a nightly cron build was set up.
> > >> - In Java 21 builds, Scala is being bumped to 2.12.18
> > >> which causes incompatibilities within Flink;
> > >> i.e. don't try to load a savepoint from a Java 8/11/17 build
> > >> - All the tests that are being skipped on Java 11/17
> > >> are also skipped on Java 21.
> > >>
> > >> Huge shout-out to everyone participating
> > >> in review of my Java 21 related PRs
> > >>
> > >> If you run into any issues, please report it in FLINK-33163
> > >>  .
> > >>
> > >> --
> > >> Best regards,
> > >> Sergey
> > >>
> >
> >
>


-- 
Best regards,
Sergey

Re: [DISCUSS] Resolve diamond inheritance of Sink.createWriter

[Márton Balassi]

Thanks, Martijn and Peter.

In terms of the concrete issue:

   - I am following up with the author of FLIP-321 [1] (Becket) to update
   the docs [2] to reflect the right state.
   - I see two reasonable approaches in terms of proceeding with the
   specific changeset:


   1. We allow the exception from FLIP-321 for this change and let the
  PublicEvolving API change happen between Flink 1.18 and 1.19, which is
  consistent with current state of the relevant documentation. [2]
We commit
  to helping the connector repos make the necessary (one liner) changes.
  2. We revert back to the original implementation plan as explicitly
  voted on in FLIP-371 [3]. That has no API breaking changes.
However we end
  up with an inconsistently named API with duplicated internal
methods. Peter
  has also discovered additional bad patterns during his work in FLIP-372
  [3], the total of these changes could be handled in a separate FLIP that
  would do multiple PublicEvolving breaking changes to clean up the API.

In terms of the general issues:

   - I agree that if a PR review of an accepted FLIP newly introduces a
   breaking API change that warrants an update to the mailing list discussion
   and possibly even a new vote.
   - I agree with the general sentiment of FLIP-321 to provide stronger API
   guarantees with the minor note that if we have changes in mind we should
   prioritize them now such that they can be validated by Flink 2.0.
   - I agree that ideally the connector repos should build against the
   latest release and not the master branch.

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-321%3A+Introduce+an+API+deprecation+process
[2]
https://nightlies.apache.org/flink/flink-docs-master/docs/ops/upgrading/#api-compatibility-guarantees
[3]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-371%3A+Provide+initialization+context+for+Committer+creation+in+TwoPhaseCommittingSink
[4]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-372%3A+Allow+TwoPhaseCommittingSink+WithPreCommitTopology+to+alter+the+type+of+the+Committable

Best,
Marton

On Mon, Nov 27, 2023 at 7:23 PM Péter Váry 
wrote:

> I think we should try to separate the discussion in a few different topics:
>
>- Concrete issue
>   - How to solve this problem in 1.19 and wrt the affected createWriter
>   interface
>   - Update the documentation [1], so FLIP-321 is visible for every
>   contributor
>- Generic issue
>   - API stability
>   - Connector dependencies
>
>
> *CreateWriter interface*
> The change on the createWriter is not strictly required for the
> functionality defined by the requirements on the FLIP.
> If the only goal is only to have a backward compatible API, we can simply
> create a separate `*CommitterInitContext*` object and do not touch the
> writer `*InitContext*`, like it was done in the original PR [2].
> The issue is that this would result in an implementation which has
> duplicated methods/implementations (internal issue only), and has
> inconsistent naming (issue for external users).
>
> If we want to create an API which is consistent (and I agree with the
> reviewer's comments), then we need to rename the parameter type (
> *WriterInitContext*) for the createWriter method.
> I have tried to keep the backward compatibility with creating a new method
> and providing a default implementation for this new method which would call
> the original method after converting the WriterInitContext to InitContext.
>
> This is failed because the following details:
>
>- *org.apache.flink.api.connector.sink2.Sink* defines
> `*SinkWriter
>createWriter(InitContext context)`*
>- *org.apache.flink.api.connector.sink2.StatefulSink* narrows it
> down to *`StatefulSinkWriterWriterStateT> createWriter(InitContext context)`*
>- *org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink* narrows
>it down to *`PrecommittingSinkWriter
>createWriter(WriterInitContext context)`*
>-
>
>  
> *org.apache.flink.streaming.runtime.operators.sink.TestSinkV2.TestStatefulSinkV2*
>implements *StatefulSink* and *TwoPhaseCommittingSink* too
>
> *TestStatefulSinkV2* is a good example where we can not achieve backward
> compatibility, since the the compiler will fail with unrelated default
> methods [3]
>
> I am open for any suggestions how to move to the new API, and keep the
> backward compatibility. If we do not find a way to keep backward
> compatibility, and we decide that we would like to honour FLIP-321, then we
> can reverting to the original solution and keep only the changes for the `
> *createCommitter*` method.
>
> *Update the documentation*
> I have not found only one place in the docs [1], where we talk about the
> compatibility guarantees.
> Based FLIP-321 and the result of the discussion here, we should update this
> page.
>
> *API stability*
> I agree with the general sentiment of FLIP-321 to keep the changes backward
> 

Re: [DISCUSS] FLIP-395: Migration to GitHub Actions

[Matthias Pohl]

Thanks for your comments, Xintong. See my answers below.


> I think it would be helpful if we can at the end migrate the CI to an
> ASF-managed Github Action, as long as it provides us a similar computation
> capacity and stability.


The current test runs in my Flink fork (using the GitHub-provided runners)
suggest that even with using generic GitHub runners we get decent
performance and stability. In this way I'm confident that we wouldn't lose
much.

Here's a comparison of the pipelines once more:
* Nightly workflow: GitHub Actions [1] vs Azure CI [2]
* PR workflow: GitHub Actions [3] vs Azure CI [4]

[1] https://github.com/XComp/flink/actions/workflows/flink-ci-extended.yml
[2]
https://dev.azure.com/apache-flink/apache-flink/_build?definitionId=1&_a=summary
[3] https://github.com/XComp/flink/actions/workflows/flink-ci-basic.yml
[4] https://dev.azure.com/apache-flink/apache-flink/_build?definitionId=2

Regarding the migration plan, I wonder if we should not disable the CIbot
> until we fully decide to migrate to Github Actions? In case the nightly
> runs don't really work well, it might be debatable whether we should
> maintain the CI in two places (i.e. PRs on Github Actions and cron builds
> on Azure).


The CIbot handles the PR CI. Disabling it would mean that users would fully
rely on the GitHub Actions workflow right away. I like the fact that for
PRs we actually have both. That makes it more obvious if CI is not on par.
For the nightly builds, I'm not too worried because they are not exposed to
the contributors that much. That's more a question for the release managers
who are monitoring the nightly runs how they want to handle it. But even
there I see benefits of having both CIs running for some time to see how
much they differ from each other in terms of stability

- What exactly are the changes that would affect contributors during the
> trial period? Is it only an additional CI report that you can potentially
> just ignore? Or would there be some larger impacts, e.g. you cannot merge a
> PR if the Github Action CI is not passed (I don't know, I just made this
> up)?


My plan would be to enable the PR CI workflow for PRs as well to have the
comparison. For contributors this would mean that they have an additional
CI point (essentially two CI runs for a PR). If that's not what we want, we
could disable it for PRs and only allow the basic CI run for pushes to
master.

On Wed, Nov 29, 2023 at 2:31 PM Alexander Fedulov <
alexander.fedu...@gmail.com> wrote:

> Thanks for driving this Mathhias! +1 for joining the INFRA trial.
>
>
> > Apache Infra did some experimenting on self-hosted runners in
> collaboration
> > with Apache Airflow (see ashb/runner with releases/pr-security-options
> branch)
> > where they only allow certain groups of users (e.g. committers) to run
> their
> > workflows on self-hosted machines. Any other group would have to rely on
> > GitHub’s runners.
>
> This is mentioned in the "Limitations of GitHub Actions in the past"
> section of
> the FLIP. Does this also apply to the Apache INFRA setup or can we expect
> contributors' runs executed there too? As you mentioned in the FLIP, there
> are
> some timeout-related test discrepancies between different setups. Similar
> discrepancies could manifest themselves between the Github runners and the
> Apache INFRA runners. It would be great if we should have a uniform setup,
> where if tests pass in the individual CI, they also pass in the main runner
> and
> vice versa.  Currently we have such memory limits-related issues in
> individual
> vs main Azure CI pipelines.
>
> >2. Disable Flink’s CI bot for PRs if step #1 is considered successful
> >3. Join trial program for ephemeral GHA runners
>
> Due to potential new kinds of instabilities manifesting themselves in the
> new setup,
> can we keep both CIs running in parallel and keep relying on the existing
> one until
> we are confident in the tests stability on the new ephemeral GHA infra
> (skip 2.)?
>
> Best,
> Alex
>
> On Wed, 29 Nov 2023 at 13:42, Xintong Song  wrote:
>
> > Thanks for the efforts, Matthias.
> >
> >
> > I think it would be helpful if we can at the end migrate the CI to an
> > ASF-managed Github Action, as long as it provides us a similar
> computation
> > capacity and stability. Given that the proposal is only to start a trial
> > and investigate whether the migration is feasible, I don't see much
> concern
> > in this.
> >
> >
> > I have only one suggestion and one question.
> >
> > - Regarding the migration plan, I wonder if we should not disable the CI
> > bot until we fully decide to migrate to Github Actions? In case the
> nightly
> > runs don't really work well, it might be debatable whether we should
> > maintain the CI in two places (i.e. PRs on Github Actions and cron builds
> > on Azure).
> >
> > - What exactly are the changes that would affect contributors during the
> > trial period? Is it only an additional CI report that you can potentially
> > just 

Re: [DISCUSS] FLIP-395: Migration to GitHub Actions

[Alexander Fedulov]

Thanks for driving this Mathhias! +1 for joining the INFRA trial.


> Apache Infra did some experimenting on self-hosted runners in
collaboration
> with Apache Airflow (see ashb/runner with releases/pr-security-options
branch)
> where they only allow certain groups of users (e.g. committers) to run
their
> workflows on self-hosted machines. Any other group would have to rely on
> GitHub’s runners.

This is mentioned in the "Limitations of GitHub Actions in the past"
section of
the FLIP. Does this also apply to the Apache INFRA setup or can we expect
contributors' runs executed there too? As you mentioned in the FLIP, there
are
some timeout-related test discrepancies between different setups. Similar
discrepancies could manifest themselves between the Github runners and the
Apache INFRA runners. It would be great if we should have a uniform setup,
where if tests pass in the individual CI, they also pass in the main runner
and
vice versa.  Currently we have such memory limits-related issues in
individual
vs main Azure CI pipelines.

>2. Disable Flink’s CI bot for PRs if step #1 is considered successful
>3. Join trial program for ephemeral GHA runners

Due to potential new kinds of instabilities manifesting themselves in the
new setup,
can we keep both CIs running in parallel and keep relying on the existing
one until
we are confident in the tests stability on the new ephemeral GHA infra
(skip 2.)?

Best,
Alex

On Wed, 29 Nov 2023 at 13:42, Xintong Song  wrote:

> Thanks for the efforts, Matthias.
>
>
> I think it would be helpful if we can at the end migrate the CI to an
> ASF-managed Github Action, as long as it provides us a similar computation
> capacity and stability. Given that the proposal is only to start a trial
> and investigate whether the migration is feasible, I don't see much concern
> in this.
>
>
> I have only one suggestion and one question.
>
> - Regarding the migration plan, I wonder if we should not disable the CI
> bot until we fully decide to migrate to Github Actions? In case the nightly
> runs don't really work well, it might be debatable whether we should
> maintain the CI in two places (i.e. PRs on Github Actions and cron builds
> on Azure).
>
> - What exactly are the changes that would affect contributors during the
> trial period? Is it only an additional CI report that you can potentially
> just ignore? Or would there be some larger impacts, e.g. you cannot merge a
> PR if the Github Action CI is not passed (I don't know, I just made this
> up)?
>
>
> Best,
>
> Xintong
>
>
>
> On Wed, Nov 29, 2023 at 8:07 PM Yuxin Tan  wrote:
>
> > Ok, Thanks for the update and the explanations.
> >
> > Best,
> > Yuxin
> >
> >
> > Matthias Pohl  于2023年11月29日周三 15:43写道：
> >
> > > >
> > > > According to the Flip, the new tests will support arm env.
> > > > I believe that's good news for arm users. I have a minor
> > > > question here. Will it be a blocker before migrating the new
> > > > tests? If not,  If not, when can we expect arm environment
> > > > support to be implemented? Thanks.
> > >
> > >
> > > Thanks for your feedback, everyone.
> > >
> > > About the ARM support. I want to underline that this FLIP is not about
> > > migrating to GitHub Actions but to start a trial run in the Apache
> Flink
> > > repository. That would allow us to come up with a proper decision
> whether
> > > GitHub Actions is what we want. I admit that the title is a bit
> > > "clickbaity". I updated the FLIP's title and its Motivation to make
> > things
> > > clear.
> > >
> > > The FLIP suggests starting a trial period until 1.19 is released to try
> > > things out. A proper decision on whether we want to migrate would be
> made
> > > at the end of the 1.19 release cycle.
> > >
> > > About the ARM support: This related content of the FLIP is entirely
> based
> > > on documentation from Apache INFRAs side. INFRA seems to offer this ARM
> > > support for their ephemeral runners. The ephemeral runners are in the
> > > testing stage, i.e. these runners are still experimental. Apache INFRA
> > asks
> > > Apache projects to join this test.
> > >
> > > Whether the ARM support is actually possible to achieve within Flink is
> > > something we have to figure out as part of the trial run. One
> conclusion
> > of
> > > the trial run could be that we still move ahead with GHA but don't use
> > arm
> > > machines due to some blocking issues.
> > >
> > > Matthias
> > >
> > >
> > >
> > > On Wed, Nov 29, 2023 at 4:46 AM Yuxin Tan 
> > wrote:
> > >
> > > > Hi, Matthias,
> > > >
> > > > Thanks for driving this.
> > > > +1 from my side.
> > > >
> > > > According to the Flip, the new tests will support arm env.
> > > > I believe that's good news for arm users. I have a minor
> > > > question here. Will it be a blocker before migrating the new
> > > > tests? If not,  If not, when can we expect arm environment
> > > > support to be implemented? Thanks.
> > > >
> > > > Best,
> > > > Yuxin
> > > >
> > > >
> > > > Márton Balassi  

[jira] [Created] (FLINK-33693) The disable unaligned checkpoint isn't respected when the aligned checkpoint timeout is enabled

[Rui Fan (Jira)]

Rui Fan created FLINK-33693:
---

 Summary: The disable unaligned checkpoint isn't respected when the 
aligned checkpoint timeout is enabled 
 Key: FLINK-33693
 URL: https://issues.apache.org/jira/browse/FLINK-33693
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Checkpointing
Affects Versions: 1.17.2, 1.18.0
Reporter: Rui Fan
Assignee: Rui Fan
 Attachments: image-2023-11-29-20-41-17-038.png

The disable unaligned checkpoint isn't respected when the aligned checkpoint 
timeout is enabled.

 

Bug at org.apache.flink.streaming.runtime.io.RecordWriterOutput#broadcastEvent, 
we will call withUnalignedUnsupported to transmit the unaligned checkpoint 
barrier to {color:#9876aa}FORCED_ALIGNED {color}barrier when the shuffle type 
cannot use unaligned checkpoint.

However, the if has one condition: isPriorityEvent. When  aligned checkpoint 
timeout is enabled, flink will emit one timeoutable barrier. It isn't 
PriorityEvent.

 

But timeoutable barrier should call withUnalignedUnsupported here.

!image-2023-11-29-20-41-17-038.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-395: Migration to GitHub Actions

[Xintong Song]

Thanks for the efforts, Matthias.


I think it would be helpful if we can at the end migrate the CI to an
ASF-managed Github Action, as long as it provides us a similar computation
capacity and stability. Given that the proposal is only to start a trial
and investigate whether the migration is feasible, I don't see much concern
in this.


I have only one suggestion and one question.

- Regarding the migration plan, I wonder if we should not disable the CI
bot until we fully decide to migrate to Github Actions? In case the nightly
runs don't really work well, it might be debatable whether we should
maintain the CI in two places (i.e. PRs on Github Actions and cron builds
on Azure).

- What exactly are the changes that would affect contributors during the
trial period? Is it only an additional CI report that you can potentially
just ignore? Or would there be some larger impacts, e.g. you cannot merge a
PR if the Github Action CI is not passed (I don't know, I just made this
up)?


Best,

Xintong



On Wed, Nov 29, 2023 at 8:07 PM Yuxin Tan  wrote:

> Ok, Thanks for the update and the explanations.
>
> Best,
> Yuxin
>
>
> Matthias Pohl  于2023年11月29日周三 15:43写道：
>
> > >
> > > According to the Flip, the new tests will support arm env.
> > > I believe that's good news for arm users. I have a minor
> > > question here. Will it be a blocker before migrating the new
> > > tests? If not,  If not, when can we expect arm environment
> > > support to be implemented? Thanks.
> >
> >
> > Thanks for your feedback, everyone.
> >
> > About the ARM support. I want to underline that this FLIP is not about
> > migrating to GitHub Actions but to start a trial run in the Apache Flink
> > repository. That would allow us to come up with a proper decision whether
> > GitHub Actions is what we want. I admit that the title is a bit
> > "clickbaity". I updated the FLIP's title and its Motivation to make
> things
> > clear.
> >
> > The FLIP suggests starting a trial period until 1.19 is released to try
> > things out. A proper decision on whether we want to migrate would be made
> > at the end of the 1.19 release cycle.
> >
> > About the ARM support: This related content of the FLIP is entirely based
> > on documentation from Apache INFRAs side. INFRA seems to offer this ARM
> > support for their ephemeral runners. The ephemeral runners are in the
> > testing stage, i.e. these runners are still experimental. Apache INFRA
> asks
> > Apache projects to join this test.
> >
> > Whether the ARM support is actually possible to achieve within Flink is
> > something we have to figure out as part of the trial run. One conclusion
> of
> > the trial run could be that we still move ahead with GHA but don't use
> arm
> > machines due to some blocking issues.
> >
> > Matthias
> >
> >
> >
> > On Wed, Nov 29, 2023 at 4:46 AM Yuxin Tan 
> wrote:
> >
> > > Hi, Matthias,
> > >
> > > Thanks for driving this.
> > > +1 from my side.
> > >
> > > According to the Flip, the new tests will support arm env.
> > > I believe that's good news for arm users. I have a minor
> > > question here. Will it be a blocker before migrating the new
> > > tests? If not,  If not, when can we expect arm environment
> > > support to be implemented? Thanks.
> > >
> > > Best,
> > > Yuxin
> > >
> > >
> > > Márton Balassi  于2023年11月29日周三 03:09写道：
> > >
> > > > Thanks, Matthias. Big +1 from me.
> > > >
> > > > On Tue, Nov 28, 2023 at 5:30 PM Matthias Pohl
> > > >  wrote:
> > > >
> > > > > Thanks for the pointer. I'm planning to join that meeting.
> > > > >
> > > > > On Tue, Nov 28, 2023 at 4:16 PM Etienne Chauchot <
> > echauc...@apache.org
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > FYI there is the ASF infra roundtable soon. One of the subjects
> for
> > > > this
> > > > > > session is GitHub Actions. It could be worth passing by:
> > > > > >
> > > > > > December 6th, 2023 at 1700 UTC on the #Roundtablechannel on
> Slack.
> > > > > >
> > > > > > For information about theroundtables, and about how to join,
> > > > > > see:https://infra.apache.org/roundtable.html
> > > > > > 
> > > > > >
> > > > > > Best
> > > > > >
> > > > > > Etienne
> > > > > >
> > > > > > Le 24/11/2023 à 14:16, Maximilian Michels a écrit :
> > > > > > > Thanks for reviving the efforts here Matthias! +1 for the
> > > transition
> > > > > > > to GitHub Actions.
> > > > > > >
> > > > > > > As for ASF Infra Jenkins, it works fine. Jenkins is extremely
> > > > > > > feature-rich. Not sure about the spare capacity though. I know
> > that
> > > > > > > for Apache Beam, Google donated a bunch of servers to get
> > > additional
> > > > > > > build capacity.
> > > > > > >
> > > > > > > -Max
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Nov 23, 2023 at 10:30 AM Matthias Pohl
> > > > > > >   wrote:
> > > > > > >> Btw. even though we've been focusing on GitHub Actions with
> this
> > > > FLIP,
> > > > > > I'm
> > > > > > >> curious whether 

Re: [DISCUSS] FLIP-364: Improve the restart-strategy

[Rui Fan]

Hi all,

The user mail[1] has started for 13 days, and it collected
one useful suggestion.

> Given that the new default feels more complex than the current behavior,
if we decide to do this I think it will be important to include the
rationale you've shared in the documentation.

I will add the related doc to explain it.

Also, based on Zhu's suggestion. This FLIP
is changed to `Improve the exponential-delay restart-strategy`.
It focuses on improving the exponential-delay restart-strategy,
and ignores the fixed delay and failure rate in this FLIP.

If you have no questions, you are welcome to vote in the
Vote thread[2], the mail title is `[VOTE] FLIP-364: Improve the
restart-strategy`,
but it can still be used as a voting thread.

Thank you to everyone who participated in the discussion.

[1] https://lists.apache.org/thread/6glz0d57r8gtpzq4f71vf9066c5x6nyw
[2] https://lists.apache.org/thread/xo03tzw6d02w1vbcj5y9ccpqyc7bqrh9

Best,
Rui

On Fri, Nov 17, 2023 at 12:02 PM Mingliang Liu  wrote:

> Thank you Rui. It makes sense to me now.
>
> On Thu, Nov 16, 2023 at 2:57 AM Rui Fan <1996fan...@gmail.com> wrote:
>
> > Hi all,
> >
> > Zhu and I had an offline discussion today. We prefer this FLIP
> > focuses on improving exponential-delay and uses exponential-delay
> > as the default strategy. It means this FLIP doesn't include
> > improvements related to fixed-delay and failover-delay, and the
> > second part of FLIP(Improve restartAttempt's counting strategy)
> > just improves exponential-delay.
> >
> > Following are reasons:
> >
> > 1. Judging from current discussion, many users want
> >   exponential-delay as the default restart strategy.
> >
> > 2. The semantics of naming and behavior are inconsistent
> >
> > If we improve the restartAttempt counting mechanism for all
> > restart strategies, we need to unify the concept of restartAttempt
> > counting. We want to increase based on the number of restarts,
> > not the number of failures. The number of failures will increase
> > too fast, so we hope to aggregate multiple failures into one restart.
> >
> > However, the failure-rate strategy's restart upper limit option is
> > named  restart-strategy.failure-rate.max-failures-per-interval,
> > it's  max-failures-per-interval instead of max-attempts-per-interval.
> > If we improve it directly, the name and behaviour aren't matched.
> >
> > 3. The restartAttempt counting mechanism and Exception History
> >   are not match
> >
> > If we aggregate multiple failures into one restartAttempt, one failure
> > is an exception in Exception History. Users allowed 10 attempts,
> > but saw 100 failures on the Exception History, and the job has not
> > exited yet. Users may be confused. It's related to concurrentExceptions,
> > and it will be followed at FLINK-33565.
> >
> > For these reasons, we prefer that current FLIP focus on
> exponential-delay.
> > After FLINK-33565 is done, we can discuss the rest of restart-strategies
> > again.
> >
> > Looking forward to your feedback, thanks~
> >
> > To Mingliang,
> >
> > Sorry, I missed one of your questions this morning.
> >
> > > One question is the max attempts. Is that the max attempt after which
> > the job will be deemed failed? I'm wondering if we just simplify the name
> > from `max-attempts-before-reset-backoff` to `max-attempts` or just
> > `attempts` > (like the static strategy
> > `restart-strategy.fixed-delay.attempts`). The wording
> `before-reset-backoff
> > ` makes me think it's setting the backoff interval to its initial value
> > after this max attempt, instead of failing the job.
> >
> > The max-attempts-before-reset-backoff isn't the same with max-attempts or
> > attempts.
> > The exponential-delay has a reset mechanism, when no exception within
> > reset-backoff-threshold. Flink will reset the delay time to
> > initial-backoff.
> > max-attempts-before-reset-backoff indicates the maximum number of
> restarts
> > we can attempt before resetting.
> > - When restartAttempt > max-attempts-before-reset-backoff, the job will
> > exit.
> > - When no exception within reset-backoff-threshold, the delay time will
> be
> >   reset to initial-backoff, and restartAttempt will be reset to 0 as
> well.
> >
> > After your feedback, I think attempts-before-reset-backoff may be better,
> > the max can be removed, and it is like
> > `restart-strategy.fixed-delay.attempts`.
> > WDYT?
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-33565
> >
> > Best,
> > Rui
> >
> > On Thu, Nov 16, 2023 at 11:48 AM Rui Fan <1996fan...@gmail.com> wrote:
> >
> >> Hi Zhu and Matthias:
> >>
> >> > 3. failure counting
> >> > Flink currently will try to recognize concurrent failures and group
> them
> >> > together, which can be seen in the web UI. So how about to align the
> >> > failure counting with the concurrent failures computing? This can make
> >> it
> >> > more consistent and easier for understanding. It will require changes
> to
> >> > the concurrent failures computing though, i.e. 

Re: [DISCUSS] FLIP-395: Migration to GitHub Actions

[Yuxin Tan]

Ok, Thanks for the update and the explanations.

Best,
Yuxin


Matthias Pohl  于2023年11月29日周三 15:43写道：

> >
> > According to the Flip, the new tests will support arm env.
> > I believe that's good news for arm users. I have a minor
> > question here. Will it be a blocker before migrating the new
> > tests? If not,  If not, when can we expect arm environment
> > support to be implemented? Thanks.
>
>
> Thanks for your feedback, everyone.
>
> About the ARM support. I want to underline that this FLIP is not about
> migrating to GitHub Actions but to start a trial run in the Apache Flink
> repository. That would allow us to come up with a proper decision whether
> GitHub Actions is what we want. I admit that the title is a bit
> "clickbaity". I updated the FLIP's title and its Motivation to make things
> clear.
>
> The FLIP suggests starting a trial period until 1.19 is released to try
> things out. A proper decision on whether we want to migrate would be made
> at the end of the 1.19 release cycle.
>
> About the ARM support: This related content of the FLIP is entirely based
> on documentation from Apache INFRAs side. INFRA seems to offer this ARM
> support for their ephemeral runners. The ephemeral runners are in the
> testing stage, i.e. these runners are still experimental. Apache INFRA asks
> Apache projects to join this test.
>
> Whether the ARM support is actually possible to achieve within Flink is
> something we have to figure out as part of the trial run. One conclusion of
> the trial run could be that we still move ahead with GHA but don't use arm
> machines due to some blocking issues.
>
> Matthias
>
>
>
> On Wed, Nov 29, 2023 at 4:46 AM Yuxin Tan  wrote:
>
> > Hi, Matthias,
> >
> > Thanks for driving this.
> > +1 from my side.
> >
> > According to the Flip, the new tests will support arm env.
> > I believe that's good news for arm users. I have a minor
> > question here. Will it be a blocker before migrating the new
> > tests? If not,  If not, when can we expect arm environment
> > support to be implemented? Thanks.
> >
> > Best,
> > Yuxin
> >
> >
> > Márton Balassi  于2023年11月29日周三 03:09写道：
> >
> > > Thanks, Matthias. Big +1 from me.
> > >
> > > On Tue, Nov 28, 2023 at 5:30 PM Matthias Pohl
> > >  wrote:
> > >
> > > > Thanks for the pointer. I'm planning to join that meeting.
> > > >
> > > > On Tue, Nov 28, 2023 at 4:16 PM Etienne Chauchot <
> echauc...@apache.org
> > >
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > FYI there is the ASF infra roundtable soon. One of the subjects for
> > > this
> > > > > session is GitHub Actions. It could be worth passing by:
> > > > >
> > > > > December 6th, 2023 at 1700 UTC on the #Roundtablechannel on Slack.
> > > > >
> > > > > For information about theroundtables, and about how to join,
> > > > > see:https://infra.apache.org/roundtable.html
> > > > > 
> > > > >
> > > > > Best
> > > > >
> > > > > Etienne
> > > > >
> > > > > Le 24/11/2023 à 14:16, Maximilian Michels a écrit :
> > > > > > Thanks for reviving the efforts here Matthias! +1 for the
> > transition
> > > > > > to GitHub Actions.
> > > > > >
> > > > > > As for ASF Infra Jenkins, it works fine. Jenkins is extremely
> > > > > > feature-rich. Not sure about the spare capacity though. I know
> that
> > > > > > for Apache Beam, Google donated a bunch of servers to get
> > additional
> > > > > > build capacity.
> > > > > >
> > > > > > -Max
> > > > > >
> > > > > >
> > > > > > On Thu, Nov 23, 2023 at 10:30 AM Matthias Pohl
> > > > > >   wrote:
> > > > > >> Btw. even though we've been focusing on GitHub Actions with this
> > > FLIP,
> > > > > I'm
> > > > > >> curious whether somebody has experience with Apache Infra's
> > Jenkins
> > > > > >> deployment. The discussion I found about Jenkins [1] is quite
> > > > out-dated
> > > > > >> (2014). I haven't worked with it myself but could imagine that
> > there
> > > > are
> > > > > >> some features provided through plugins which are missing in
> GitHub
> > > > > Actions.
> > > > > >>
> > > > > >> [1]
> > https://lists.apache.org/thread/vs81xdhn3q777r7x9k7wd4dyl9kvoqn4
> > > > > >>
> > > > > >> On Tue, Nov 21, 2023 at 4:19 PM Matthias Pohl<
> > > matthias.p...@aiven.io>
> > > > > >> wrote:
> > > > > >>
> > > > > >>> That's a valid point. I updated the FLIP accordingly:
> > > > > >>>
> > > > >  Currently, the secrets (e.g. for S3 access tokens) are
> > maintained
> > > by
> > > > >  certain PMC members with access to the corresponding
> > configuration
> > > > in
> > > > > the
> > > > >  Azure CI project. This responsibility will be moved to Apache
> > > Infra.
> > > > > They
> > > > >  are in charge of handling secrets in the Apache organization.
> > As a
> > > > >  consequence, updating secrets is becoming a bit more
> > complicated.
> > > > > This can
> > > > >  be still considered an improvement from a legal standpoint
> > because
> > > > the
> > > > >  responsibility is 

Re: [ANNOUNCE] Experimental Java 21 support now available on master

[Yun Tang]

Thanks Sergey for the great work.

I feel doubt about the conclusion that "don't try to load a savepoint from a 
Java 8/11/17 build due to bummping to scala-2.12.18", since the snapshotted 
state (operator/keyed state-backend),  and most key/value serializer snapshots 
are generated by pure-java code. The only left part is that the developer uses 
scala UDF or scala types for key/value types. However, since all user-facing 
scala APIs have been deprecated [1], I don't think we have so many cases. Maybe 
we can give descriptions without such strong suggestions.

Please correct me if I am wrong.


[1] https://issues.apache.org/jira/browse/FLINK-29740

Best
Yun Tang


From: Rui Fan <1996fan...@gmail.com>
Sent: Wednesday, November 29, 2023 16:43
To: dev@flink.apache.org 
Subject: Re: [ANNOUNCE] Experimental Java 21 support now available on master

Thanks Sergey for the great work!

Best,
Rui

On Wed, Nov 29, 2023 at 4:42 PM Leonard Xu  wrote:

> Cool !
>
> Thanks Sergey for the great effort and all involved.
>
>
> Best,
> Leonard
>
> > 2023年11月29日 下午4:31，Swapnal Varma  写道：
> >
> > Congratulations Sergey, and everyone involved!
> >
> > Excited to work with and on this!
> >
> > Best,
> > Swapnal
> >
> >
> > On Wed, 29 Nov 2023, 13:58 Sergey Nuyanzin,  wrote:
> >
> >> The master branch now builds and runs with Java 21 out-of-the-box.
> >>
> >> Notes:
> >> - a nightly cron build was set up.
> >> - In Java 21 builds, Scala is being bumped to 2.12.18
> >> which causes incompatibilities within Flink;
> >> i.e. don't try to load a savepoint from a Java 8/11/17 build
> >> - All the tests that are being skipped on Java 11/17
> >> are also skipped on Java 21.
> >>
> >> Huge shout-out to everyone participating
> >> in review of my Java 21 related PRs
> >>
> >> If you run into any issues, please report it in FLINK-33163
> >>  .
> >>
> >> --
> >> Best regards,
> >> Sergey
> >>
>
>

Re: [VOTE] Release flink-shaded 18.0, release candidate #1

[Matthias Pohl]

+1 (binding)

* Downloaded all resources
* Extracts sources and compilation on these sources
* Diff of git tag checkout with downloaded sources
* Verifies SHA512 checksums & GPG certification
* Checks that all POMs have the right expected version
* Generated diffs to compare pom file changes with NOTICE files: Nothing
suspicious found except for a minor (non-blocking) typo [1]

Thanks for driving this effort, Sergey. :)

[1] https://github.com/apache/flink-shaded/pull/126/files#r1409080162

On Wed, Nov 29, 2023 at 10:25 AM Rui Fan <1996fan...@gmail.com> wrote:

> Sorry, it's non-binding.
>
> On Wed, Nov 29, 2023 at 5:19 PM Rui Fan <1996fan...@gmail.com> wrote:
>
> > Thanks Matthias for the clarification!
> >
> > After I import the latest KEYS, it works fine.
> >
> > +1(binding)
> >
> > - Validated checksum hash
> > - Verified signature
> > - Verified that no binaries exist in the source archive
> > - Build the source with Maven and jdk8
> > - Verified licenses
> > - Verified web PRs, and left a comment
> >
> > Best,
> > Rui
> >
> > On Wed, Nov 29, 2023 at 5:05 PM Matthias Pohl
> >  wrote:
> >
> >> The key is the last key in the KEYS file. It's just having a different
> >> format with spaces being added (due to different gpg versions?): F752
> 9FAE
> >> 2481 1A5C 0DF3  CA74 1596 BBF0 7268 35D8
> >>
> >> On Wed, Nov 29, 2023 at 9:41 AM Rui Fan <1996fan...@gmail.com> wrote:
> >>
> >> > Hey Sergey,
> >> >
> >> > Thank you for driving this release.
> >> >
> >> > I try to check this signature, the whole key is
> >> > F7529FAE24811A5C0DF3CA741596BBF0726835D8,
> >> > it matches your 1596BBF0726835D8, but I cannot
> >> > find it from the Flink KEYS[1].
> >> >
> >> > Please correct me if my operation is wrong, thanks~
> >> >
> >> > [1] https://dist.apache.org/repos/dist/release/flink/KEYS
> >> >
> >> > Best,
> >> > Rui
> >> >
> >> >
> >> > On Wed, Nov 29, 2023 at 6:09 AM Sergey Nuyanzin 
> >> > wrote:
> >> >
> >> > > Hi everyone,
> >> > > Please review and vote on the release candidate #1 for the version
> >> 18.0,
> >> > as
> >> > > follows:
> >> > > [ ] +1, Approve the release
> >> > > [ ] -1, Do not approve the release (please provide specific
> comments)
> >> > >
> >> > >
> >> > > The complete staging area is available for your review, which
> >> includes:
> >> > > * JIRA release notes [1],
> >> > > * the official Apache source release to be deployed to
> >> dist.apache.org
> >> > > [2],
> >> > > which are signed with the key with fingerprint 1596BBF0726835D8 [3],
> >> > > * all artifacts to be deployed to the Maven Central Repository [4],
> >> > > * source code tag "release-18.0-rc1" [5],
> >> > > * website pull request listing the new release [6].
> >> > >
> >> > > The vote will be open for at least 72 hours. It is adopted by
> majority
> >> > > approval, with at least 3 PMC affirmative votes.
> >> > >
> >> > > Thanks,
> >> > > Sergey
> >> > >
> >> > > [1] https://issues.apache.org/jira/projects/FLINK/versions/12353081
> >> > > [2]
> >> https://dist.apache.org/repos/dist/dev/flink/flink-shaded-18.0-rc1
> >> > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> >> > > [4]
> >> > >
> >> https://repository.apache.org/content/repositories/orgapacheflink-1676/
> >> > > [5]
> >> https://github.com/apache/flink-shaded/releases/tag/release-18.0-rc1
> >> > > [6] https://github.com/apache/flink-web/pull/701
> >> > >
> >> >
> >>
> >
>

[DISCUSS] Release flink-connector-parent v1.01

[Etienne Chauchot]

Hi all,

I would like to discuss making a v1.0.1 release of flink-connector-parent.

Since last release, there were only 2 changes:

- https://github.com/apache/flink-connector-shared-utils/pull/19 
(spotless addition)


- https://github.com/apache/flink-connector-shared-utils/pull/26 
(surefire configuration)


The new release would bring the ability to skip some tests in the 
connectors and among other things skip the archunit tests. It is 
important for connectors to skip archunit tests when tested against a 
version of Flink that changes the archunit rules leading to a change of 
the violation store. As there is only one violation store and the 
connector needs to be tested against last 2 minor Flink versions, only 
the version the connector was built against needs to run the archunit 
tests and have them reflected in the violation store.



I volunteer to make the release. As it would be my first ASF release, I 
might require the guidance of one of the PMC members.



Best

Etienne

RE: [VOTE] FLIP-384: Introduce TraceReporter and use it to create checkpointing and recovery traces

[David Radley]

+1(non-binding)

From: Stefan Richter 
Date: Wednesday, 29 November 2023 at 09:44
To: dev@flink.apache.org 
Subject: [EXTERNAL] Re: [VOTE] FLIP-384: Introduce TraceReporter and use it to 
create checkpointing and recovery traces
+1 (binding)

Best,
Stefan


> On 22. Nov 2023, at 11:20, Roman Khachatryan  wrote:
>
> +1 (binding)
>
> Regards,
> Roman
>
> On Wed, Nov 22, 2023, 7:08 AM Zakelly Lan  > wrote:
>
>> +1(non-binding)
>>
>> Best,
>> Zakelly
>>
>> On Wed, Nov 22, 2023 at 3:04 PM Hangxiang Yu  wrote:
>>
>>> +1 (binding)
>>> Thanks for driving this again!
>>>
>>> On Wed, Nov 22, 2023 at 10:30 AM Rui Fan <1996fan...@gmail.com> wrote:
>>>
 +1(binding)

 Best,
 Rui

 On Wed, Nov 22, 2023 at 6:43 AM Jing Ge 
 wrote:

> +1(binding) Thanks!
>
> Best regards,
> Jing
>
> On Tue, Nov 21, 2023 at 6:17 PM Piotr Nowojski >>
> wrote:
>
>> Hi All,
>>
>> I'd like to start a vote on the FLIP-384: Introduce TraceReporter
>> and
 use
>> it to create checkpointing and recovery traces [1]. The discussion
 thread
>> is here [2].
>>
>> The vote will be open for at least 72 hours unless there is an
 objection
> or
>> not enough votes.
>>
>> [1] 
>> https://www.google.com/url?q=https://cwiki.apache.org/confluence/x/TguZE=gmail-imap=170125329000=AOvVaw3QR8LNFApod9Cz_gw2y64w
>> [2]
>> https://www.google.com/url?q=https://lists.apache.org/thread/7lql5f5q1np68fw1wc9trq3d9l2ox8f4=gmail-imap=170125329000=AOvVaw28yzl2wfrtrnoPLsdLW-7q
>>
>>
>> Best,
>> Piotrek
>>
>

>>>
>>>
>>> --
>>> Best,
>>> Hangxiang.

Unless otherwise stated above:

IBM United Kingdom Limited
Registered in England and Wales with number 741598
Registered office: PO Box 41, North Harbour, Portsmouth, Hants. PO6 3AU

Re: [DISCUSSION] flink-connector-shared-utils release process

[Etienne Chauchot]

Thanks Leonard for the review.

Best,

Etienne

Le 29/11/2023 à 08:25, Leonard Xu a écrit :

Thanks Etienne for update the wiki,

I just checked the change by comparing with history version, LGTM


Best,
Leonard


2023年11月28日 下午10:43，Etienne Chauchot  写道：

Hi all,

I just updated the 
doc:https://cwiki.apache.org/confluence/display/FLINK/Externalized+Connector+development

Best

Etienne

Le 27/11/2023 à 12:43, Etienne Chauchot a écrit :

Sure!

Le 23/11/2023 à 02:57, Leonard Xu a écrit :

Thanks Etienne for driving this.


- a flink-connector-shared-utils-*test* clone repo and a 
*io.github.user.flink*:flink-connector-parent custom artifact to be able to 
directly commit and install the artifact in the CI
- a custom ci script that does the cloning and mvn install in the ci.yml github 
action script for testing with the new flink-connector-parent artifact
If people agree on the process and location

+1 for the process and location, could we also add a short paragraph and the 
location link as well in [1] to remind connector developers?

Best,
Leonard

[1]https://cwiki.apache.org/confluence/display/FLINK/Externalized+Connector+development

Re: [VOTE] FLIP-386: Support adding custom metrics in Recovery Spans

[Stefan Richter]

+1 (binding)

Best,
Stefan


> On 23. Nov 2023, at 09:49, Roman Khachatryan  wrote:
> 
> +1 (binding)
> 
> Regards,
> Roman
> 
> 
> On Wed, Nov 22, 2023 at 12:55 PM Rui Fan <1996fan...@gmail.com 
> > wrote:
> 
>> +1(binding)
>> 
>> Thanks for driving this  proposal!
>> 
>> Best,
>> Rui
>> 
>> On Wed, Nov 22, 2023 at 7:44 PM Piotr Nowojski 
>> wrote:
>> 
>>> Hi All,
>>> 
>>> I'd like to start a vote on the FLIP-386: Support adding custom metrics
>> in
>>> Recovery Spans [1]. The discussion thread is here [2].
>>> 
>>> The vote will be open for at least 72 hours unless there is an objection
>> or
>>> not enough votes.
>>> 
>>> [1] 
>>> https://www.google.com/url?q=https://cwiki.apache.org/confluence/x/VAuZE=gmail-imap=170133425800=AOvVaw3oQ0TtRfjerwf5nJ0hQkPA
>>> [2] 
>>> https://www.google.com/url?q=https://lists.apache.org/thread/zt4ykyhv6cco83j9hjngn52b1oprj1tv=gmail-imap=170133425800=AOvVaw0LqLxSwykk0rLpBTXAB0WA

Re: [VOTE] FLIP-385: Add OpenTelemetryTraceReporter and OpenTelemetryMetricReporter

[Stefan Richter]

+1 (binding)

Best,
Stefan


> On 22. Nov 2023, at 12:34, Jing Ge  wrote:
> 
> +1(binding)
> Thanks!
> 
> Best Regards,
> Jing
> 
> On Wed, Nov 22, 2023 at 11:21 AM Roman Khachatryan  > wrote:
> 
>> +1 (binding)
>> 
>> Regards,
>> Roman
>> 
>> On Wed, Nov 22, 2023, 7:30 AM Hangxiang Yu  wrote:
>> 
>>> +1(binding)
>>> 
>>> On Wed, Nov 22, 2023 at 10:29 AM Rui Fan <1996fan...@gmail.com> wrote:
>>> 
 +1(binding)
 
 Best,
 Rui
 
 On Wed, Nov 22, 2023 at 1:20 AM Piotr Nowojski 
 wrote:
 
> Hi All,
> 
> I'd like to start a vote on the FLIP-385: Add
>>> OpenTelemetryTraceReporter
> and OpenTelemetryMetricReporter [1]. The discussion thread is here
>> [2].
> 
> The vote will be open for at least 72 hours unless there is an
>>> objection
 or
> not enough votes.
> 
> [1] 
> https://www.google.com/url?q=https://cwiki.apache.org/confluence/x/UAuZE=gmail-imap=170125778900=AOvVaw2pEEbI5Qe9cEDMKKf77bb-
> [2] 
> https://www.google.com/url?q=https://lists.apache.org/thread/1rqp8czz8wnplpzgn8m4qmzvf14lyx0k=gmail-imap=170125778900=AOvVaw1prt9Abiq4GEXW4zLSIEaW
> 
> 
> Best,
> Piotrek
> 
 
>>> 
>>> 
>>> --
>>> Best,
>>> Hangxiang.

Re: [VOTE] FLIP-384: Introduce TraceReporter and use it to create checkpointing and recovery traces

[Stefan Richter]

+1 (binding)

Best,
Stefan


> On 22. Nov 2023, at 11:20, Roman Khachatryan  wrote:
> 
> +1 (binding)
> 
> Regards,
> Roman
> 
> On Wed, Nov 22, 2023, 7:08 AM Zakelly Lan  > wrote:
> 
>> +1(non-binding)
>> 
>> Best,
>> Zakelly
>> 
>> On Wed, Nov 22, 2023 at 3:04 PM Hangxiang Yu  wrote:
>> 
>>> +1 (binding)
>>> Thanks for driving this again!
>>> 
>>> On Wed, Nov 22, 2023 at 10:30 AM Rui Fan <1996fan...@gmail.com> wrote:
>>> 
 +1(binding)
 
 Best,
 Rui
 
 On Wed, Nov 22, 2023 at 6:43 AM Jing Ge 
 wrote:
 
> +1(binding) Thanks!
> 
> Best regards,
> Jing
> 
> On Tue, Nov 21, 2023 at 6:17 PM Piotr Nowojski >> 
> wrote:
> 
>> Hi All,
>> 
>> I'd like to start a vote on the FLIP-384: Introduce TraceReporter
>> and
 use
>> it to create checkpointing and recovery traces [1]. The discussion
 thread
>> is here [2].
>> 
>> The vote will be open for at least 72 hours unless there is an
 objection
> or
>> not enough votes.
>> 
>> [1] 
>> https://www.google.com/url?q=https://cwiki.apache.org/confluence/x/TguZE=gmail-imap=170125329000=AOvVaw3QR8LNFApod9Cz_gw2y64w
>> [2]
>> https://www.google.com/url?q=https://lists.apache.org/thread/7lql5f5q1np68fw1wc9trq3d9l2ox8f4=gmail-imap=170125329000=AOvVaw28yzl2wfrtrnoPLsdLW-7q
>> 
>> 
>> Best,
>> Piotrek
>> 
> 
 
>>> 
>>> 
>>> --
>>> Best,
>>> Hangxiang.

[jira] [Created] (FLINK-33692) FlinkKafkaProducer could miss super.close

[zhuming (Jira)]

zhuming created FLINK-33692:
---

 Summary: FlinkKafkaProducer could miss super.close
 Key: FLINK-33692
 URL: https://issues.apache.org/jira/browse/FLINK-33692
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Kafka
Affects Versions: 1.17.1, 1.13.6
Reporter: zhuming


 

When flink job restarted 、canceled or failed. It will execute close method. But 
in following FlinkKafkaProducer source code. If flush or close throw 
InterruptedExecption. super.close method must be missed.
{code:java}
@Override
public void close() throws FlinkKafkaException {
// First close the producer for current transaction.
try {
final KafkaTransactionState currentTransaction = currentTransaction();
if (currentTransaction != null) {
// to avoid exceptions on aborting transactions with some pending 
records
flush(currentTransaction);

// normal abort for AT_LEAST_ONCE and NONE do not clean up 
resources because of
// producer reusing, thus
// we need to close it manually
switch (semantic) {
case EXACTLY_ONCE:
break;
case AT_LEAST_ONCE:
case NONE:
currentTransaction.producer.flush();
currentTransaction.producer.close(Duration.ofSeconds(0));
break;
}
}
// If flush() or close() wasinterrupted, super.close might be missed.
super.close();
} catch (Exception e) {
asyncException = ExceptionUtils.firstOrSuppressed(e, asyncException);
} finally {
// We may have to close producer of the current transaction in case 
some exception was
// thrown before
// the normal close routine finishes.
if (currentTransaction() != null) {
try {
currentTransaction().producer.close(Duration.ofSeconds(0));
} catch (Throwable t) {
LOG.warn("Error closing producer.", t);
}
}
// Make sure all the producers for pending transactions are closed.
pendingTransactions()
.forEach(
transaction -> {
try {

transaction.getValue().producer.close(Duration.ofSeconds(0));
} catch (Throwable t) {
LOG.warn("Error closing producer.", t);
}
});
// make sure we propagate pending errors
checkErroneous();
}
} {code}
super.close() method is to execute following code.  It ensures  
'{*}KafkaTransactionState{*}'  released correctlly. 
{code:java}
@Override
public void close() throws Exception {
super.close();

if (currentTransactionHolder != null) {
abort(currentTransactionHolder.handle);
currentTransactionHolder = null;
}
} {code}
 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Release flink-shaded 18.0, release candidate #1

[Rui Fan]

Sorry, it's non-binding.

On Wed, Nov 29, 2023 at 5:19 PM Rui Fan <1996fan...@gmail.com> wrote:

> Thanks Matthias for the clarification!
>
> After I import the latest KEYS, it works fine.
>
> +1(binding)
>
> - Validated checksum hash
> - Verified signature
> - Verified that no binaries exist in the source archive
> - Build the source with Maven and jdk8
> - Verified licenses
> - Verified web PRs, and left a comment
>
> Best,
> Rui
>
> On Wed, Nov 29, 2023 at 5:05 PM Matthias Pohl
>  wrote:
>
>> The key is the last key in the KEYS file. It's just having a different
>> format with spaces being added (due to different gpg versions?): F752 9FAE
>> 2481 1A5C 0DF3  CA74 1596 BBF0 7268 35D8
>>
>> On Wed, Nov 29, 2023 at 9:41 AM Rui Fan <1996fan...@gmail.com> wrote:
>>
>> > Hey Sergey,
>> >
>> > Thank you for driving this release.
>> >
>> > I try to check this signature, the whole key is
>> > F7529FAE24811A5C0DF3CA741596BBF0726835D8,
>> > it matches your 1596BBF0726835D8, but I cannot
>> > find it from the Flink KEYS[1].
>> >
>> > Please correct me if my operation is wrong, thanks~
>> >
>> > [1] https://dist.apache.org/repos/dist/release/flink/KEYS
>> >
>> > Best,
>> > Rui
>> >
>> >
>> > On Wed, Nov 29, 2023 at 6:09 AM Sergey Nuyanzin 
>> > wrote:
>> >
>> > > Hi everyone,
>> > > Please review and vote on the release candidate #1 for the version
>> 18.0,
>> > as
>> > > follows:
>> > > [ ] +1, Approve the release
>> > > [ ] -1, Do not approve the release (please provide specific comments)
>> > >
>> > >
>> > > The complete staging area is available for your review, which
>> includes:
>> > > * JIRA release notes [1],
>> > > * the official Apache source release to be deployed to
>> dist.apache.org
>> > > [2],
>> > > which are signed with the key with fingerprint 1596BBF0726835D8 [3],
>> > > * all artifacts to be deployed to the Maven Central Repository [4],
>> > > * source code tag "release-18.0-rc1" [5],
>> > > * website pull request listing the new release [6].
>> > >
>> > > The vote will be open for at least 72 hours. It is adopted by majority
>> > > approval, with at least 3 PMC affirmative votes.
>> > >
>> > > Thanks,
>> > > Sergey
>> > >
>> > > [1] https://issues.apache.org/jira/projects/FLINK/versions/12353081
>> > > [2]
>> https://dist.apache.org/repos/dist/dev/flink/flink-shaded-18.0-rc1
>> > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
>> > > [4]
>> > >
>> https://repository.apache.org/content/repositories/orgapacheflink-1676/
>> > > [5]
>> https://github.com/apache/flink-shaded/releases/tag/release-18.0-rc1
>> > > [6] https://github.com/apache/flink-web/pull/701
>> > >
>> >
>>
>

Re: [VOTE] Release flink-shaded 18.0, release candidate #1

[Rui Fan]

Thanks Matthias for the clarification!

After I import the latest KEYS, it works fine.

+1(binding)

- Validated checksum hash
- Verified signature
- Verified that no binaries exist in the source archive
- Build the source with Maven and jdk8
- Verified licenses
- Verified web PRs, and left a comment

Best,
Rui

On Wed, Nov 29, 2023 at 5:05 PM Matthias Pohl
 wrote:

> The key is the last key in the KEYS file. It's just having a different
> format with spaces being added (due to different gpg versions?): F752 9FAE
> 2481 1A5C 0DF3  CA74 1596 BBF0 7268 35D8
>
> On Wed, Nov 29, 2023 at 9:41 AM Rui Fan <1996fan...@gmail.com> wrote:
>
> > Hey Sergey,
> >
> > Thank you for driving this release.
> >
> > I try to check this signature, the whole key is
> > F7529FAE24811A5C0DF3CA741596BBF0726835D8,
> > it matches your 1596BBF0726835D8, but I cannot
> > find it from the Flink KEYS[1].
> >
> > Please correct me if my operation is wrong, thanks~
> >
> > [1] https://dist.apache.org/repos/dist/release/flink/KEYS
> >
> > Best,
> > Rui
> >
> >
> > On Wed, Nov 29, 2023 at 6:09 AM Sergey Nuyanzin 
> > wrote:
> >
> > > Hi everyone,
> > > Please review and vote on the release candidate #1 for the version
> 18.0,
> > as
> > > follows:
> > > [ ] +1, Approve the release
> > > [ ] -1, Do not approve the release (please provide specific comments)
> > >
> > >
> > > The complete staging area is available for your review, which includes:
> > > * JIRA release notes [1],
> > > * the official Apache source release to be deployed to dist.apache.org
> > > [2],
> > > which are signed with the key with fingerprint 1596BBF0726835D8 [3],
> > > * all artifacts to be deployed to the Maven Central Repository [4],
> > > * source code tag "release-18.0-rc1" [5],
> > > * website pull request listing the new release [6].
> > >
> > > The vote will be open for at least 72 hours. It is adopted by majority
> > > approval, with at least 3 PMC affirmative votes.
> > >
> > > Thanks,
> > > Sergey
> > >
> > > [1] https://issues.apache.org/jira/projects/FLINK/versions/12353081
> > > [2] https://dist.apache.org/repos/dist/dev/flink/flink-shaded-18.0-rc1
> > > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > [4]
> > >
> https://repository.apache.org/content/repositories/orgapacheflink-1676/
> > > [5]
> https://github.com/apache/flink-shaded/releases/tag/release-18.0-rc1
> > > [6] https://github.com/apache/flink-web/pull/701
> > >
> >
>

Re: [VOTE] Release flink-shaded 18.0, release candidate #1

[Matthias Pohl]

The key is the last key in the KEYS file. It's just having a different
format with spaces being added (due to different gpg versions?): F752 9FAE
2481 1A5C 0DF3  CA74 1596 BBF0 7268 35D8

On Wed, Nov 29, 2023 at 9:41 AM Rui Fan <1996fan...@gmail.com> wrote:

> Hey Sergey,
>
> Thank you for driving this release.
>
> I try to check this signature, the whole key is
> F7529FAE24811A5C0DF3CA741596BBF0726835D8,
> it matches your 1596BBF0726835D8, but I cannot
> find it from the Flink KEYS[1].
>
> Please correct me if my operation is wrong, thanks~
>
> [1] https://dist.apache.org/repos/dist/release/flink/KEYS
>
> Best,
> Rui
>
>
> On Wed, Nov 29, 2023 at 6:09 AM Sergey Nuyanzin 
> wrote:
>
> > Hi everyone,
> > Please review and vote on the release candidate #1 for the version 18.0,
> as
> > follows:
> > [ ] +1, Approve the release
> > [ ] -1, Do not approve the release (please provide specific comments)
> >
> >
> > The complete staging area is available for your review, which includes:
> > * JIRA release notes [1],
> > * the official Apache source release to be deployed to dist.apache.org
> > [2],
> > which are signed with the key with fingerprint 1596BBF0726835D8 [3],
> > * all artifacts to be deployed to the Maven Central Repository [4],
> > * source code tag "release-18.0-rc1" [5],
> > * website pull request listing the new release [6].
> >
> > The vote will be open for at least 72 hours. It is adopted by majority
> > approval, with at least 3 PMC affirmative votes.
> >
> > Thanks,
> > Sergey
> >
> > [1] https://issues.apache.org/jira/projects/FLINK/versions/12353081
> > [2] https://dist.apache.org/repos/dist/dev/flink/flink-shaded-18.0-rc1
> > [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > [4]
> > https://repository.apache.org/content/repositories/orgapacheflink-1676/
> > [5] https://github.com/apache/flink-shaded/releases/tag/release-18.0-rc1
> > [6] https://github.com/apache/flink-web/pull/701
> >
>

[jira] [Created] (FLINK-33691) Support agg push down for 'count(*)/count(1)/count(column not null)'

[Yunhong Zheng (Jira)]

Yunhong Zheng created FLINK-33691:
-

 Summary: Support agg push down for 'count(*)/count(1)/count(column 
not null)'
 Key: FLINK-33691
 URL: https://issues.apache.org/jira/browse/FLINK-33691
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Planner
Affects Versions: 1.19.0
Reporter: Yunhong Zheng
 Fix For: 1.19.0


Now,  PushLocalAggIntoScanRule cannot push down 'count(*)/count(1)/count(column 
not null)', but it can push down count(column nullable). The reason is that 
count(*) and count(n) will be optimized to a scan with calc as '0 AS $f0' to 
reduce read cost, which will not match the push down rule



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] Experimental Java 21 support now available on master

[Rui Fan]

Thanks Sergey for the great work!

Best,
Rui

On Wed, Nov 29, 2023 at 4:42 PM Leonard Xu  wrote:

> Cool !
>
> Thanks Sergey for the great effort and all involved.
>
>
> Best,
> Leonard
>
> > 2023年11月29日 下午4:31，Swapnal Varma  写道：
> >
> > Congratulations Sergey, and everyone involved!
> >
> > Excited to work with and on this!
> >
> > Best,
> > Swapnal
> >
> >
> > On Wed, 29 Nov 2023, 13:58 Sergey Nuyanzin,  wrote:
> >
> >> The master branch now builds and runs with Java 21 out-of-the-box.
> >>
> >> Notes:
> >> - a nightly cron build was set up.
> >> - In Java 21 builds, Scala is being bumped to 2.12.18
> >> which causes incompatibilities within Flink;
> >> i.e. don't try to load a savepoint from a Java 8/11/17 build
> >> - All the tests that are being skipped on Java 11/17
> >> are also skipped on Java 21.
> >>
> >> Huge shout-out to everyone participating
> >> in review of my Java 21 related PRs
> >>
> >> If you run into any issues, please report it in FLINK-33163
> >>  .
> >>
> >> --
> >> Best regards,
> >> Sergey
> >>
>
>

Re: [ANNOUNCE] Experimental Java 21 support now available on master

[Leonard Xu]

Cool ! 

Thanks Sergey for the great effort and all involved.


Best,
Leonard

> 2023年11月29日 下午4:31，Swapnal Varma  写道：
> 
> Congratulations Sergey, and everyone involved!
> 
> Excited to work with and on this!
> 
> Best,
> Swapnal
> 
> 
> On Wed, 29 Nov 2023, 13:58 Sergey Nuyanzin,  wrote:
> 
>> The master branch now builds and runs with Java 21 out-of-the-box.
>> 
>> Notes:
>> - a nightly cron build was set up.
>> - In Java 21 builds, Scala is being bumped to 2.12.18
>> which causes incompatibilities within Flink;
>> i.e. don't try to load a savepoint from a Java 8/11/17 build
>> - All the tests that are being skipped on Java 11/17
>> are also skipped on Java 21.
>> 
>> Huge shout-out to everyone participating
>> in review of my Java 21 related PRs
>> 
>> If you run into any issues, please report it in FLINK-33163
>>  .
>> 
>> --
>> Best regards,
>> Sergey
>> 

Re: [VOTE] Release flink-shaded 18.0, release candidate #1

[Rui Fan]

Hey Sergey,

Thank you for driving this release.

I try to check this signature, the whole key is
F7529FAE24811A5C0DF3CA741596BBF0726835D8,
it matches your 1596BBF0726835D8, but I cannot
find it from the Flink KEYS[1].

Please correct me if my operation is wrong, thanks~

[1] https://dist.apache.org/repos/dist/release/flink/KEYS

Best,
Rui


On Wed, Nov 29, 2023 at 6:09 AM Sergey Nuyanzin  wrote:

> Hi everyone,
> Please review and vote on the release candidate #1 for the version 18.0, as
> follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
>
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release to be deployed to dist.apache.org
> [2],
> which are signed with the key with fingerprint 1596BBF0726835D8 [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag "release-18.0-rc1" [5],
> * website pull request listing the new release [6].
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Thanks,
> Sergey
>
> [1] https://issues.apache.org/jira/projects/FLINK/versions/12353081
> [2] https://dist.apache.org/repos/dist/dev/flink/flink-shaded-18.0-rc1
> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> [4]
> https://repository.apache.org/content/repositories/orgapacheflink-1676/
> [5] https://github.com/apache/flink-shaded/releases/tag/release-18.0-rc1
> [6] https://github.com/apache/flink-web/pull/701
>

Re: [ANNOUNCE] Experimental Java 21 support now available on master

[Swapnal Varma]

Congratulations Sergey, and everyone involved!

Excited to work with and on this!

Best,
Swapnal


On Wed, 29 Nov 2023, 13:58 Sergey Nuyanzin,  wrote:

> The master branch now builds and runs with Java 21 out-of-the-box.
>
> Notes:
> - a nightly cron build was set up.
> - In Java 21 builds, Scala is being bumped to 2.12.18
> which causes incompatibilities within Flink;
> i.e. don't try to load a savepoint from a Java 8/11/17 build
> - All the tests that are being skipped on Java 11/17
> are also skipped on Java 21.
>
> Huge shout-out to everyone participating
> in review of my Java 21 related PRs
>
> If you run into any issues, please report it in FLINK-33163
>  .
>
> --
> Best regards,
> Sergey
>

[ANNOUNCE] Experimental Java 21 support now available on master

[Sergey Nuyanzin]

The master branch now builds and runs with Java 21 out-of-the-box.

Notes:
- a nightly cron build was set up.
- In Java 21 builds, Scala is being bumped to 2.12.18
which causes incompatibilities within Flink;
i.e. don't try to load a savepoint from a Java 8/11/17 build
- All the tests that are being skipped on Java 11/17
are also skipped on Java 21.

Huge shout-out to everyone participating
in review of my Java 21 related PRs

If you run into any issues, please report it in FLINK-33163
 .

-- 
Best regards,
Sergey