Re: [DISCUSS] The performance of serializerHeavyString starts regress since April 3

[Sam Barker]

> I guess that improvement is a fluctuation. You can double check the
performance results[1] of the last few days. The performance isn't
recovered.

Hmm yeah the improvement was a fluctuation and smaller than I remembered
seeing (maybe I had zoomed into the timeline too much).

> I fixed an issue related to kryo serialization in FLINK-35215. IIUC,
serializerHeavyString doesn't use the kryo serialization. I try to
run serializerHeavyString demo locally, and didn't see the
kryo serialization related code is called.

I don't see it either, but then again I don't see commons-io in the call
stacks either despite the regression...

I'm continuing to investigate the regression.

On Mon, 27 May 2024 at 20:15, Rui Fan <1996fan...@gmail.com> wrote:

> Thanks Sam for the comment!
>
> > It looks like the most recent run of JDK 11 saw a big improvement of the
> > performance of the test.
>
> I guess that improvement is a fluctuation. You can double check the
> performance results[1] of the last few days. The performance isn't
> recovered.
>



>
> > That improvement seems related to which is a fix for FLINK-35215.
>
> I fixed an issue related to kryo serialization in FLINK-35215. IIUC,
> serializerHeavyString doesn't use the kryo serialization. I try to
> run serializerHeavyString demo locally, and didn't see the
> kryo serialization related code is called.
>
> Please correct me if I'm wrong, thanks~
>
> [1]
>
> http://flink-speed.xyz/timeline/#/?exe=6=serializerHeavyString=on=on=off=3=200
>
> Best,
> Rui
>
> On Thu, May 23, 2024 at 1:27 PM Sam Barker  wrote:
>
> > It looks like the most recent run of JDK 11 saw a big improvement[1] of
> the
> > performance of the test. That improvement seems related to [2] which is a
> > fix for FLINK-35215 [3]. That suggests to me that the test isn't as
> > isolated to the performance of the code its trying to test as would be
> > ideal. However I've only just started looking at the test suite and
> trying
> > to run locally so I'm not very well placed to judge.
> >
> > It does however suggest that this shouldn't be a blocker for the release.
> >
> >
> >
> > [1] http://flink-speed.xyz/changes/?rev=c1baf07d76=6=3
> > [2]
> >
> >
> https://github.com/apache/flink/commit/c1baf07d7601a683f42997dc35dfaef4e41bc928
> > [3] https://issues.apache.org/jira/browse/FLINK-35215
> >
> > On Wed, 22 May 2024 at 00:15, Piotr Nowojski 
> wrote:
> >
> > > Hi,
> > >
> > > Given what you wrote, that you have investigated the issue and couldn't
> > > find any easy explanation, I would suggest closing this ticket as
> "Won't
> > > do" or "Can not reproduce" and ignoring the problem.
> > >
> > > In the past there have been quite a bit of cases where some benchmark
> > > detected a performance regression. Sometimes those can not be
> reproduced,
> > > other times (as it's the case here), some seemingly unrelated change is
> > > causing the regression. The same thing happened in this benchmark many
> > > times in the past [1], [2], [3], [4]. Generally speaking this benchmark
> > has
> > > been in the spotlight a couple of times [5].
> > >
> > > Note that there have been cases where this benchmark did detect a
> > > performance regression :)
> > >
> > > My personal suspicion is that after that commons-io version bump,
> > > something poked JVM/JIT to compile the code a bit differently for
> string
> > > serialization causing this regression. We have a couple of benchmarks
> > that
> > > seem to be prone to such semi intermittent issues. For example the same
> > > benchmark was subject to this annoying pattern [6], that I've spotted
> in
> > > quite a bit of benchmarks over the years [6]:
> > >
> > > [image: image.png]
> > > (https://imgur.com/a/AoygmWS)
> > >
> > > Where benchmark results are very stable within a single JVM fork. But
> > > between two forks, they can reach two different "stable" levels. Here
> it
> > > looks like 50% of the chance of getting stable "200 records/ms" and 50%
> > > chances of "250 records/ms".
> > >
> > > A small interlude. Each of our benchmarks run in 3 different JVM forks,
> > 10
> > > warm up iterations and 10 measurement iterations. Each iteration
> > > lasts/invokes the benchmarking method at least for one second. So by
> > "very
> > > stable" results, I mean that for example after the 2nd or 3rd warm up
> > > iteration, the results stabilize < +/-1%, and stay on that level for
> the
> > > whole duration of the fork.
> > >
> > > Given that we are repeating the same benchmark in 3 different forks, we
> > > can have by pure chance:
> > > - 3 slow fork - total average 200 records/ms
> > > - 2 slow fork, 1 fast fork - average 216 r/ms
> > > - 1 slow fork, 2 fast forks - average 233 r/ms
> > > - 3 fast forks - average 250 r/ms
> > >
> > > So this benchmark is susceptible to enter some different semi stable
> > > states. As I wrote above, I guess something with the commons-io version
> > > bump just swayed it to a different semi stable state :( I have never
> > gotten
> > > 

Re: [DISCUSS] Add Flink CDC Channel to Apache Flink Slack Workspace

[Hang Ruan]

Hi, zhongqiang.

Thanks for the proposal. +1 for it.

Best,
Hang

Leonard Xu  于2024年5月28日周二 11:58写道：

>
> Thanks Zhongqiang for the proposal, we need the Channel and I should have
> been created it but not yet, +1 from my side.
>
> Best,
> Leonard
>
> > 2024年5月28日 上午11:54，gongzhongqiang  写道：
> >
> > Hi devs,
> >
> > I would like to propose adding a dedicated Flink CDC channel to the
> Apache
> > Flink Slack workspace.
> >
> > Creating a channel focused on Flink CDC will help community members
> easily
> > find previous discussions
> > and target new discussions and questions to the correct place. Flink CDC
> is
> > a sufficiently distinct component
> > within the Apache Flink ecosystem, and having a dedicated channel will
> make
> > it viable and useful for
> > those specifically working with or interested in this technology.
> >
> > Looking forward to your feedback and support on this proposal.
> >
> >
> > Best,
> > Zhongqiang Gong
>
>

Re: [DISCUSS] Flink CDC 3.1.1 Release

[ConradJam]

+1 Thanks discussion

Leonard Xu  于2024年5月29日周三 09:45写道：

> Thanks XIqian for kicking off the discussion, +1 from my side.
>
> Best,
> Leonard
>
>
> > 2024年5月28日 下午7:43，Xiqian YU  写道：
> >
> > Hi devs,
> >
> > I would like to make a proposal about creating a new Flink CDC 3.1 patch
> release (3.1.1). It’s been a week since the last CDC version 3.1.0 got
> released [1], and since then, 7 tickets have been closed, 4 of them are of
> high priority.
> >
> > Currently, there are 5 items open at the moment: 1 of them is a blocker,
> which stops users from restoring with existed checkpoints after upgrading
> [2]. There’s a PR ready and will be merged soon. Other 4 of them have
> approved PRs, and will be merged soon [3][4][5][6]. I propose that a patch
> version could be released after all pending tickets closed.
> >
> > Please reply if there are any unresolved blocking issues you’d like to
> include in this release.
> >
> > Regards,
> > Xiqian
> >
> > [1]
> https://flink.apache.org/2024/05/17/apache-flink-cdc-3.1.0-release-announcement/
> > [2] https://issues.apache.org/jira/browse/FLINK-35464
> > [3] https://issues.apache.org/jira/browse/FLINK-35149
> > [4] https://issues.apache.org/jira/browse/FLINK-35323
> > [5] https://issues.apache.org/jira/browse/FLINK-35430
> > [6] https://issues.apache.org/jira/browse/FLINK-35447
> >
>
>

-- 
Best

ConradJam

[jira] [Created] (FLINK-35478) Pipeline sink Unified primary.key parameter

[melin (Jira)]

melin created FLINK-35478:
-

 Summary: Pipeline sink Unified primary.key parameter
 Key: FLINK-35478
 URL: https://issues.apache.org/jira/browse/FLINK-35478
 Project: Flink
  Issue Type: Improvement
Reporter: melin


Customize the primary key of the sink table. For example, if multiple different 
area tables write to the same paimon table, the primary key must be combined to 
avoid duplicate primary keys



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[DISCUSS] FLIP-460: Display source/sink I/O metrics on Flink Web UI

[Zhanghao Chen]

Hi all,

I'd like start a discussion on FLIP-460: Display source/sink I/O metrics on 
Flink Web UI [1].

Currently, the numRecordsIn & numBytesIn metrics for sources and the 
numRecordsOut & numBytesOut metrics for sinks are always 0 on the Flink web 
dashboard. It is especially confusing for simple ETL jobs where there's a 
single chained operator with 0 input rate and 0 output rate. For years, Flink 
newbies have been asking "Why my job has zero consumption rate and zero 
production rate, is it actually working?"

Connectors implementing FLIP-33 [2] have already exposed these metrics on the 
operator level, this FLIP takes a further step to expose them on the job 
overview page on Flink Web UI.

Looking forward to everyone's feedback and suggestions. Thanks!

[1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=309496355
[2] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-33%3A+Standardize+Connector+Metrics

Best,
Zhanghao Chen

Re: [DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

[Yuxin Tan]

Hi, Xintong,

>  I think we can also publish the prototype codes so the
community can better understand and help with it.

Ok, I agree on the point. I will prepare and publish the code
recently.

Rui,

> Kindly reminder: the image of CIP-6[1] cannot be loaded.

Thanks for the reminder. I've updated the images.


Best,
Yuxin


Rui Fan <1996fan...@gmail.com> 于2024年5月29日周三 09:33写道：

> Thanks Yuxin for driving this proposal!
>
> Kindly reminder: the image of CIP-6[1] cannot be loaded.
>
> [1]
>
> https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn
>
> Best,
> Rui
>
> On Wed, May 29, 2024 at 9:03 AM Xintong Song 
> wrote:
>
> > +1 for this proposal.
> >
> > We have been prototyping this feature internally at Alibaba for a couple
> of
> > months. Yuxin, I think we can also publish the prototype codes so the
> > community can better understand and help with it.
> >
> > Best,
> >
> > Xintong
> >
> >
> >
> > On Tue, May 28, 2024 at 8:34 PM Yuxin Tan 
> wrote:
> >
> > > Hi all,
> > >
> > > I would like to start a discussion on FLIP-459 Support Flink hybrid
> > shuffle
> > > integration with
> > > Apache Celeborn[1]. Flink hybrid shuffle supports transitions between
> > > memory, disk, and
> > > remote storage to improve performance and job stability. Concurrently,
> > > Apache Celeborn
> > > provides a stable, performant, scalable remote shuffle service. This
> > > integration proposal is to
> > > harness the benefits from both hybrid shuffle and Celeborn
> > simultaneously.
> > >
> > > Note that this proposal has two parts.
> > > 1. The Flink-side modifications are in FLIP-459[1].
> > > 2. The Celeborn-side changes are in CIP-6[2].
> > >
> > > Looking forward to everyone's feedback and suggestions. Thank you!
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-459%3A+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn
> > > [2]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn
> > >
> > > Best,
> > > Yuxin
> > >
> >
>

Re: [DISCUSS] Flink CDC 3.1.1 Release

[Leonard Xu]

Thanks XIqian for kicking off the discussion, +1 from my side.

Best,
Leonard


> 2024年5月28日 下午7:43，Xiqian YU  写道：
> 
> Hi devs,
> 
> I would like to make a proposal about creating a new Flink CDC 3.1 patch 
> release (3.1.1). It’s been a week since the last CDC version 3.1.0 got 
> released [1], and since then, 7 tickets have been closed, 4 of them are of 
> high priority.
> 
> Currently, there are 5 items open at the moment: 1 of them is a blocker, 
> which stops users from restoring with existed checkpoints after upgrading 
> [2]. There’s a PR ready and will be merged soon. Other 4 of them have 
> approved PRs, and will be merged soon [3][4][5][6]. I propose that a patch 
> version could be released after all pending tickets closed.
> 
> Please reply if there are any unresolved blocking issues you’d like to 
> include in this release.
> 
> Regards,
> Xiqian
> 
> [1] 
> https://flink.apache.org/2024/05/17/apache-flink-cdc-3.1.0-release-announcement/
> [2] https://issues.apache.org/jira/browse/FLINK-35464
> [3] https://issues.apache.org/jira/browse/FLINK-35149
> [4] https://issues.apache.org/jira/browse/FLINK-35323
> [5] https://issues.apache.org/jira/browse/FLINK-35430
> [6] https://issues.apache.org/jira/browse/FLINK-35447
> 

Re: [DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

[Rui Fan]

Thanks Yuxin for driving this proposal!

Kindly reminder: the image of CIP-6[1] cannot be loaded.

[1]
https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn

Best,
Rui

On Wed, May 29, 2024 at 9:03 AM Xintong Song  wrote:

> +1 for this proposal.
>
> We have been prototyping this feature internally at Alibaba for a couple of
> months. Yuxin, I think we can also publish the prototype codes so the
> community can better understand and help with it.
>
> Best,
>
> Xintong
>
>
>
> On Tue, May 28, 2024 at 8:34 PM Yuxin Tan  wrote:
>
> > Hi all,
> >
> > I would like to start a discussion on FLIP-459 Support Flink hybrid
> shuffle
> > integration with
> > Apache Celeborn[1]. Flink hybrid shuffle supports transitions between
> > memory, disk, and
> > remote storage to improve performance and job stability. Concurrently,
> > Apache Celeborn
> > provides a stable, performant, scalable remote shuffle service. This
> > integration proposal is to
> > harness the benefits from both hybrid shuffle and Celeborn
> simultaneously.
> >
> > Note that this proposal has two parts.
> > 1. The Flink-side modifications are in FLIP-459[1].
> > 2. The Celeborn-side changes are in CIP-6[2].
> >
> > Looking forward to everyone's feedback and suggestions. Thank you!
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-459%3A+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn
> > [2]
> >
> >
> https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn
> >
> > Best,
> > Yuxin
> >
>

Re: [DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

[Xintong Song]

+1 for this proposal.

We have been prototyping this feature internally at Alibaba for a couple of
months. Yuxin, I think we can also publish the prototype codes so the
community can better understand and help with it.

Best,

Xintong



On Tue, May 28, 2024 at 8:34 PM Yuxin Tan  wrote:

> Hi all,
>
> I would like to start a discussion on FLIP-459 Support Flink hybrid shuffle
> integration with
> Apache Celeborn[1]. Flink hybrid shuffle supports transitions between
> memory, disk, and
> remote storage to improve performance and job stability. Concurrently,
> Apache Celeborn
> provides a stable, performant, scalable remote shuffle service. This
> integration proposal is to
> harness the benefits from both hybrid shuffle and Celeborn simultaneously.
>
> Note that this proposal has two parts.
> 1. The Flink-side modifications are in FLIP-459[1].
> 2. The Celeborn-side changes are in CIP-6[2].
>
> Looking forward to everyone's feedback and suggestions. Thank you!
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-459%3A+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn
> [2]
>
> https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn
>
> Best,
> Yuxin
>

[jira] [Created] (FLINK-35477) Pulsar connector resets existing subscription

[Igor Basov (Jira)]

Igor Basov created FLINK-35477:
--

 Summary: Pulsar connector resets existing subscription
 Key: FLINK-35477
 URL: https://issues.apache.org/jira/browse/FLINK-35477
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Pulsar
Affects Versions: 1.18.1, 1.19.0
 Environment: Flink version: 1.18.1

Pulsar connector version: 4.1.0-1.18

Pulsar version: 3.1.0
Reporter: Igor Basov


 

The issue occurs in the following circumstances:
 * There is an existing subscription in a Pulsar topic, and it has some 
accumulated backlog;
 * Flink job is deployed from a clear state (no checkpoints)
 * Flink job uses the same subscription name as the existing one; the start 
cursor is the default one (earliest)
 * {{pulsar.source.resetSubscriptionCursor}} is not set (default: false)

 

*Expected behaviour*

Based on the docs 
[here|https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/connectors/datastream/pulsar/#starting-position],
 the priority for setting up the cursor position should be: {{{}checkpoint > 
existed subscription position > StartCursor{}}}. So, since there are no 
checkpoints, the job should get the existing position from Pulsar and start 
reading from there.

 

*Observed behaviour*

As soon as the job is connected to the topic, the number of messages in the 
subscription backlog jumps to a new high, and JM logs show messages:
{code:java}
Seeking subscription to the message -1:-1:-1
Successfully reset subscription to the message -1:-1:-1 {code}
Apparently, Flink ignored the existing subscription position and reset its 
cursor position to the earliest in the topic.

 

*Further observations*

The related code seems to be 
[here|https://github.com/apache/flink-connector-pulsar/blob/b37a8b32f30683664ff25888d403c4de414043e1/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/source/enumerator/PulsarSourceEnumerator.java#L223].

I believe the breaking changes were introduced in this 
[commit:|https://github.com/apache/flink-connector-pulsar/commit/78d00ea9e3e278d4ce2fbb0c8a8d380abef7b858#]
 * The check if the subscription already exists is removed 
[here|https://github.com/apache/flink-connector-pulsar/commit/78d00ea9e3e278d4ce2fbb0c8a8d380abef7b858#diff-ce7a6c1d29387077c2b19992312c0120bd16580ba5cf9bf222c718dd18a0db2aL86]
 * The check for {{isResetSubscriptionCursor()}} is removed 
[here|https://github.com/apache/flink-connector-pulsar/commit/78d00ea9e3e278d4ce2fbb0c8a8d380abef7b858#diff-4db00b10562cef1def73b06f0e2765a650c51954b4cf13487984204495d8a776L231].

I was able to confirm that it works as expected if I downgrade connector to 
{{4.0.0-1.17.}}

This issue will be blocking us from upgrading to Flink 1.18 and later versions.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Release flink-connector-opensearch v1.2.0, release candidate #1

[gongzhongqiang]

+1(non-binding)

- Verified signatures and hash sums
- Reviewed the web PR
- Built from source code with JDK 1.8 on Ubuntu 22.04
- Checked release notes

Best,
Zhongqiang Gong


Sergey Nuyanzin  于2024年5月16日周四 06:03写道：

> Hi everyone,
> Please review and vote on release candidate #1 for
> flink-connector-opensearch v1.2.0, as follows:
> [ ] +1, Approve the release
> [ ] -1, Do not approve the release (please provide specific comments)
>
>
> The complete staging area is available for your review, which includes:
> * JIRA release notes [1],
> * the official Apache source release to be deployed to dist.apache.org
> [2],
> which are signed with the key with fingerprint
> F7529FAE24811A5C0DF3CA741596BBF0726835D8 [3],
> * all artifacts to be deployed to the Maven Central Repository [4],
> * source code tag v1.2.0-rc1 [5],
> * website pull request listing the new release [6].
> * CI build of the tag [7].
>
> The vote will be open for at least 72 hours. It is adopted by majority
> approval, with at least 3 PMC affirmative votes.
>
> Note that this release is for Opensearch v1.x
>
> Thanks,
> Release Manager
>
> [1] https://issues.apache.org/jira/projects/FLINK/versions/12353812
> [2]
>
> https://dist.apache.org/repos/dist/dev/flink/flink-connector-opensearch-1.2.0-rc1
> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> [4] https://repository.apache.org/content/repositories/orgapacheflink-1734
> [5]
>
> https://github.com/apache/flink-connector-opensearch/releases/tag/v1.2.0-rc1
> [6] https://github.com/apache/flink-web/pull/740
> [7]
>
> https://github.com/apache/flink-connector-opensearch/actions/runs/9102334125
>

Re: [DISCUSS] Merge "flink run" and "flink run-application" in Flink 2.0

[Ferenc Csaky]

Thank you Xintong for your input.

I prepared a FLIP for this change [1], looking forward for any
other opinions.

Thanks,
Ferenc

[1] 
https://docs.google.com/document/d/1EX74rFp9bMKdfoGkz1ASOM6Ibw32rRxIadX72zs2zoY/edit?usp=sharing



On Friday, 17 May 2024 at 07:04, Xintong Song  wrote:

> 
> 
> AFAIK, the main purpose of having `run-application` was to make sure
> the user is aware that application mode is used, which executes the main
> method of the user program in JM rather than in client. This was important
> at the time application mode was first introduced, but maybe not that
> important anymore, given that per-job mode is deprecated and likely removed
> in 2.0. Therefore, +1 for the proposal.
> 
> Best,
> 
> Xintong
> 
> 
> 
> On Thu, May 16, 2024 at 11:35 PM Ferenc Csaky ferenc.cs...@pm.me.invalid
> 
> wrote:
> 
> > Hello devs,
> > 
> > I saw quite some examples when customers were confused about run, and run-
> > application in the Flink CLI and I was wondering about the necessity of
> > deploying
> > Application Mode (AM) jobs with a different command, than Session and
> > Per-Job mode jobs.
> > 
> > I can see a point that YarnDeploymentTarget [1] and
> > KubernetesDeploymentTarget
> > [2] are part of their own maven modules and not known in flink-clients,
> > so the
> > deployment mode validations are happening during cluster deployment in
> > their specific
> > ClusterDescriptor implementation [3]. Although these are implementation
> > details that
> > IMO should not define user-facing APIs.
> > 
> > The command line setup is the same for both run and run-application, so
> > I think there
> > is a quite simple way to achieve a unified flink run experience, but I
> > might missed
> > something so I would appreciate any inputs on this topic.
> > 
> > Based on my assumptions I think it would be possible to deprecate the run-
> > application in Flink 1.20 and remove it completely in Flink 2.0. I
> > already put together a
> > PoC [4], and I was able to deploy AM jobs like this:
> > 
> > flink run --target kubernetes-application ...
> > 
> > If others also agree with this, I would be happy to open a FLIP. WDYT?
> > 
> > Thanks,
> > Ferenc
> > 
> > [1]
> > https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/configuration/YarnDeploymentTarget.java
> > [2]
> > https://github.com/apache/flink/blob/master/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/configuration/KubernetesDeploymentTarget.java
> > [3]
> > https://github.com/apache/flink/blob/48e5a39c9558083afa7589d2d8b054b625f61ee9/flink-kubernetes/src/main/java/org/apache/flink/kubernetes/KubernetesClusterDescriptor.java#L206
> > [4]
> > https://github.com/ferenc-csaky/flink/commit/40b3e1b998c7a4273eaaff71d9162c9f1ee039c0

[RESULT][VOTE] FLIP-443: Interruptible watermark processing

[Piotr Nowojski]

Hi all!

I'm happy to announce that FLIP-443 [1] has been accepted [2]. There were 7
votes in favour, all binding:

- Zakelly
- Yanfei Lei
- Rui Fan
- Wejie Guo
- Martijn Visser
- Stefan Richter
- Piotr Nowojski

[1] https://cwiki.apache.org/confluence/x/qgn9EQ
[2] https://lists.apache.org/thread/tf4hrfy4o3flk0f4zqlrbln390n6w4yq

Re: [VOTE] FLIP-443: Interruptible watermark processing

[Piotr Nowojski]

+1 (binding) from my side as well.

Voting is closed. Thank you all for voting.

Best,
Piotrek

pon., 27 maj 2024 o 11:37 Stefan Richter 
napisał(a):

>
> +1 (binding)
>
>
>
> > On 24. May 2024, at 09:59, Martijn Visser 
> wrote:
> >
> > +1 (binding)
> >
> > On Fri, May 24, 2024 at 7:31 AM weijie guo  >
> > wrote:
> >
> >> +1(binding)
> >>
> >> Thanks for driving this!
> >>
> >> Best regards,
> >>
> >> Weijie
> >>
> >>
> >> Rui Fan <1996fan...@gmail.com> 于2024年5月24日周五 13:03写道：
> >>
> >>> +1(binding)
> >>>
> >>> Best,
> >>> Rui
> >>>
> >>> On Fri, May 24, 2024 at 12:01 PM Yanfei Lei 
> wrote:
> >>>
>  Thanks for driving this!
> 
>  +1 (binding)
> 
>  Best,
>  Yanfei
> 
>  Zakelly Lan  于2024年5月24日周五 10:13写道：
> 
> >
> > +1 (binding)
> >
> > Best,
> > Zakelly
> >
> > On Thu, May 23, 2024 at 8:21 PM Piotr Nowojski  >>>
>  wrote:
> >
> >> Hi all,
> >>
> >> After reaching what looks like a consensus in the discussion thread
>  [1], I
> >> would like to put FLIP-443 [2] to the vote.
> >>
> >> The vote will be open for at least 72 hours unless there is an
>  objection or
> >> insufficient votes.
> >>
> >> [1]
> >>
> https://www.google.com/url?q=https://lists.apache.org/thread/flxm7rphvfgqdn2gq2z0bb7kl007olpz=gmail-imap=171714246900=AOvVaw1sxqcTTJfXbE_qaBA0l1FH
> >> [2]
> https://www.google.com/url?q=https://cwiki.apache.org/confluence/x/qgn9EQ=gmail-imap=171714246900=AOvVaw3yQ55VLWPxkY2OHXf0k72Q
> >>
> >> Bets,
> >> Piotrek
>
>

[DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

[Yuxin Tan]

Hi all,

I would like to start a discussion on FLIP-459 Support Flink hybrid shuffle
integration with
Apache Celeborn[1]. Flink hybrid shuffle supports transitions between
memory, disk, and
remote storage to improve performance and job stability. Concurrently,
Apache Celeborn
provides a stable, performant, scalable remote shuffle service. This
integration proposal is to
harness the benefits from both hybrid shuffle and Celeborn simultaneously.

Note that this proposal has two parts.
1. The Flink-side modifications are in FLIP-459[1].
2. The Celeborn-side changes are in CIP-6[2].

Looking forward to everyone's feedback and suggestions. Thank you!

[1]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-459%3A+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn
[2]
https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn

Best,
Yuxin

Re: [DISCUSS] Connector releases for Flink 1.19

[Jing Ge]

Hi,

Thanks Danny for driving it! Do we have an umbrella ticket for Flink 1.19
connectors release?

@Sergei
Thanks for the hint wrt JDBC connector. Where could users know that it
already supports 1.19?

Best regards,
Jing

On Fri, May 17, 2024 at 4:07 AM Sergey Nuyanzin  wrote:

> >, it looks like opensearch-2.0.0 has been created now, all good.
> yep, thanks to Martijn
>
> I've created RCs for Opensearch connector
>
> On Tue, May 14, 2024 at 12:38 PM Danny Cranmer 
> wrote:
>
> > Hello,
> >
> > @Sergey Nuyanzin , it looks like opensearch-2.0.0
> > has been created now, all good.
> >
> > @Hongshun Wang, thanks, since the CDC connectors are not yet released I
> > had omitted them from this task. But happy to include them, thanks for
> the
> > support.
> >
> > Thanks,
> > Danny
> >
> > On Mon, May 13, 2024 at 3:40 AM Hongshun Wang 
> > wrote:
> >
> >> Hello Danny,
> >> Thanks for pushing this forward.  I am available to assist with the CDC
> >> connector[1].
> >>
> >> [1] https://github.com/apache/flink-cdc
> >>
> >> Best
> >> Hongshun
> >>
> >> On Sun, May 12, 2024 at 8:48 PM Sergey Nuyanzin 
> >> wrote:
> >>
> >> > I'm in a process of preparation of RC for OpenSearch connector
> >> >
> >> > however it seems I need PMC help: need to create opensearch-2.0.0 on
> >> jira
> >> > since as it was proposed in another ML[1] to have 1.x for OpenSearch
> >> > v1 and 2.x for OpenSearch v2
> >> >
> >> > would be great if someone from PMC could help here
> >> >
> >> > [1] https://lists.apache.org/thread/3w1rnjp5y612xy5k9yv44hy37zm9ph15
> >> >
> >> > On Wed, Apr 17, 2024 at 12:42 PM Ferenc Csaky
> >> >  wrote:
> >> > >
> >> > > Thank you Danny and Sergey for pushing this!
> >> > >
> >> > > I can help with the HBase connector if necessary, will comment the
> >> > > details to the relevant Jira ticket.
> >> > >
> >> > > Best,
> >> > > Ferenc
> >> > >
> >> > >
> >> > >
> >> > >
> >> > > On Wednesday, April 17th, 2024 at 11:17, Danny Cranmer <
> >> > dannycran...@apache.org> wrote:
> >> > >
> >> > > >
> >> > > >
> >> > > > Hello all,
> >> > > >
> >> > > > I have created a parent Jira to cover the releases [1]. I have
> >> > assigned AWS
> >> > > > and MongoDB to myself and OpenSearch to Sergey. Please assign the
> >> > > > relevant issue to yourself as you pick up the tasks.
> >> > > >
> >> > > > Thanks!
> >> > > >
> >> > > > [1] https://issues.apache.org/jira/browse/FLINK-35131
> >> > > >
> >> > > > On Tue, Apr 16, 2024 at 2:41 PM Muhammet Orazov
> >> > > > mor+fl...@morazow.com.invalid wrote:
> >> > > >
> >> > > > > Thanks Sergey and Danny for clarifying, indeed it
> >> > > > > requires committer to go through the process.
> >> > > > >
> >> > > > > Anyway, please let me know if I can be any help.
> >> > > > >
> >> > > > > Best,
> >> > > > > Muhammet
> >> > > > >
> >> > > > > On 2024-04-16 11:19, Danny Cranmer wrote:
> >> > > > >
> >> > > > > > Hello,
> >> > > > > >
> >> > > > > > I have opened the VOTE thread for the AWS connectors release
> >> [1].
> >> > > > > >
> >> > > > > > > If I'm not mistaking (please correct me if I'm wrong) this
> >> > request is
> >> > > > > > > not
> >> > > > > > > about version update it is about new releases for connectors
> >> > > > > >
> >> > > > > > Yes, correct. If there are any other code changes required
> then
> >> > help
> >> > > > > > would be appreciated.
> >> > > > > >
> >> > > > > > > Are you going to create an umbrella issue for it?
> >> > > > > >
> >> > > > > > We do not usually create JIRA issues for releases. That being
> >> said
> >> > it
> >> > > > > > sounds like a good idea to have one place to track the status
> of
> >> > the
> >> > > > > > connector releases and pre-requisite code changes.
> >> > > > > >
> >> > > > > > > I would like to work on this task, thanks for initiating it!
> >> > > > > >
> >> > > > > > The actual release needs to be performed by a committer.
> >> However,
> >> > help
> >> > > > > > getting the connectors building against Flink 1.19 and testing
> >> the
> >> > RC
> >> > > > > > is
> >> > > > > > appreciated.
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > > Danny
> >> > > > > >
> >> > > > > > [1]
> >> > https://lists.apache.org/thread/0nw9smt23crx4gwkf6p1dd4jwvp1g5s0
> >> > > > > >
> >> > > > > > On Tue, Apr 16, 2024 at 6:34 AM Sergey Nuyanzin
> >> > snuyan...@gmail.com
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > Thanks for volunteering Muhammet!
> >> > > > > > > And thanks Danny for starting the activity.
> >> > > > > > >
> >> > > > > > > If I'm not mistaking (please correct me if I'm wrong)
> >> > > > > > >
> >> > > > > > > this request is not about version update it is about new
> >> > releases for
> >> > > > > > > connectors
> >> > > > > > > btw for jdbc connector support of 1.19 and 1.20-SNAPSHOT is
> >> > already
> >> > > > > > > done
> >> > > > > > >
> >> > > > > > > I would volunteer for Opensearch connector since currently
> I'm
> >> > working
> >> > > > > > > on
> >> > > > > > > support of Opensearch v2
> >> > > > > > > 

[DISCUSS] Flink CDC 3.1.1 Release

[Xiqian YU]

Hi devs,

I would like to make a proposal about creating a new Flink CDC 3.1 patch 
release (3.1.1). It’s been a week since the last CDC version 3.1.0 got released 
[1], and since then, 7 tickets have been closed, 4 of them are of high priority.

Currently, there are 5 items open at the moment: 1 of them is a blocker, which 
stops users from restoring with existed checkpoints after upgrading [2]. 
There’s a PR ready and will be merged soon. Other 4 of them have approved PRs, 
and will be merged soon [3][4][5][6]. I propose that a patch version could be 
released after all pending tickets closed.

Please reply if there are any unresolved blocking issues you’d like to include 
in this release.

Regards,
Xiqian

[1] 
https://flink.apache.org/2024/05/17/apache-flink-cdc-3.1.0-release-announcement/
[2] https://issues.apache.org/jira/browse/FLINK-35464
[3] https://issues.apache.org/jira/browse/FLINK-35149
[4] https://issues.apache.org/jira/browse/FLINK-35323
[5] https://issues.apache.org/jira/browse/FLINK-35430
[6] https://issues.apache.org/jira/browse/FLINK-35447

[jira] [Created] (FLINK-35476) The IF built-in function returns an exception when returning a string

[jianhao.Li (Jira)]

jianhao.Li created FLINK-35476:
--

 Summary: The IF built-in function returns an exception when 
returning a string
 Key: FLINK-35476
 URL: https://issues.apache.org/jira/browse/FLINK-35476
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / API, Table SQL / Runtime
Affects Versions: 1.15.3
Reporter: jianhao.Li
 Attachments: image-2024-05-28-17-04-59-069.png

The IF built-in function returns an exception when returning a string

There is no explanation in the official documentation！

offlical link：

https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/table/functions/systemfunctions/

Flink version：1.15.1/1.15.3

 

CASE SQL:
{code:java}
select 
if('asd'='asd', '1234', '12345' )  as expect_1234
, if('asd'='123', '1234', '12345') as expect_12345
;  {code}
Expected Result：

 
{code:java}
expect_1234 value is 1234
expect_12345 value is 12345{code}
 

 

Real Returend Result( test in sql-client):

!image-2024-05-28-17-04-59-069.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35475) Introduce isInternalSorterSupport to OperatorAttributes

[Xuannan Su (Jira)]

Xuannan Su created FLINK-35475:
--

 Summary: Introduce isInternalSorterSupport to OperatorAttributes
 Key: FLINK-35475
 URL: https://issues.apache.org/jira/browse/FLINK-35475
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Task
Reporter: Xuannan Su


Introduce isInternalSorterSupport to OperatorAttributes to notify Flink whether 
the operator will sort the data internally in batch mode or during backlog.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35474) During the process of catching up after a restart, Flink CDC uses the latest table information to parse old binlogs.

[stevedengzhi (Jira)]

stevedengzhi created FLINK-35474:


 Summary: During the process of catching up after a restart, Flink 
CDC uses the latest table information to parse old binlogs.
 Key: FLINK-35474
 URL: https://issues.apache.org/jira/browse/FLINK-35474
 Project: Flink
  Issue Type: Bug
  Components: Flink CDC
Affects Versions: 1.17.2
Reporter: stevedengzhi
 Attachments: image-2024-05-28-16-13-58-566.png

!image-2024-05-28-16-13-58-566.png!

 

I don't know how to handle it.
This DDL issue can prevent recovery from checkpoints, leading to infinite 
restarts.
How to reproduce this issue?（翻译by chatGPT，English not well ）
Step 1: Turn off Flink CDC.
Step 2: Perform DDL operations on multiple tables, including deleting columns, 
reducing columns, and adding columns.
Step 3: Restore Flink CDC from the last checkpoint before it was turned off.
 
 
 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35473) FLIP-457: Improve Table/SQL Configuration for Flink 2.0

[Jane Chan (Jira)]

Jane Chan created FLINK-35473:
-

 Summary: FLIP-457: Improve Table/SQL Configuration for Flink 2.0
 Key: FLINK-35473
 URL: https://issues.apache.org/jira/browse/FLINK-35473
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / API
Affects Versions: 1.20.0
Reporter: Jane Chan






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[RESULT][VOTE] FLIP-457: Improve Table/SQL Configuration for Flink 2.0

[Jane Chan]

Hi all,

I'm happy to announce that FLIP-457[1] has been accepted.

There are 11 votes, of which 6 are binding[2].

- Benchao Li (binding)
- Leonard Xu (binding)
- Lincoln Lee (binding)
- Jark Wu (binding)
- Rui Fan (binding)
- Weijie Guo (binding)

- Feng Jin (non-binding)
- Hang Ruan (non-binding)
- Xuannan Su (non-binding)
- Yubin Li (non-binding)
- Zhongqiang Gong (non-binding)


[1]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=307136992
[2] https://lists.apache.org/thread/t2dnyrd6wnlmkolpnosy40wmlzps3d58

Best,
Jane

[jira] [Created] (FLINK-35472) Improve tests for Elasticsearch 8 connector

[Mingliang Liu (Jira)]

Mingliang Liu created FLINK-35472:
-

 Summary: Improve tests for Elasticsearch 8 connector
 Key: FLINK-35472
 URL: https://issues.apache.org/jira/browse/FLINK-35472
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / ElasticSearch, Tests
Reporter: Mingliang Liu


Per discussion in [this 
PR|https://github.com/apache/flink-connector-elasticsearch/pull/104],  it makes 
the tests more reusable if we use parameterized tests. It requires some changes 
of the existing tests, which includes:
# Make base test class parameterized with secure parameter. As JUnit 5 has 
limited support for parameterized tests with inheritance, we can use the 
{{ParameterizedTestExtension}} introduced in Flink, see this doc
# Manage the test container lifecycle instead of using the managed annotation 
{{@Testcontainers}} and {{@Container}} so that the test containers can be used 
as a singleton for all tests in the suite
# Create and use common methods in the base class that concrete test classes 
can be mostly parameter-agnostic

This JIRA intends to not change any logic or functionality. Instead it focuses 
on tests refactoring for more reusable tests and future proof.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35471) Kubernetes operator bumps the flink version to 1.19

[Rui Fan (Jira)]

Rui Fan created FLINK-35471:
---

 Summary: Kubernetes operator bumps the flink version to 1.19
 Key: FLINK-35471
 URL: https://issues.apache.org/jira/browse/FLINK-35471
 Project: Flink
  Issue Type: Improvement
  Components: Kubernetes Operator
Reporter: Rui Fan
Assignee: Rui Fan
 Fix For: kubernetes-operator-1.9.0


Kubernetes operator bumps the flink version to 1.19 after 1.19.1 is released.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35470) Flink 1.18.1 and 1.19.0 having build jdk 1.8 and causing incompatibilities with Java 17

[Rajat Pratap Singh (Jira)]

Rajat Pratap Singh created FLINK-35470:
--

 Summary: Flink 1.18.1 and 1.19.0 having build jdk 1.8 and causing 
incompatibilities with Java 17
 Key: FLINK-35470
 URL: https://issues.apache.org/jira/browse/FLINK-35470
 Project: Flink
  Issue Type: Bug
  Components: Flink CDC
Affects Versions: 1.18.1
Reporter: Rajat Pratap Singh


I am writing flink jobs with latest release version for flink (1.18.1). The 
jobmanager is also deployed with the same version build. But we faced issues 
when we deployed the jobs. On further investigation, I noticed all libraries 
from flink have build jdk 1.8. is this the correct version? How to write flink 
jobs with Java 17 code and which library versions to use? Is Java 17 even 
supported on the client side or is it just on the server side support for java 
17?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)