[Discuss] Non-retriable (partial) errors in Elasticsearch 8 connector

[Mingliang Liu]

Hi all,

Currently the Elasticsearch 8 connector retries all items if the request
fails as a whole, and retries failed items if the request has partial
failures [1]. I think this infinitely retries might be problematic in some
cases when retrying can never eventually succeed. For example, if the
request is 400 (bad request) or 404 (not found), retries do not help. If
there are too many failed items non-retriable, new requests will get
processed less effectively. In extreme cases, it may stall the pipeline if
in-flight requests are occupied by those failed items.

FLIP-451 proposes timeout for retrying which helps with un-acknowledged
requests, but not addressing the case when request gets processed and
failed items keep failing no matter how many times we retry. Correct me if
I'm wrong.

One opinionated option is to fail fast for non-retriable errors like 400 /
404 and to drop items for 409. Or we can allow users to configure
"drop/fail" behavior for non-retriable errors. I prefer the latter. I
checked how LogStash ingests data to Elasticsearch and it takes a similar
approach for non-retriable errors [2]. In my day job, we have a
dead-letter-queue in AsynSinkWriter for failed entries that exhaust
retries. I guess that is too specific to our setup and seems an overkill
here for Elasticsearch connector.

Any thoughts on this?

[1]
https://github.com/apache/flink-connector-elasticsearch/blob/5d1f8d03e3cff197ed7fe30b79951e44808b48fe/flink-connector-elasticsearch8/src/main/java/org/apache/flink/connector/elasticsearch/sink/Elasticsearch8AsyncWriter.java#L152-L170
[2]
https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/main/lib/logstash/plugin_mixins/elasticsearch/common.rb#L283-L304

Re: [DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

[Junrui Lee]

Thanks Yuxin for your answer. +1 for this proposal.

Best,
Junrui.

Yuxin Tan  于2024年6月6日周四 13:42写道：

> Thanks Junrui for your question.
>
> > I wonder if the current interface design support the
> future adaptation for batch job recovery
>
> I noticed that FLIP-383 supports batch job recovery by introducing
> some new APIs. These APIs can also be added to the Tier-related
> interfaces to facilitate the feature. Since these modifications are not
> directly related to the current integration tasks and the integration
> does not conflict with the batch job recovery, I propose that this FLIP
> doesn't involve these particular changes. Moreover, considering that
> the Tier interfaces are not public currently, it is also feasible to add
> the interfaces directly if necessary.
> WDYT?
>
> Best,
> Yuxin
>
>
> Junrui Lee  于2024年6月6日周四 11:02写道：
>
> > Thanks Yuxin for driving this proposal!
> >
> > I have a question about the public interface compatibility in the context
> > of FLIP-459. As we've supported batch job recovery from jobMaster
> failures
> > in FLIP-383 which will be released in Flink 1.20. I wonder if the current
> > interface design support the future adaptation for batch job recovery?
> >
> > Looking forward to your feedback.
> >
> > Best,
> > Junrui.
> >
> > weijie guo  于2024年6月5日周三 10:13写道：
> >
> > > Thanks Yuxin for the proposal!
> > >
> > > When we first proposed Hybrid Shuffle, I wanted to support pluggable
> > > storage tier in the future. However, limited by the architecture of the
> > > legacy Hybrid Shuffle at that time, this idea has not been realized.
> The
> > > new architecture abstracts the tier nicely, and now it's time to
> > introduce
> > > support for external storage.
> > >
> > > Big +1 for this one!
> > >
> > > Best regards,
> > >
> > > Weijie
> > >
> > >
> > > rexxiong  于2024年6月5日周三 00:08写道：
> > >
> > > > Thanks Yuxin for the proposal. +1,  as a member of the Apache
> Celeborn
> > > > community, I am very excited about the integration of Flink's Hybrid
> > > > Shuffle with Apache Celeborn. The whole design of CIP-6 looks good to
> > > me. I
> > > > am looking forward to this integration.
> > > >
> > > > Thanks,
> > > > Jiashu Xiong
> > > >
> > > > Ethan Feng  于2024年6月4日周二 16:47写道：
> > > >
> > > > > +1 for this proposal.
> > > > >
> > > > > After internally reviewing the prototype of CIP-6, this would
> improve
> > > > > performance and stability for Flink users using Celeborn.
> > > > >
> > > > > Expect to see this feature come out to the community.
> > > > >
> > > > > As I come from the Celeborn community, I hope more users can try to
> > > > > use Celeborn when there are Flink batch jobs.
> > > > >
> > > > > Thanks,
> > > > > Ethan Feng
> > > > >
> > > > > Yuxin Tan  于2024年6月4日周二 16:34写道：
> > > > > >
> > > > > > Hi, Venkatakrishnan,
> > > > > >
> > > > > > Thanks for joining the discussion. We appreciate your interest
> > > > > > in contributing to the work. Once the FLIP and CIP proposals
> > > > > > have been approved, we will create some JIRA tickets in Flink
> > > > > > and Celeborn projects. Please feel free to take a look at the
> > > > > > tickets and select any that resonate with your interests.
> > > > > >
> > > > > > Best,
> > > > > > Yuxin
> > > > > >
> > > > > >
> > > > > > Venkatakrishnan Sowrirajan  于2024年5月31日周五
> > 23:11写道：
> > > > > >
> > > > > > > Thanks for this FLIP. We are also interested in
> > > learning/contributing
> > > > > to
> > > > > > > the hybrid shuffle integration with celeborn for batch
> > executions.
> > > > > > >
> > > > > > > On Tue, May 28, 2024, 7:07 PM Yuxin Tan <
> tanyuxinw...@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi, Xintong,
> > > > > > > >
> > > > > > > > >  I think we can also publish the prototype codes so the
> > > > > > > > community can better understand and help with it.
> > > > > > > >
> > > > > > > > Ok, I agree on the point. I will prepare and publish the code
> > > > > > > > recently.
> > > > > > > >
> > > > > > > > Rui,
> > > > > > > >
> > > > > > > > > Kindly reminder: the image of CIP-6[1] cannot be loaded.
> > > > > > > >
> > > > > > > > Thanks for the reminder. I've updated the images.
> > > > > > > >
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Yuxin
> > > > > > > >
> > > > > > > >
> > > > > > > > Rui Fan <1996fan...@gmail.com> 于2024年5月29日周三 09:33写道：
> > > > > > > >
> > > > > > > > > Thanks Yuxin for driving this proposal!
> > > > > > > > >
> > > > > > > > > Kindly reminder: the image of CIP-6[1] cannot be loaded.
> > > > > > > > >
> > > > > > > > > [1]
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;KysrKysrKys!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9OJ6v9uGw$
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Rui
> > > > > > 

Re: [DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

[Yuxin Tan]

Thanks Junrui for your question.

> I wonder if the current interface design support the
future adaptation for batch job recovery

I noticed that FLIP-383 supports batch job recovery by introducing
some new APIs. These APIs can also be added to the Tier-related
interfaces to facilitate the feature. Since these modifications are not
directly related to the current integration tasks and the integration
does not conflict with the batch job recovery, I propose that this FLIP
doesn't involve these particular changes. Moreover, considering that
the Tier interfaces are not public currently, it is also feasible to add
the interfaces directly if necessary.
WDYT?

Best,
Yuxin


Junrui Lee  于2024年6月6日周四 11:02写道：

> Thanks Yuxin for driving this proposal!
>
> I have a question about the public interface compatibility in the context
> of FLIP-459. As we've supported batch job recovery from jobMaster failures
> in FLIP-383 which will be released in Flink 1.20. I wonder if the current
> interface design support the future adaptation for batch job recovery?
>
> Looking forward to your feedback.
>
> Best,
> Junrui.
>
> weijie guo  于2024年6月5日周三 10:13写道：
>
> > Thanks Yuxin for the proposal!
> >
> > When we first proposed Hybrid Shuffle, I wanted to support pluggable
> > storage tier in the future. However, limited by the architecture of the
> > legacy Hybrid Shuffle at that time, this idea has not been realized. The
> > new architecture abstracts the tier nicely, and now it's time to
> introduce
> > support for external storage.
> >
> > Big +1 for this one!
> >
> > Best regards,
> >
> > Weijie
> >
> >
> > rexxiong  于2024年6月5日周三 00:08写道：
> >
> > > Thanks Yuxin for the proposal. +1,  as a member of the Apache Celeborn
> > > community, I am very excited about the integration of Flink's Hybrid
> > > Shuffle with Apache Celeborn. The whole design of CIP-6 looks good to
> > me. I
> > > am looking forward to this integration.
> > >
> > > Thanks,
> > > Jiashu Xiong
> > >
> > > Ethan Feng  于2024年6月4日周二 16:47写道：
> > >
> > > > +1 for this proposal.
> > > >
> > > > After internally reviewing the prototype of CIP-6, this would improve
> > > > performance and stability for Flink users using Celeborn.
> > > >
> > > > Expect to see this feature come out to the community.
> > > >
> > > > As I come from the Celeborn community, I hope more users can try to
> > > > use Celeborn when there are Flink batch jobs.
> > > >
> > > > Thanks,
> > > > Ethan Feng
> > > >
> > > > Yuxin Tan  于2024年6月4日周二 16:34写道：
> > > > >
> > > > > Hi, Venkatakrishnan,
> > > > >
> > > > > Thanks for joining the discussion. We appreciate your interest
> > > > > in contributing to the work. Once the FLIP and CIP proposals
> > > > > have been approved, we will create some JIRA tickets in Flink
> > > > > and Celeborn projects. Please feel free to take a look at the
> > > > > tickets and select any that resonate with your interests.
> > > > >
> > > > > Best,
> > > > > Yuxin
> > > > >
> > > > >
> > > > > Venkatakrishnan Sowrirajan  于2024年5月31日周五
> 23:11写道：
> > > > >
> > > > > > Thanks for this FLIP. We are also interested in
> > learning/contributing
> > > > to
> > > > > > the hybrid shuffle integration with celeborn for batch
> executions.
> > > > > >
> > > > > > On Tue, May 28, 2024, 7:07 PM Yuxin Tan 
> > > > wrote:
> > > > > >
> > > > > > > Hi, Xintong,
> > > > > > >
> > > > > > > >  I think we can also publish the prototype codes so the
> > > > > > > community can better understand and help with it.
> > > > > > >
> > > > > > > Ok, I agree on the point. I will prepare and publish the code
> > > > > > > recently.
> > > > > > >
> > > > > > > Rui,
> > > > > > >
> > > > > > > > Kindly reminder: the image of CIP-6[1] cannot be loaded.
> > > > > > >
> > > > > > > Thanks for the reminder. I've updated the images.
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > > Yuxin
> > > > > > >
> > > > > > >
> > > > > > > Rui Fan <1996fan...@gmail.com> 于2024年5月29日周三 09:33写道：
> > > > > > >
> > > > > > > > Thanks Yuxin for driving this proposal!
> > > > > > > >
> > > > > > > > Kindly reminder: the image of CIP-6[1] cannot be loaded.
> > > > > > > >
> > > > > > > > [1]
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;KysrKysrKys!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9OJ6v9uGw$
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Rui
> > > > > > > >
> > > > > > > > On Wed, May 29, 2024 at 9:03 AM Xintong Song <
> > > > tonysong...@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > +1 for this proposal.
> > > > > > > > >
> > > > > > > > > We have been prototyping this feature internally at Alibaba
> > > for a
> > > > > > > couple
> > > > > > > > of
> > > > > > > > > months. Yuxin, I think we can also publish the prototype
> > codes
> 

[jira] [Created] (FLINK-35536) FileSystem sink on S3 produces invalid Avros when compaction is disabled

[Juliusz Nadberezny (Jira)]

Juliusz Nadberezny created FLINK-35536:
--

 Summary: FileSystem sink on S3 produces invalid Avros when 
compaction is disabled
 Key: FLINK-35536
 URL: https://issues.apache.org/jira/browse/FLINK-35536
 Project: Flink
  Issue Type: Bug
  Components: Connectors / FileSystem
Affects Versions: 1.19.0
Reporter: Juliusz Nadberezny


Compaction on FileSystem sink on S3 uses multipart upload process. 
When compaction is disabled after being enabled, the files that where being 
kept by multipart upload and then are "released" with CompleteMultipartUpload 
will be broken.

Broken Avro files seem to have Avro schema duplicated at the beginning of the 
file.

 

Steps to reproduce:
1. Deploy job with FileSystem sink with compaction enabled writing to S3/MinIO.

2. Wait for job to produce some output.

3. Redeploy job with compaction disabled.

4. Wait for multipart upload complete and verify released files.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

[Paul Lam]

Congrats, Rui!

Best,
Paul Lam

> 2024年6月6日 11:02，Junrui Lee  写道：
> 
> Congratulations, Rui.
> 
> Best,
> Junrui
> 
> Hang Ruan  于2024年6月6日周四 10:35写道：
> 
>> Congratulations, Rui!
>> 
>> Best,
>> Hang
>> 
>> Samrat Deb  于2024年6月6日周四 10:28写道：
>> 
>>> Congratulations Rui
>>> 
>>> Bests,
>>> Samrat
>>> 
>>> On Thu, 6 Jun 2024 at 7:45 AM, Yuxin Tan  wrote:
>>> 
 Congratulations, Rui!
 
 Best,
 Yuxin
 
 
 Xuannan Su  于2024年6月6日周四 09:58写道：
 
> Congratulations!
> 
> Best regards,
> Xuannan
> 
> On Thu, Jun 6, 2024 at 9:53 AM Hangxiang Yu 
>>> wrote:
>> 
>> Congratulations, Rui !
>> 
>> On Thu, Jun 6, 2024 at 9:18 AM Lincoln Lee >> 
> wrote:
>> 
>>> Congratulations, Rui!
>>> 
>>> Best,
>>> Lincoln Lee
>>> 
>>> 
>>> Lijie Wang  于2024年6月6日周四 09:11写道：
>>> 
 Congratulations, Rui!
 
 Best,
 Lijie
 
 Rodrigo Meneses  于2024年6月5日周三 21:35写道：
 
> All the best
> 
> On Wed, Jun 5, 2024 at 5:56 AM xiangyu feng <
 xiangyu...@gmail.com>
 wrote:
> 
>> Congratulations, Rui!
>> 
>> Regards,
>> Xiangyu Feng
>> 
>> Feng Jin  于2024年6月5日周三 20:42写道：
>> 
>>> Congratulations, Rui!
>>> 
>>> 
>>> Best,
>>> Feng Jin
>>> 
>>> On Wed, Jun 5, 2024 at 8:23 PM Yanfei Lei <
 fredia...@gmail.com
>> 
 wrote:
>>> 
 Congratulations, Rui!
 
 Best,
 Yanfei
 
 Luke Chen  于2024年6月5日周三 20:08写道：
> 
> Congrats, Rui!
> 
> Luke
> 
> On Wed, Jun 5, 2024 at 8:02 PM Jiabao Sun <
>>> jiabao...@apache.org>
>>> wrote:
> 
>> Congrats, Rui. Well-deserved!
>> 
>> Best,
>> Jiabao
>> 
>> Zhanghao Chen 
>>> 于2024年6月5日周三
 19:29写道：
>> 
>>> Congrats, Rui!
>>> 
>>> Best,
>>> Zhanghao Chen
>>> 
>>> From: Piotr Nowojski 
>>> Sent: Wednesday, June 5, 2024 18:01
>>> To: dev ; rui fan <
 1996fan...@gmail.com>
>>> Subject: [ANNOUNCE] New Apache Flink PMC Member -
>>> Fan
> Rui
>>> 
>>> Hi everyone,
>>> 
>>> On behalf of the PMC, I'm very happy to announce
> another
>>> new
>> Apache
 Flink
>>> PMC Member - Fan Rui.
>>> 
>>> Rui has been active in the community since August
 2019.
 During
>> this
 time
>> he
>>> has contributed a lot of new features. Among
>>> others:
>>>  - Decoupling Autoscaler from Kubernetes
>> Operator,
 and
>> supporting
>>> Standalone Autoscaler
>>>  - Improvements to checkpointing, flamegraphs,
 restart
>> strategies,
>>> watermark alignment, network shuffles
>>>  - Optimizing the memory and CPU usage of large
> operators,
>> greatly
>>> reducing the risk and probability of TaskManager
>>> OOM
>>> 
>>> He reviewed a significant amount of PRs and has
>>> been
> active
> both
>> on
 the
>>> mailing lists and in Jira helping to both
>> maintain
 and
> grow
>> Apache
>> Flink's
>>> community. He is also our current Flink 1.20
>>> release
>>> manager.
>>> 
>>> In the last 12 months, Rui has been the most
>> active
 contributor
>> in
 the
>>> Flink Kubernetes Operator project, while being
>> the
 2nd
> most
>> active
 Flink
>>> contributor at the same time.
>>> 
>>> Please join me in welcoming and congratulating
>> Fan
 Rui!
>>> 
>>> Best,
>>> Piotrek (on behalf of the Flink PMC)
>>> 
>> 
 
>>> 
>> 
> 
 
>>> 
>> 
>> 
>> --
>> Best,
>> Hangxiang.
> 
 
>>> 
>> 

[jira] [Created] (FLINK-35535) Enable benchmark profiling in daily run

[Zakelly Lan (Jira)]

Zakelly Lan created FLINK-35535:
---

 Summary: Enable benchmark profiling in daily run
 Key: FLINK-35535
 URL: https://issues.apache.org/jira/browse/FLINK-35535
 Project: Flink
  Issue Type: Improvement
  Components: Benchmarks
Reporter: Zakelly Lan
Assignee: Zakelly Lan
 Fix For: 1.20.0


After FLINK-35534, the flink-benchmark supports profiling. We could consider 
enabling this in daily run.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35534) Support profiler for benchmarks

[Zakelly Lan (Jira)]

Zakelly Lan created FLINK-35534:
---

 Summary: Support profiler for benchmarks
 Key: FLINK-35534
 URL: https://issues.apache.org/jira/browse/FLINK-35534
 Project: Flink
  Issue Type: Improvement
  Components: Benchmarks
Reporter: Zakelly Lan
 Fix For: 1.20.0


As JMH support profiling during benchmark, the flink-benchmark could leverage 
this.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

[Junrui Lee]

Congratulations, Rui.

Best,
Junrui

Hang Ruan  于2024年6月6日周四 10:35写道：

> Congratulations, Rui!
>
> Best,
> Hang
>
> Samrat Deb  于2024年6月6日周四 10:28写道：
>
> > Congratulations Rui
> >
> > Bests,
> > Samrat
> >
> > On Thu, 6 Jun 2024 at 7:45 AM, Yuxin Tan  wrote:
> >
> > > Congratulations, Rui!
> > >
> > > Best,
> > > Yuxin
> > >
> > >
> > > Xuannan Su  于2024年6月6日周四 09:58写道：
> > >
> > > > Congratulations!
> > > >
> > > > Best regards,
> > > > Xuannan
> > > >
> > > > On Thu, Jun 6, 2024 at 9:53 AM Hangxiang Yu 
> > wrote:
> > > > >
> > > > > Congratulations, Rui !
> > > > >
> > > > > On Thu, Jun 6, 2024 at 9:18 AM Lincoln Lee  >
> > > > wrote:
> > > > >
> > > > > > Congratulations, Rui!
> > > > > >
> > > > > > Best,
> > > > > > Lincoln Lee
> > > > > >
> > > > > >
> > > > > > Lijie Wang  于2024年6月6日周四 09:11写道：
> > > > > >
> > > > > > > Congratulations, Rui!
> > > > > > >
> > > > > > > Best,
> > > > > > > Lijie
> > > > > > >
> > > > > > > Rodrigo Meneses  于2024年6月5日周三 21:35写道：
> > > > > > >
> > > > > > > > All the best
> > > > > > > >
> > > > > > > > On Wed, Jun 5, 2024 at 5:56 AM xiangyu feng <
> > > xiangyu...@gmail.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Congratulations, Rui!
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > Xiangyu Feng
> > > > > > > > >
> > > > > > > > > Feng Jin  于2024年6月5日周三 20:42写道：
> > > > > > > > >
> > > > > > > > > > Congratulations, Rui!
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Feng Jin
> > > > > > > > > >
> > > > > > > > > > On Wed, Jun 5, 2024 at 8:23 PM Yanfei Lei <
> > > fredia...@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Congratulations, Rui!
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Yanfei
> > > > > > > > > > >
> > > > > > > > > > > Luke Chen  于2024年6月5日周三 20:08写道：
> > > > > > > > > > > >
> > > > > > > > > > > > Congrats, Rui!
> > > > > > > > > > > >
> > > > > > > > > > > > Luke
> > > > > > > > > > > >
> > > > > > > > > > > > On Wed, Jun 5, 2024 at 8:02 PM Jiabao Sun <
> > > > > > jiabao...@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Congrats, Rui. Well-deserved!
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best,
> > > > > > > > > > > > > Jiabao
> > > > > > > > > > > > >
> > > > > > > > > > > > > Zhanghao Chen 
> > 于2024年6月5日周三
> > > > > > > 19:29写道：
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Congrats, Rui!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > Zhanghao Chen
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > From: Piotr Nowojski 
> > > > > > > > > > > > > > Sent: Wednesday, June 5, 2024 18:01
> > > > > > > > > > > > > > To: dev ; rui fan <
> > > > > > > 1996fan...@gmail.com>
> > > > > > > > > > > > > > Subject: [ANNOUNCE] New Apache Flink PMC Member -
> > Fan
> > > > Rui
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On behalf of the PMC, I'm very happy to announce
> > > > another
> > > > > > new
> > > > > > > > > Apache
> > > > > > > > > > > Flink
> > > > > > > > > > > > > > PMC Member - Fan Rui.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Rui has been active in the community since August
> > > 2019.
> > > > > > > During
> > > > > > > > > this
> > > > > > > > > > > time
> > > > > > > > > > > > > he
> > > > > > > > > > > > > > has contributed a lot of new features. Among
> > others:
> > > > > > > > > > > > > >   - Decoupling Autoscaler from Kubernetes
> Operator,
> > > and
> > > > > > > > > supporting
> > > > > > > > > > > > > > Standalone Autoscaler
> > > > > > > > > > > > > >   - Improvements to checkpointing, flamegraphs,
> > > restart
> > > > > > > > > strategies,
> > > > > > > > > > > > > > watermark alignment, network shuffles
> > > > > > > > > > > > > >   - Optimizing the memory and CPU usage of large
> > > > operators,
> > > > > > > > > greatly
> > > > > > > > > > > > > > reducing the risk and probability of TaskManager
> > OOM
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > He reviewed a significant amount of PRs and has
> > been
> > > > active
> > > > > > > > both
> > > > > > > > > on
> > > > > > > > > > > the
> > > > > > > > > > > > > > mailing lists and in Jira helping to both
> maintain
> > > and
> > > > grow
> > > > > > > > > Apache
> > > > > > > > > > > > > Flink's
> > > > > > > > > > > > > > community. He is also our current Flink 1.20
> > release
> > > > > > manager.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > In the last 12 months, Rui has been the most
> active
> > > > > > > contributor
> > > > > > > > > in
> > > > > > > > > > > the
> > > > > > > > > > > > > > Flink Kubernetes Operator project, while being
> the
> > > 2nd
> > > > most
> > > > > > > > > active
> > > > > > > > > > > Flink
> > > > > > > > > > > > > > contributor at the same time.
> > > > > > > 

Re: [DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

[Junrui Lee]

Thanks Yuxin for driving this proposal!

I have a question about the public interface compatibility in the context
of FLIP-459. As we've supported batch job recovery from jobMaster failures
in FLIP-383 which will be released in Flink 1.20. I wonder if the current
interface design support the future adaptation for batch job recovery?

Looking forward to your feedback.

Best,
Junrui.

weijie guo  于2024年6月5日周三 10:13写道：

> Thanks Yuxin for the proposal!
>
> When we first proposed Hybrid Shuffle, I wanted to support pluggable
> storage tier in the future. However, limited by the architecture of the
> legacy Hybrid Shuffle at that time, this idea has not been realized. The
> new architecture abstracts the tier nicely, and now it's time to introduce
> support for external storage.
>
> Big +1 for this one!
>
> Best regards,
>
> Weijie
>
>
> rexxiong  于2024年6月5日周三 00:08写道：
>
> > Thanks Yuxin for the proposal. +1,  as a member of the Apache Celeborn
> > community, I am very excited about the integration of Flink's Hybrid
> > Shuffle with Apache Celeborn. The whole design of CIP-6 looks good to
> me. I
> > am looking forward to this integration.
> >
> > Thanks,
> > Jiashu Xiong
> >
> > Ethan Feng  于2024年6月4日周二 16:47写道：
> >
> > > +1 for this proposal.
> > >
> > > After internally reviewing the prototype of CIP-6, this would improve
> > > performance and stability for Flink users using Celeborn.
> > >
> > > Expect to see this feature come out to the community.
> > >
> > > As I come from the Celeborn community, I hope more users can try to
> > > use Celeborn when there are Flink batch jobs.
> > >
> > > Thanks,
> > > Ethan Feng
> > >
> > > Yuxin Tan  于2024年6月4日周二 16:34写道：
> > > >
> > > > Hi, Venkatakrishnan,
> > > >
> > > > Thanks for joining the discussion. We appreciate your interest
> > > > in contributing to the work. Once the FLIP and CIP proposals
> > > > have been approved, we will create some JIRA tickets in Flink
> > > > and Celeborn projects. Please feel free to take a look at the
> > > > tickets and select any that resonate with your interests.
> > > >
> > > > Best,
> > > > Yuxin
> > > >
> > > >
> > > > Venkatakrishnan Sowrirajan  于2024年5月31日周五 23:11写道：
> > > >
> > > > > Thanks for this FLIP. We are also interested in
> learning/contributing
> > > to
> > > > > the hybrid shuffle integration with celeborn for batch executions.
> > > > >
> > > > > On Tue, May 28, 2024, 7:07 PM Yuxin Tan 
> > > wrote:
> > > > >
> > > > > > Hi, Xintong,
> > > > > >
> > > > > > >  I think we can also publish the prototype codes so the
> > > > > > community can better understand and help with it.
> > > > > >
> > > > > > Ok, I agree on the point. I will prepare and publish the code
> > > > > > recently.
> > > > > >
> > > > > > Rui,
> > > > > >
> > > > > > > Kindly reminder: the image of CIP-6[1] cannot be loaded.
> > > > > >
> > > > > > Thanks for the reminder. I've updated the images.
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Yuxin
> > > > > >
> > > > > >
> > > > > > Rui Fan <1996fan...@gmail.com> 于2024年5月29日周三 09:33写道：
> > > > > >
> > > > > > > Thanks Yuxin for driving this proposal!
> > > > > > >
> > > > > > > Kindly reminder: the image of CIP-6[1] cannot be loaded.
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/CELEBORN/CIP-6*Support*Flink*hybrid*shuffle*integration*with*Apache*Celeborn__;KysrKysrKys!!IKRxdwAv5BmarQ!ZRTc1aUSYMDBazuIwlet1Dzk2_DD9qKTgoDLH9jSwAVLgwplcuId_8JoXkH0i7AeWxKWXkL0sxM3AeW-H9OJ6v9uGw$
> > > > > > >
> > > > > > > Best,
> > > > > > > Rui
> > > > > > >
> > > > > > > On Wed, May 29, 2024 at 9:03 AM Xintong Song <
> > > tonysong...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > +1 for this proposal.
> > > > > > > >
> > > > > > > > We have been prototyping this feature internally at Alibaba
> > for a
> > > > > > couple
> > > > > > > of
> > > > > > > > months. Yuxin, I think we can also publish the prototype
> codes
> > > so the
> > > > > > > > community can better understand and help with it.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > >
> > > > > > > > Xintong
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, May 28, 2024 at 8:34 PM Yuxin Tan <
> > > tanyuxinw...@gmail.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > I would like to start a discussion on FLIP-459 Support
> Flink
> > > hybrid
> > > > > > > > shuffle
> > > > > > > > > integration with
> > > > > > > > > Apache Celeborn[1]. Flink hybrid shuffle supports
> transitions
> > > > > between
> > > > > > > > > memory, disk, and
> > > > > > > > > remote storage to improve performance and job stability.
> > > > > > Concurrently,
> > > > > > > > > Apache Celeborn
> > > > > > > > > provides a stable, performant, scalable remote shuffle
> > service.
> > > > > This
> > > > > > > > > integration proposal is to
> > > > > > > > > harness 

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

[Yuepeng Pan]

Congratulations !

Best,
Yuepeng Pan

On 2024/06/05 10:01:58 Piotr Nowojski wrote:
> Hi everyone,
> 
> On behalf of the PMC, I'm very happy to announce another new Apache Flink
> PMC Member - Fan Rui.
> 
> Rui has been active in the community since August 2019. During this time he
> has contributed a lot of new features. Among others:
>   - Decoupling Autoscaler from Kubernetes Operator, and supporting
> Standalone Autoscaler
>   - Improvements to checkpointing, flamegraphs, restart strategies,
> watermark alignment, network shuffles
>   - Optimizing the memory and CPU usage of large operators, greatly
> reducing the risk and probability of TaskManager OOM
> 
> He reviewed a significant amount of PRs and has been active both on the
> mailing lists and in Jira helping to both maintain and grow Apache Flink's
> community. He is also our current Flink 1.20 release manager.
> 
> In the last 12 months, Rui has been the most active contributor in the
> Flink Kubernetes Operator project, while being the 2nd most active Flink
> contributor at the same time.
> 
> Please join me in welcoming and congratulating Fan Rui!
> 
> Best,
> Piotrek (on behalf of the Flink PMC)
> 

Re: Poor Load Balancing across TaskManagers for Multiple Kafka Sources

[Zhanghao Chen]

Hi Kevin,

The problem here is about how to evenly distribute partitions from multiple 
Kafka topics to tasks, while FLIP-370 is only concerned about how to evenly 
distribute tasks to slots & taskmanagers, so FLIP-370 won't help here.

Best,
Zhanghao Chen

From: Kevin Lam 
Sent: Thursday, June 6, 2024 2:32
To: dev@flink.apache.org 
Subject: Poor Load Balancing across TaskManagers for Multiple Kafka Sources

Hey all,

I'm seeing an issue with poor load balancing across TaskManagers for Kafka
Sources using the Flink SQL API and wondering if FLIP-370 will help with
it, or if not, interested in any ideas the community has to mitigate the
issue.

The Kafka SplitEnumerator uses the following logic to assign split owners (code
pointer

):

```
  static int getSplitOwner(TopicPartition tp, int numReaders) {
int startIndex = ((tp.topic().hashCode() * 31) & 0x7FFF) %
numReaders;
return (startIndex + tp.partition()) % numReaders;
}
```

However this can result in imbalanced distribution of kafka partition
consumers across task managers.

To illustrate, I created a pipeline that consumes from 2 kafka topics, each
with 8 partitions, and just sinks them to a blackhole connector sink. For a
parallelism of 16 and 1 task slot per TaskManager, we'd ideally expect each
TaskManager to get its own kafka partition. ie. 16 partitions (8 partitions
from each topic) split evenly across TaskManagers. However, due the
algorithm I linked and how the startIndex is computed, I have observed a
bunch of TaskManagers with 2 partitions (one from each topic), and some
TaskManager completely idle.

I've also run an experiment with the same pipeline where I set parallelism
such that each task manager gets exactly 1 partition, and compared it
against when each task manager gets exactly 2 partitions (one from each
topic). I ensured this was the case by setting an appropriate parallelism,
and ran the jobs on an application cluster. Since the partitions are fixed,
the extra parallelism if any isn't used. The case where there is exactly 1
partition per TaskManager processes a fixed set of data 20% faster.

I was reading FLIP-370

and understand it will improve task scheduling in certain scenarios. Will
FLIP-370 help with this KafkaSource scenario? If not any ideas to improve
the subtask scheduling for KafkaSources? Ideally we don't need to carefully
consider the partition + resulting task distribution when selecting our
parallelism values.

Thanks for your help!

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

[Hang Ruan]

Congratulations, Rui!

Best,
Hang

Samrat Deb  于2024年6月6日周四 10:28写道：

> Congratulations Rui
>
> Bests,
> Samrat
>
> On Thu, 6 Jun 2024 at 7:45 AM, Yuxin Tan  wrote:
>
> > Congratulations, Rui!
> >
> > Best,
> > Yuxin
> >
> >
> > Xuannan Su  于2024年6月6日周四 09:58写道：
> >
> > > Congratulations!
> > >
> > > Best regards,
> > > Xuannan
> > >
> > > On Thu, Jun 6, 2024 at 9:53 AM Hangxiang Yu 
> wrote:
> > > >
> > > > Congratulations, Rui !
> > > >
> > > > On Thu, Jun 6, 2024 at 9:18 AM Lincoln Lee 
> > > wrote:
> > > >
> > > > > Congratulations, Rui!
> > > > >
> > > > > Best,
> > > > > Lincoln Lee
> > > > >
> > > > >
> > > > > Lijie Wang  于2024年6月6日周四 09:11写道：
> > > > >
> > > > > > Congratulations, Rui!
> > > > > >
> > > > > > Best,
> > > > > > Lijie
> > > > > >
> > > > > > Rodrigo Meneses  于2024年6月5日周三 21:35写道：
> > > > > >
> > > > > > > All the best
> > > > > > >
> > > > > > > On Wed, Jun 5, 2024 at 5:56 AM xiangyu feng <
> > xiangyu...@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Congratulations, Rui!
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Xiangyu Feng
> > > > > > > >
> > > > > > > > Feng Jin  于2024年6月5日周三 20:42写道：
> > > > > > > >
> > > > > > > > > Congratulations, Rui!
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Feng Jin
> > > > > > > > >
> > > > > > > > > On Wed, Jun 5, 2024 at 8:23 PM Yanfei Lei <
> > fredia...@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Congratulations, Rui!
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Yanfei
> > > > > > > > > >
> > > > > > > > > > Luke Chen  于2024年6月5日周三 20:08写道：
> > > > > > > > > > >
> > > > > > > > > > > Congrats, Rui!
> > > > > > > > > > >
> > > > > > > > > > > Luke
> > > > > > > > > > >
> > > > > > > > > > > On Wed, Jun 5, 2024 at 8:02 PM Jiabao Sun <
> > > > > jiabao...@apache.org>
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Congrats, Rui. Well-deserved!
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Jiabao
> > > > > > > > > > > >
> > > > > > > > > > > > Zhanghao Chen 
> 于2024年6月5日周三
> > > > > > 19:29写道：
> > > > > > > > > > > >
> > > > > > > > > > > > > Congrats, Rui!
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best,
> > > > > > > > > > > > > Zhanghao Chen
> > > > > > > > > > > > > 
> > > > > > > > > > > > > From: Piotr Nowojski 
> > > > > > > > > > > > > Sent: Wednesday, June 5, 2024 18:01
> > > > > > > > > > > > > To: dev ; rui fan <
> > > > > > 1996fan...@gmail.com>
> > > > > > > > > > > > > Subject: [ANNOUNCE] New Apache Flink PMC Member -
> Fan
> > > Rui
> > > > > > > > > > > > >
> > > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > > >
> > > > > > > > > > > > > On behalf of the PMC, I'm very happy to announce
> > > another
> > > > > new
> > > > > > > > Apache
> > > > > > > > > > Flink
> > > > > > > > > > > > > PMC Member - Fan Rui.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Rui has been active in the community since August
> > 2019.
> > > > > > During
> > > > > > > > this
> > > > > > > > > > time
> > > > > > > > > > > > he
> > > > > > > > > > > > > has contributed a lot of new features. Among
> others:
> > > > > > > > > > > > >   - Decoupling Autoscaler from Kubernetes Operator,
> > and
> > > > > > > > supporting
> > > > > > > > > > > > > Standalone Autoscaler
> > > > > > > > > > > > >   - Improvements to checkpointing, flamegraphs,
> > restart
> > > > > > > > strategies,
> > > > > > > > > > > > > watermark alignment, network shuffles
> > > > > > > > > > > > >   - Optimizing the memory and CPU usage of large
> > > operators,
> > > > > > > > greatly
> > > > > > > > > > > > > reducing the risk and probability of TaskManager
> OOM
> > > > > > > > > > > > >
> > > > > > > > > > > > > He reviewed a significant amount of PRs and has
> been
> > > active
> > > > > > > both
> > > > > > > > on
> > > > > > > > > > the
> > > > > > > > > > > > > mailing lists and in Jira helping to both maintain
> > and
> > > grow
> > > > > > > > Apache
> > > > > > > > > > > > Flink's
> > > > > > > > > > > > > community. He is also our current Flink 1.20
> release
> > > > > manager.
> > > > > > > > > > > > >
> > > > > > > > > > > > > In the last 12 months, Rui has been the most active
> > > > > > contributor
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > > Flink Kubernetes Operator project, while being the
> > 2nd
> > > most
> > > > > > > > active
> > > > > > > > > > Flink
> > > > > > > > > > > > > contributor at the same time.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Please join me in welcoming and congratulating Fan
> > Rui!
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best,
> > > > > > > > > > > > > Piotrek (on behalf of the Flink PMC)
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Best,
> > > > 

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

[Samrat Deb]

Congratulations Rui

Bests,
Samrat

On Thu, 6 Jun 2024 at 7:45 AM, Yuxin Tan  wrote:

> Congratulations, Rui!
>
> Best,
> Yuxin
>
>
> Xuannan Su  于2024年6月6日周四 09:58写道：
>
> > Congratulations!
> >
> > Best regards,
> > Xuannan
> >
> > On Thu, Jun 6, 2024 at 9:53 AM Hangxiang Yu  wrote:
> > >
> > > Congratulations, Rui !
> > >
> > > On Thu, Jun 6, 2024 at 9:18 AM Lincoln Lee 
> > wrote:
> > >
> > > > Congratulations, Rui!
> > > >
> > > > Best,
> > > > Lincoln Lee
> > > >
> > > >
> > > > Lijie Wang  于2024年6月6日周四 09:11写道：
> > > >
> > > > > Congratulations, Rui!
> > > > >
> > > > > Best,
> > > > > Lijie
> > > > >
> > > > > Rodrigo Meneses  于2024年6月5日周三 21:35写道：
> > > > >
> > > > > > All the best
> > > > > >
> > > > > > On Wed, Jun 5, 2024 at 5:56 AM xiangyu feng <
> xiangyu...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > Congratulations, Rui!
> > > > > > >
> > > > > > > Regards,
> > > > > > > Xiangyu Feng
> > > > > > >
> > > > > > > Feng Jin  于2024年6月5日周三 20:42写道：
> > > > > > >
> > > > > > > > Congratulations, Rui!
> > > > > > > >
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Feng Jin
> > > > > > > >
> > > > > > > > On Wed, Jun 5, 2024 at 8:23 PM Yanfei Lei <
> fredia...@gmail.com
> > >
> > > > > wrote:
> > > > > > > >
> > > > > > > > > Congratulations, Rui!
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Yanfei
> > > > > > > > >
> > > > > > > > > Luke Chen  于2024年6月5日周三 20:08写道：
> > > > > > > > > >
> > > > > > > > > > Congrats, Rui!
> > > > > > > > > >
> > > > > > > > > > Luke
> > > > > > > > > >
> > > > > > > > > > On Wed, Jun 5, 2024 at 8:02 PM Jiabao Sun <
> > > > jiabao...@apache.org>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Congrats, Rui. Well-deserved!
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Jiabao
> > > > > > > > > > >
> > > > > > > > > > > Zhanghao Chen  于2024年6月5日周三
> > > > > 19:29写道：
> > > > > > > > > > >
> > > > > > > > > > > > Congrats, Rui!
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Zhanghao Chen
> > > > > > > > > > > > 
> > > > > > > > > > > > From: Piotr Nowojski 
> > > > > > > > > > > > Sent: Wednesday, June 5, 2024 18:01
> > > > > > > > > > > > To: dev ; rui fan <
> > > > > 1996fan...@gmail.com>
> > > > > > > > > > > > Subject: [ANNOUNCE] New Apache Flink PMC Member - Fan
> > Rui
> > > > > > > > > > > >
> > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > >
> > > > > > > > > > > > On behalf of the PMC, I'm very happy to announce
> > another
> > > > new
> > > > > > > Apache
> > > > > > > > > Flink
> > > > > > > > > > > > PMC Member - Fan Rui.
> > > > > > > > > > > >
> > > > > > > > > > > > Rui has been active in the community since August
> 2019.
> > > > > During
> > > > > > > this
> > > > > > > > > time
> > > > > > > > > > > he
> > > > > > > > > > > > has contributed a lot of new features. Among others:
> > > > > > > > > > > >   - Decoupling Autoscaler from Kubernetes Operator,
> and
> > > > > > > supporting
> > > > > > > > > > > > Standalone Autoscaler
> > > > > > > > > > > >   - Improvements to checkpointing, flamegraphs,
> restart
> > > > > > > strategies,
> > > > > > > > > > > > watermark alignment, network shuffles
> > > > > > > > > > > >   - Optimizing the memory and CPU usage of large
> > operators,
> > > > > > > greatly
> > > > > > > > > > > > reducing the risk and probability of TaskManager OOM
> > > > > > > > > > > >
> > > > > > > > > > > > He reviewed a significant amount of PRs and has been
> > active
> > > > > > both
> > > > > > > on
> > > > > > > > > the
> > > > > > > > > > > > mailing lists and in Jira helping to both maintain
> and
> > grow
> > > > > > > Apache
> > > > > > > > > > > Flink's
> > > > > > > > > > > > community. He is also our current Flink 1.20 release
> > > > manager.
> > > > > > > > > > > >
> > > > > > > > > > > > In the last 12 months, Rui has been the most active
> > > > > contributor
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > > Flink Kubernetes Operator project, while being the
> 2nd
> > most
> > > > > > > active
> > > > > > > > > Flink
> > > > > > > > > > > > contributor at the same time.
> > > > > > > > > > > >
> > > > > > > > > > > > Please join me in welcoming and congratulating Fan
> Rui!
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Piotrek (on behalf of the Flink PMC)
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > Best,
> > > Hangxiang.
> >
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Weijie Guo

[Samrat Deb]

Congratulations

Bests,
Samrat

On Thu, 6 Jun 2024 at 7:28 AM, Xuannan Su  wrote:

> Congratulations!
>
> Best regards,
> Xuannan
>
> On Thu, Jun 6, 2024 at 9:20 AM Jiabao Sun  wrote:
> >
> > Congratulations, Weijie!
> >
> > Best,
> > Jiabao
> >
> > Lincoln Lee  于2024年6月6日周四 09:17写道：
> >
> > > Congratulations, Weijie!
> > >
> > > Best,
> > > Lincoln Lee
> > >
> > >
> > > ConradJam  于2024年6月5日周三 10:46写道：
> > >
> > > > Congratulations！
> > > >
> > > > Biao Geng  于2024年6月5日周三 10:44写道：
> > > >
> > > > > Congratulations, Weijie!
> > > > > Best,
> > > > > Biao Geng
> > > > >
> > > > >
> > > > > Yun Tang  于2024年6月5日周三 10:42写道：
> > > > >
> > > > > > Congratulations, Weijie!
> > > > > >
> > > > > > Best
> > > > > > Yun Tang
> > > > > > 
> > > > > > From: Hangxiang Yu 
> > > > > > Sent: Wednesday, June 5, 2024 10:00
> > > > > > To: dev@flink.apache.org 
> > > > > > Subject: Re: [ANNOUNCE] New Apache Flink PMC Member - Weijie Guo
> > > > > >
> > > > > > Congratulations, Weijie!
> > > > > >
> > > > > > On Tue, Jun 4, 2024 at 11:40 PM Zhanghao Chen <
> > > > zhanghao.c...@outlook.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Congrats, Weijie!
> > > > > > >
> > > > > > > Best,
> > > > > > > Zhanghao Chen
> > > > > > > 
> > > > > > > From: Hang Ruan 
> > > > > > > Sent: Tuesday, June 4, 2024 16:37
> > > > > > > To: dev@flink.apache.org 
> > > > > > > Subject: Re: [ANNOUNCE] New Apache Flink PMC Member - Weijie
> Guo
> > > > > > >
> > > > > > > Congratulations Weijie!
> > > > > > >
> > > > > > > Best,
> > > > > > > Hang
> > > > > > >
> > > > > > > Yanfei Lei  于2024年6月4日周二 16:24写道：
> > > > > > >
> > > > > > > > Congratulations!
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Yanfei
> > > > > > > >
> > > > > > > > Leonard Xu  于2024年6月4日周二 16:20写道：
> > > > > > > > >
> > > > > > > > > Congratulations！
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Leonard
> > > > > > > > >
> > > > > > > > > > 2024年6月4日 下午4:02，Yangze Guo  写道：
> > > > > > > > > >
> > > > > > > > > > Congratulations!
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Yangze Guo
> > > > > > > > > >
> > > > > > > > > > On Tue, Jun 4, 2024 at 4:00 PM Weihua Hu <
> > > > huweihua@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > >>
> > > > > > > > > >> Congratulations, Weijie!
> > > > > > > > > >>
> > > > > > > > > >> Best,
> > > > > > > > > >> Weihua
> > > > > > > > > >>
> > > > > > > > > >>
> > > > > > > > > >> On Tue, Jun 4, 2024 at 3:03 PM Yuxin Tan <
> > > > > tanyuxinw...@gmail.com>
> > > > > > > > wrote:
> > > > > > > > > >>
> > > > > > > > > >>> Congratulations, Weijie!
> > > > > > > > > >>>
> > > > > > > > > >>> Best,
> > > > > > > > > >>> Yuxin
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > > > > >>> Yuepeng Pan  于2024年6月4日周二
> 14:57写道：
> > > > > > > > > >>>
> > > > > > > > >  Congratulations !
> > > > > > > > > 
> > > > > > > > > 
> > > > > > > > >  Best,
> > > > > > > > >  Yuepeng Pan
> > > > > > > > > 
> > > > > > > > >  At 2024-06-04 14:45:45, "Xintong Song" <
> > > > tonysong...@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > > Hi everyone,
> > > > > > > > > >
> > > > > > > > > > On behalf of the PMC, I'm very happy to announce that
> > > > Weijie
> > > > > > Guo
> > > > > > > > has
> > > > > > > > >  joined
> > > > > > > > > > the Flink PMC!
> > > > > > > > > >
> > > > > > > > > > Weijie has been an active member of the Apache Flink
> > > > > community
> > > > > > > for
> > > > > > > > many
> > > > > > > > > > years. He has made significant contributions in many
> > > > > > components,
> > > > > > > > > >>> including
> > > > > > > > > > runtime, shuffle, sdk, connectors, etc. He has
> driven /
> > > > > > > > participated in
> > > > > > > > > > many FLIPs, authored and reviewed hundreds of PRs,
> been
> > > > > > > > consistently
> > > > > > > > >  active
> > > > > > > > > > on mailing lists, and also helped with release
> management
> > > > of
> > > > > > 1.20
> > > > > > > > and
> > > > > > > > > > several other bugfix releases.
> > > > > > > > > >
> > > > > > > > > > Congratulations and welcome Weijie!
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > >
> > > > > > > > > > Xintong (on behalf of the Flink PMC)
> > > > > > > > > 
> > > > > > > > > >>>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best,
> > > > > > Hangxiang.
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Best
> > > >
> > > > ConradJam
> > > >
> > >
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

[Yuxin Tan]

Congratulations, Rui!

Best,
Yuxin


Xuannan Su  于2024年6月6日周四 09:58写道：

> Congratulations!
>
> Best regards,
> Xuannan
>
> On Thu, Jun 6, 2024 at 9:53 AM Hangxiang Yu  wrote:
> >
> > Congratulations, Rui !
> >
> > On Thu, Jun 6, 2024 at 9:18 AM Lincoln Lee 
> wrote:
> >
> > > Congratulations, Rui!
> > >
> > > Best,
> > > Lincoln Lee
> > >
> > >
> > > Lijie Wang  于2024年6月6日周四 09:11写道：
> > >
> > > > Congratulations, Rui!
> > > >
> > > > Best,
> > > > Lijie
> > > >
> > > > Rodrigo Meneses  于2024年6月5日周三 21:35写道：
> > > >
> > > > > All the best
> > > > >
> > > > > On Wed, Jun 5, 2024 at 5:56 AM xiangyu feng 
> > > > wrote:
> > > > >
> > > > > > Congratulations, Rui!
> > > > > >
> > > > > > Regards,
> > > > > > Xiangyu Feng
> > > > > >
> > > > > > Feng Jin  于2024年6月5日周三 20:42写道：
> > > > > >
> > > > > > > Congratulations, Rui!
> > > > > > >
> > > > > > >
> > > > > > > Best,
> > > > > > > Feng Jin
> > > > > > >
> > > > > > > On Wed, Jun 5, 2024 at 8:23 PM Yanfei Lei  >
> > > > wrote:
> > > > > > >
> > > > > > > > Congratulations, Rui!
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Yanfei
> > > > > > > >
> > > > > > > > Luke Chen  于2024年6月5日周三 20:08写道：
> > > > > > > > >
> > > > > > > > > Congrats, Rui!
> > > > > > > > >
> > > > > > > > > Luke
> > > > > > > > >
> > > > > > > > > On Wed, Jun 5, 2024 at 8:02 PM Jiabao Sun <
> > > jiabao...@apache.org>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Congrats, Rui. Well-deserved!
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Jiabao
> > > > > > > > > >
> > > > > > > > > > Zhanghao Chen  于2024年6月5日周三
> > > > 19:29写道：
> > > > > > > > > >
> > > > > > > > > > > Congrats, Rui!
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Zhanghao Chen
> > > > > > > > > > > 
> > > > > > > > > > > From: Piotr Nowojski 
> > > > > > > > > > > Sent: Wednesday, June 5, 2024 18:01
> > > > > > > > > > > To: dev ; rui fan <
> > > > 1996fan...@gmail.com>
> > > > > > > > > > > Subject: [ANNOUNCE] New Apache Flink PMC Member - Fan
> Rui
> > > > > > > > > > >
> > > > > > > > > > > Hi everyone,
> > > > > > > > > > >
> > > > > > > > > > > On behalf of the PMC, I'm very happy to announce
> another
> > > new
> > > > > > Apache
> > > > > > > > Flink
> > > > > > > > > > > PMC Member - Fan Rui.
> > > > > > > > > > >
> > > > > > > > > > > Rui has been active in the community since August 2019.
> > > > During
> > > > > > this
> > > > > > > > time
> > > > > > > > > > he
> > > > > > > > > > > has contributed a lot of new features. Among others:
> > > > > > > > > > >   - Decoupling Autoscaler from Kubernetes Operator, and
> > > > > > supporting
> > > > > > > > > > > Standalone Autoscaler
> > > > > > > > > > >   - Improvements to checkpointing, flamegraphs, restart
> > > > > > strategies,
> > > > > > > > > > > watermark alignment, network shuffles
> > > > > > > > > > >   - Optimizing the memory and CPU usage of large
> operators,
> > > > > > greatly
> > > > > > > > > > > reducing the risk and probability of TaskManager OOM
> > > > > > > > > > >
> > > > > > > > > > > He reviewed a significant amount of PRs and has been
> active
> > > > > both
> > > > > > on
> > > > > > > > the
> > > > > > > > > > > mailing lists and in Jira helping to both maintain and
> grow
> > > > > > Apache
> > > > > > > > > > Flink's
> > > > > > > > > > > community. He is also our current Flink 1.20 release
> > > manager.
> > > > > > > > > > >
> > > > > > > > > > > In the last 12 months, Rui has been the most active
> > > > contributor
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > Flink Kubernetes Operator project, while being the 2nd
> most
> > > > > > active
> > > > > > > > Flink
> > > > > > > > > > > contributor at the same time.
> > > > > > > > > > >
> > > > > > > > > > > Please join me in welcoming and congratulating Fan Rui!
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Piotrek (on behalf of the Flink PMC)
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> > Best,
> > Hangxiang.
>

[jira] [Created] (FLINK-35533) FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

[Yuxin Tan (Jira)]

Yuxin Tan created FLINK-35533:
-

 Summary: FLIP-459: Support Flink hybrid shuffle integration with 
Apache Celeborn
 Key: FLINK-35533
 URL: https://issues.apache.org/jira/browse/FLINK-35533
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Network
Affects Versions: 1.20.0
Reporter: Yuxin Tan
Assignee: Yuxin Tan


This is the umbrella jira for 
[FLIP-459|https://cwiki.apache.org/confluence/display/FLINK/FLIP-459%3A+Support+Flink+hybrid+shuffle+integration+with+Apache+Celeborn].



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

[Xuannan Su]

Congratulations!

Best regards,
Xuannan

On Thu, Jun 6, 2024 at 9:53 AM Hangxiang Yu  wrote:
>
> Congratulations, Rui !
>
> On Thu, Jun 6, 2024 at 9:18 AM Lincoln Lee  wrote:
>
> > Congratulations, Rui!
> >
> > Best,
> > Lincoln Lee
> >
> >
> > Lijie Wang  于2024年6月6日周四 09:11写道：
> >
> > > Congratulations, Rui!
> > >
> > > Best,
> > > Lijie
> > >
> > > Rodrigo Meneses  于2024年6月5日周三 21:35写道：
> > >
> > > > All the best
> > > >
> > > > On Wed, Jun 5, 2024 at 5:56 AM xiangyu feng 
> > > wrote:
> > > >
> > > > > Congratulations, Rui!
> > > > >
> > > > > Regards,
> > > > > Xiangyu Feng
> > > > >
> > > > > Feng Jin  于2024年6月5日周三 20:42写道：
> > > > >
> > > > > > Congratulations, Rui!
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Feng Jin
> > > > > >
> > > > > > On Wed, Jun 5, 2024 at 8:23 PM Yanfei Lei 
> > > wrote:
> > > > > >
> > > > > > > Congratulations, Rui!
> > > > > > >
> > > > > > > Best,
> > > > > > > Yanfei
> > > > > > >
> > > > > > > Luke Chen  于2024年6月5日周三 20:08写道：
> > > > > > > >
> > > > > > > > Congrats, Rui!
> > > > > > > >
> > > > > > > > Luke
> > > > > > > >
> > > > > > > > On Wed, Jun 5, 2024 at 8:02 PM Jiabao Sun <
> > jiabao...@apache.org>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Congrats, Rui. Well-deserved!
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Jiabao
> > > > > > > > >
> > > > > > > > > Zhanghao Chen  于2024年6月5日周三
> > > 19:29写道：
> > > > > > > > >
> > > > > > > > > > Congrats, Rui!
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Zhanghao Chen
> > > > > > > > > > 
> > > > > > > > > > From: Piotr Nowojski 
> > > > > > > > > > Sent: Wednesday, June 5, 2024 18:01
> > > > > > > > > > To: dev ; rui fan <
> > > 1996fan...@gmail.com>
> > > > > > > > > > Subject: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui
> > > > > > > > > >
> > > > > > > > > > Hi everyone,
> > > > > > > > > >
> > > > > > > > > > On behalf of the PMC, I'm very happy to announce another
> > new
> > > > > Apache
> > > > > > > Flink
> > > > > > > > > > PMC Member - Fan Rui.
> > > > > > > > > >
> > > > > > > > > > Rui has been active in the community since August 2019.
> > > During
> > > > > this
> > > > > > > time
> > > > > > > > > he
> > > > > > > > > > has contributed a lot of new features. Among others:
> > > > > > > > > >   - Decoupling Autoscaler from Kubernetes Operator, and
> > > > > supporting
> > > > > > > > > > Standalone Autoscaler
> > > > > > > > > >   - Improvements to checkpointing, flamegraphs, restart
> > > > > strategies,
> > > > > > > > > > watermark alignment, network shuffles
> > > > > > > > > >   - Optimizing the memory and CPU usage of large operators,
> > > > > greatly
> > > > > > > > > > reducing the risk and probability of TaskManager OOM
> > > > > > > > > >
> > > > > > > > > > He reviewed a significant amount of PRs and has been active
> > > > both
> > > > > on
> > > > > > > the
> > > > > > > > > > mailing lists and in Jira helping to both maintain and grow
> > > > > Apache
> > > > > > > > > Flink's
> > > > > > > > > > community. He is also our current Flink 1.20 release
> > manager.
> > > > > > > > > >
> > > > > > > > > > In the last 12 months, Rui has been the most active
> > > contributor
> > > > > in
> > > > > > > the
> > > > > > > > > > Flink Kubernetes Operator project, while being the 2nd most
> > > > > active
> > > > > > > Flink
> > > > > > > > > > contributor at the same time.
> > > > > > > > > >
> > > > > > > > > > Please join me in welcoming and congratulating Fan Rui!
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Piotrek (on behalf of the Flink PMC)
> > > > > > > > > >
> > > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
> --
> Best,
> Hangxiang.

Re: [ANNOUNCE] New Apache Flink PMC Member - Weijie Guo

[Xuannan Su]

Congratulations!

Best regards,
Xuannan

On Thu, Jun 6, 2024 at 9:20 AM Jiabao Sun  wrote:
>
> Congratulations, Weijie!
>
> Best,
> Jiabao
>
> Lincoln Lee  于2024年6月6日周四 09:17写道：
>
> > Congratulations, Weijie!
> >
> > Best,
> > Lincoln Lee
> >
> >
> > ConradJam  于2024年6月5日周三 10:46写道：
> >
> > > Congratulations！
> > >
> > > Biao Geng  于2024年6月5日周三 10:44写道：
> > >
> > > > Congratulations, Weijie!
> > > > Best,
> > > > Biao Geng
> > > >
> > > >
> > > > Yun Tang  于2024年6月5日周三 10:42写道：
> > > >
> > > > > Congratulations, Weijie!
> > > > >
> > > > > Best
> > > > > Yun Tang
> > > > > 
> > > > > From: Hangxiang Yu 
> > > > > Sent: Wednesday, June 5, 2024 10:00
> > > > > To: dev@flink.apache.org 
> > > > > Subject: Re: [ANNOUNCE] New Apache Flink PMC Member - Weijie Guo
> > > > >
> > > > > Congratulations, Weijie!
> > > > >
> > > > > On Tue, Jun 4, 2024 at 11:40 PM Zhanghao Chen <
> > > zhanghao.c...@outlook.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Congrats, Weijie!
> > > > > >
> > > > > > Best,
> > > > > > Zhanghao Chen
> > > > > > 
> > > > > > From: Hang Ruan 
> > > > > > Sent: Tuesday, June 4, 2024 16:37
> > > > > > To: dev@flink.apache.org 
> > > > > > Subject: Re: [ANNOUNCE] New Apache Flink PMC Member - Weijie Guo
> > > > > >
> > > > > > Congratulations Weijie!
> > > > > >
> > > > > > Best,
> > > > > > Hang
> > > > > >
> > > > > > Yanfei Lei  于2024年6月4日周二 16:24写道：
> > > > > >
> > > > > > > Congratulations!
> > > > > > >
> > > > > > > Best,
> > > > > > > Yanfei
> > > > > > >
> > > > > > > Leonard Xu  于2024年6月4日周二 16:20写道：
> > > > > > > >
> > > > > > > > Congratulations！
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Leonard
> > > > > > > >
> > > > > > > > > 2024年6月4日 下午4:02，Yangze Guo  写道：
> > > > > > > > >
> > > > > > > > > Congratulations!
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Yangze Guo
> > > > > > > > >
> > > > > > > > > On Tue, Jun 4, 2024 at 4:00 PM Weihua Hu <
> > > huweihua@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > >>
> > > > > > > > >> Congratulations, Weijie!
> > > > > > > > >>
> > > > > > > > >> Best,
> > > > > > > > >> Weihua
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> On Tue, Jun 4, 2024 at 3:03 PM Yuxin Tan <
> > > > tanyuxinw...@gmail.com>
> > > > > > > wrote:
> > > > > > > > >>
> > > > > > > > >>> Congratulations, Weijie!
> > > > > > > > >>>
> > > > > > > > >>> Best,
> > > > > > > > >>> Yuxin
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>> Yuepeng Pan  于2024年6月4日周二 14:57写道：
> > > > > > > > >>>
> > > > > > > >  Congratulations !
> > > > > > > > 
> > > > > > > > 
> > > > > > > >  Best,
> > > > > > > >  Yuepeng Pan
> > > > > > > > 
> > > > > > > >  At 2024-06-04 14:45:45, "Xintong Song" <
> > > tonysong...@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > > Hi everyone,
> > > > > > > > >
> > > > > > > > > On behalf of the PMC, I'm very happy to announce that
> > > Weijie
> > > > > Guo
> > > > > > > has
> > > > > > > >  joined
> > > > > > > > > the Flink PMC!
> > > > > > > > >
> > > > > > > > > Weijie has been an active member of the Apache Flink
> > > > community
> > > > > > for
> > > > > > > many
> > > > > > > > > years. He has made significant contributions in many
> > > > > components,
> > > > > > > > >>> including
> > > > > > > > > runtime, shuffle, sdk, connectors, etc. He has driven /
> > > > > > > participated in
> > > > > > > > > many FLIPs, authored and reviewed hundreds of PRs, been
> > > > > > > consistently
> > > > > > > >  active
> > > > > > > > > on mailing lists, and also helped with release management
> > > of
> > > > > 1.20
> > > > > > > and
> > > > > > > > > several other bugfix releases.
> > > > > > > > >
> > > > > > > > > Congratulations and welcome Weijie!
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > >
> > > > > > > > > Xintong (on behalf of the Flink PMC)
> > > > > > > > 
> > > > > > > > >>>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best,
> > > > > Hangxiang.
> > > > >
> > > >
> > >
> > >
> > > --
> > > Best
> > >
> > > ConradJam
> > >
> >

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

[Hangxiang Yu]

Congratulations, Rui !

On Thu, Jun 6, 2024 at 9:18 AM Lincoln Lee  wrote:

> Congratulations, Rui!
>
> Best,
> Lincoln Lee
>
>
> Lijie Wang  于2024年6月6日周四 09:11写道：
>
> > Congratulations, Rui!
> >
> > Best,
> > Lijie
> >
> > Rodrigo Meneses  于2024年6月5日周三 21:35写道：
> >
> > > All the best
> > >
> > > On Wed, Jun 5, 2024 at 5:56 AM xiangyu feng 
> > wrote:
> > >
> > > > Congratulations, Rui!
> > > >
> > > > Regards,
> > > > Xiangyu Feng
> > > >
> > > > Feng Jin  于2024年6月5日周三 20:42写道：
> > > >
> > > > > Congratulations, Rui!
> > > > >
> > > > >
> > > > > Best,
> > > > > Feng Jin
> > > > >
> > > > > On Wed, Jun 5, 2024 at 8:23 PM Yanfei Lei 
> > wrote:
> > > > >
> > > > > > Congratulations, Rui!
> > > > > >
> > > > > > Best,
> > > > > > Yanfei
> > > > > >
> > > > > > Luke Chen  于2024年6月5日周三 20:08写道：
> > > > > > >
> > > > > > > Congrats, Rui!
> > > > > > >
> > > > > > > Luke
> > > > > > >
> > > > > > > On Wed, Jun 5, 2024 at 8:02 PM Jiabao Sun <
> jiabao...@apache.org>
> > > > > wrote:
> > > > > > >
> > > > > > > > Congrats, Rui. Well-deserved!
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Jiabao
> > > > > > > >
> > > > > > > > Zhanghao Chen  于2024年6月5日周三
> > 19:29写道：
> > > > > > > >
> > > > > > > > > Congrats, Rui!
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Zhanghao Chen
> > > > > > > > > 
> > > > > > > > > From: Piotr Nowojski 
> > > > > > > > > Sent: Wednesday, June 5, 2024 18:01
> > > > > > > > > To: dev ; rui fan <
> > 1996fan...@gmail.com>
> > > > > > > > > Subject: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui
> > > > > > > > >
> > > > > > > > > Hi everyone,
> > > > > > > > >
> > > > > > > > > On behalf of the PMC, I'm very happy to announce another
> new
> > > > Apache
> > > > > > Flink
> > > > > > > > > PMC Member - Fan Rui.
> > > > > > > > >
> > > > > > > > > Rui has been active in the community since August 2019.
> > During
> > > > this
> > > > > > time
> > > > > > > > he
> > > > > > > > > has contributed a lot of new features. Among others:
> > > > > > > > >   - Decoupling Autoscaler from Kubernetes Operator, and
> > > > supporting
> > > > > > > > > Standalone Autoscaler
> > > > > > > > >   - Improvements to checkpointing, flamegraphs, restart
> > > > strategies,
> > > > > > > > > watermark alignment, network shuffles
> > > > > > > > >   - Optimizing the memory and CPU usage of large operators,
> > > > greatly
> > > > > > > > > reducing the risk and probability of TaskManager OOM
> > > > > > > > >
> > > > > > > > > He reviewed a significant amount of PRs and has been active
> > > both
> > > > on
> > > > > > the
> > > > > > > > > mailing lists and in Jira helping to both maintain and grow
> > > > Apache
> > > > > > > > Flink's
> > > > > > > > > community. He is also our current Flink 1.20 release
> manager.
> > > > > > > > >
> > > > > > > > > In the last 12 months, Rui has been the most active
> > contributor
> > > > in
> > > > > > the
> > > > > > > > > Flink Kubernetes Operator project, while being the 2nd most
> > > > active
> > > > > > Flink
> > > > > > > > > contributor at the same time.
> > > > > > > > >
> > > > > > > > > Please join me in welcoming and congratulating Fan Rui!
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Piotrek (on behalf of the Flink PMC)
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>


-- 
Best,
Hangxiang.

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

[Wei Zhong]

Congratulations, Rui!

Best,
Wei

> 2024年6月6日 09:16，Lincoln Lee  写道：
> 
> Congratulations, Rui!
> 
> Best,
> Lincoln Lee
> 
> 
> Lijie Wang  于2024年6月6日周四 09:11写道：
> 
>> Congratulations, Rui!
>> 
>> Best,
>> Lijie
>> 
>> Rodrigo Meneses  于2024年6月5日周三 21:35写道：
>> 
>>> All the best
>>> 
>>> On Wed, Jun 5, 2024 at 5:56 AM xiangyu feng 
>> wrote:
>>> 
 Congratulations, Rui!
 
 Regards,
 Xiangyu Feng
 
 Feng Jin  于2024年6月5日周三 20:42写道：
 
> Congratulations, Rui!
> 
> 
> Best,
> Feng Jin
> 
> On Wed, Jun 5, 2024 at 8:23 PM Yanfei Lei 
>> wrote:
> 
>> Congratulations, Rui!
>> 
>> Best,
>> Yanfei
>> 
>> Luke Chen  于2024年6月5日周三 20:08写道：
>>> 
>>> Congrats, Rui!
>>> 
>>> Luke
>>> 
>>> On Wed, Jun 5, 2024 at 8:02 PM Jiabao Sun 
> wrote:
>>> 
 Congrats, Rui. Well-deserved!
 
 Best,
 Jiabao
 
 Zhanghao Chen  于2024年6月5日周三
>> 19:29写道：
 
> Congrats, Rui!
> 
> Best,
> Zhanghao Chen
> 
> From: Piotr Nowojski 
> Sent: Wednesday, June 5, 2024 18:01
> To: dev ; rui fan <
>> 1996fan...@gmail.com>
> Subject: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui
> 
> Hi everyone,
> 
> On behalf of the PMC, I'm very happy to announce another new
 Apache
>> Flink
> PMC Member - Fan Rui.
> 
> Rui has been active in the community since August 2019.
>> During
 this
>> time
 he
> has contributed a lot of new features. Among others:
>  - Decoupling Autoscaler from Kubernetes Operator, and
 supporting
> Standalone Autoscaler
>  - Improvements to checkpointing, flamegraphs, restart
 strategies,
> watermark alignment, network shuffles
>  - Optimizing the memory and CPU usage of large operators,
 greatly
> reducing the risk and probability of TaskManager OOM
> 
> He reviewed a significant amount of PRs and has been active
>>> both
 on
>> the
> mailing lists and in Jira helping to both maintain and grow
 Apache
 Flink's
> community. He is also our current Flink 1.20 release manager.
> 
> In the last 12 months, Rui has been the most active
>> contributor
 in
>> the
> Flink Kubernetes Operator project, while being the 2nd most
 active
>> Flink
> contributor at the same time.
> 
> Please join me in welcoming and congratulating Fan Rui!
> 
> Best,
> Piotrek (on behalf of the Flink PMC)
> 
 
>> 
> 
 
>>> 
>> 

Re: [ANNOUNCE] New Apache Flink PMC Member - Weijie Guo

[Jiabao Sun]

Congratulations, Weijie!

Best,
Jiabao

Lincoln Lee  于2024年6月6日周四 09:17写道：

> Congratulations, Weijie!
>
> Best,
> Lincoln Lee
>
>
> ConradJam  于2024年6月5日周三 10:46写道：
>
> > Congratulations！
> >
> > Biao Geng  于2024年6月5日周三 10:44写道：
> >
> > > Congratulations, Weijie!
> > > Best,
> > > Biao Geng
> > >
> > >
> > > Yun Tang  于2024年6月5日周三 10:42写道：
> > >
> > > > Congratulations, Weijie!
> > > >
> > > > Best
> > > > Yun Tang
> > > > 
> > > > From: Hangxiang Yu 
> > > > Sent: Wednesday, June 5, 2024 10:00
> > > > To: dev@flink.apache.org 
> > > > Subject: Re: [ANNOUNCE] New Apache Flink PMC Member - Weijie Guo
> > > >
> > > > Congratulations, Weijie!
> > > >
> > > > On Tue, Jun 4, 2024 at 11:40 PM Zhanghao Chen <
> > zhanghao.c...@outlook.com
> > > >
> > > > wrote:
> > > >
> > > > > Congrats, Weijie!
> > > > >
> > > > > Best,
> > > > > Zhanghao Chen
> > > > > 
> > > > > From: Hang Ruan 
> > > > > Sent: Tuesday, June 4, 2024 16:37
> > > > > To: dev@flink.apache.org 
> > > > > Subject: Re: [ANNOUNCE] New Apache Flink PMC Member - Weijie Guo
> > > > >
> > > > > Congratulations Weijie!
> > > > >
> > > > > Best,
> > > > > Hang
> > > > >
> > > > > Yanfei Lei  于2024年6月4日周二 16:24写道：
> > > > >
> > > > > > Congratulations!
> > > > > >
> > > > > > Best,
> > > > > > Yanfei
> > > > > >
> > > > > > Leonard Xu  于2024年6月4日周二 16:20写道：
> > > > > > >
> > > > > > > Congratulations！
> > > > > > >
> > > > > > > Best,
> > > > > > > Leonard
> > > > > > >
> > > > > > > > 2024年6月4日 下午4:02，Yangze Guo  写道：
> > > > > > > >
> > > > > > > > Congratulations!
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Yangze Guo
> > > > > > > >
> > > > > > > > On Tue, Jun 4, 2024 at 4:00 PM Weihua Hu <
> > huweihua@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > >>
> > > > > > > >> Congratulations, Weijie!
> > > > > > > >>
> > > > > > > >> Best,
> > > > > > > >> Weihua
> > > > > > > >>
> > > > > > > >>
> > > > > > > >> On Tue, Jun 4, 2024 at 3:03 PM Yuxin Tan <
> > > tanyuxinw...@gmail.com>
> > > > > > wrote:
> > > > > > > >>
> > > > > > > >>> Congratulations, Weijie!
> > > > > > > >>>
> > > > > > > >>> Best,
> > > > > > > >>> Yuxin
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>> Yuepeng Pan  于2024年6月4日周二 14:57写道：
> > > > > > > >>>
> > > > > > >  Congratulations !
> > > > > > > 
> > > > > > > 
> > > > > > >  Best,
> > > > > > >  Yuepeng Pan
> > > > > > > 
> > > > > > >  At 2024-06-04 14:45:45, "Xintong Song" <
> > tonysong...@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > > Hi everyone,
> > > > > > > >
> > > > > > > > On behalf of the PMC, I'm very happy to announce that
> > Weijie
> > > > Guo
> > > > > > has
> > > > > > >  joined
> > > > > > > > the Flink PMC!
> > > > > > > >
> > > > > > > > Weijie has been an active member of the Apache Flink
> > > community
> > > > > for
> > > > > > many
> > > > > > > > years. He has made significant contributions in many
> > > > components,
> > > > > > > >>> including
> > > > > > > > runtime, shuffle, sdk, connectors, etc. He has driven /
> > > > > > participated in
> > > > > > > > many FLIPs, authored and reviewed hundreds of PRs, been
> > > > > > consistently
> > > > > > >  active
> > > > > > > > on mailing lists, and also helped with release management
> > of
> > > > 1.20
> > > > > > and
> > > > > > > > several other bugfix releases.
> > > > > > > >
> > > > > > > > Congratulations and welcome Weijie!
> > > > > > > >
> > > > > > > > Best,
> > > > > > > >
> > > > > > > > Xintong (on behalf of the Flink PMC)
> > > > > > > 
> > > > > > > >>>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Best,
> > > > Hangxiang.
> > > >
> > >
> >
> >
> > --
> > Best
> >
> > ConradJam
> >
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

[Lincoln Lee]

Congratulations, Rui!

Best,
Lincoln Lee


Lijie Wang  于2024年6月6日周四 09:11写道：

> Congratulations, Rui!
>
> Best,
> Lijie
>
> Rodrigo Meneses  于2024年6月5日周三 21:35写道：
>
> > All the best
> >
> > On Wed, Jun 5, 2024 at 5:56 AM xiangyu feng 
> wrote:
> >
> > > Congratulations, Rui!
> > >
> > > Regards,
> > > Xiangyu Feng
> > >
> > > Feng Jin  于2024年6月5日周三 20:42写道：
> > >
> > > > Congratulations, Rui!
> > > >
> > > >
> > > > Best,
> > > > Feng Jin
> > > >
> > > > On Wed, Jun 5, 2024 at 8:23 PM Yanfei Lei 
> wrote:
> > > >
> > > > > Congratulations, Rui!
> > > > >
> > > > > Best,
> > > > > Yanfei
> > > > >
> > > > > Luke Chen  于2024年6月5日周三 20:08写道：
> > > > > >
> > > > > > Congrats, Rui!
> > > > > >
> > > > > > Luke
> > > > > >
> > > > > > On Wed, Jun 5, 2024 at 8:02 PM Jiabao Sun 
> > > > wrote:
> > > > > >
> > > > > > > Congrats, Rui. Well-deserved!
> > > > > > >
> > > > > > > Best,
> > > > > > > Jiabao
> > > > > > >
> > > > > > > Zhanghao Chen  于2024年6月5日周三
> 19:29写道：
> > > > > > >
> > > > > > > > Congrats, Rui!
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Zhanghao Chen
> > > > > > > > 
> > > > > > > > From: Piotr Nowojski 
> > > > > > > > Sent: Wednesday, June 5, 2024 18:01
> > > > > > > > To: dev ; rui fan <
> 1996fan...@gmail.com>
> > > > > > > > Subject: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui
> > > > > > > >
> > > > > > > > Hi everyone,
> > > > > > > >
> > > > > > > > On behalf of the PMC, I'm very happy to announce another new
> > > Apache
> > > > > Flink
> > > > > > > > PMC Member - Fan Rui.
> > > > > > > >
> > > > > > > > Rui has been active in the community since August 2019.
> During
> > > this
> > > > > time
> > > > > > > he
> > > > > > > > has contributed a lot of new features. Among others:
> > > > > > > >   - Decoupling Autoscaler from Kubernetes Operator, and
> > > supporting
> > > > > > > > Standalone Autoscaler
> > > > > > > >   - Improvements to checkpointing, flamegraphs, restart
> > > strategies,
> > > > > > > > watermark alignment, network shuffles
> > > > > > > >   - Optimizing the memory and CPU usage of large operators,
> > > greatly
> > > > > > > > reducing the risk and probability of TaskManager OOM
> > > > > > > >
> > > > > > > > He reviewed a significant amount of PRs and has been active
> > both
> > > on
> > > > > the
> > > > > > > > mailing lists and in Jira helping to both maintain and grow
> > > Apache
> > > > > > > Flink's
> > > > > > > > community. He is also our current Flink 1.20 release manager.
> > > > > > > >
> > > > > > > > In the last 12 months, Rui has been the most active
> contributor
> > > in
> > > > > the
> > > > > > > > Flink Kubernetes Operator project, while being the 2nd most
> > > active
> > > > > Flink
> > > > > > > > contributor at the same time.
> > > > > > > >
> > > > > > > > Please join me in welcoming and congratulating Fan Rui!
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Piotrek (on behalf of the Flink PMC)
> > > > > > > >
> > > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Weijie Guo

[Lincoln Lee]

Congratulations, Weijie!

Best,
Lincoln Lee


ConradJam  于2024年6月5日周三 10:46写道：

> Congratulations！
>
> Biao Geng  于2024年6月5日周三 10:44写道：
>
> > Congratulations, Weijie!
> > Best,
> > Biao Geng
> >
> >
> > Yun Tang  于2024年6月5日周三 10:42写道：
> >
> > > Congratulations, Weijie!
> > >
> > > Best
> > > Yun Tang
> > > 
> > > From: Hangxiang Yu 
> > > Sent: Wednesday, June 5, 2024 10:00
> > > To: dev@flink.apache.org 
> > > Subject: Re: [ANNOUNCE] New Apache Flink PMC Member - Weijie Guo
> > >
> > > Congratulations, Weijie!
> > >
> > > On Tue, Jun 4, 2024 at 11:40 PM Zhanghao Chen <
> zhanghao.c...@outlook.com
> > >
> > > wrote:
> > >
> > > > Congrats, Weijie!
> > > >
> > > > Best,
> > > > Zhanghao Chen
> > > > 
> > > > From: Hang Ruan 
> > > > Sent: Tuesday, June 4, 2024 16:37
> > > > To: dev@flink.apache.org 
> > > > Subject: Re: [ANNOUNCE] New Apache Flink PMC Member - Weijie Guo
> > > >
> > > > Congratulations Weijie!
> > > >
> > > > Best,
> > > > Hang
> > > >
> > > > Yanfei Lei  于2024年6月4日周二 16:24写道：
> > > >
> > > > > Congratulations!
> > > > >
> > > > > Best,
> > > > > Yanfei
> > > > >
> > > > > Leonard Xu  于2024年6月4日周二 16:20写道：
> > > > > >
> > > > > > Congratulations！
> > > > > >
> > > > > > Best,
> > > > > > Leonard
> > > > > >
> > > > > > > 2024年6月4日 下午4:02，Yangze Guo  写道：
> > > > > > >
> > > > > > > Congratulations!
> > > > > > >
> > > > > > > Best,
> > > > > > > Yangze Guo
> > > > > > >
> > > > > > > On Tue, Jun 4, 2024 at 4:00 PM Weihua Hu <
> huweihua@gmail.com
> > >
> > > > > wrote:
> > > > > > >>
> > > > > > >> Congratulations, Weijie!
> > > > > > >>
> > > > > > >> Best,
> > > > > > >> Weihua
> > > > > > >>
> > > > > > >>
> > > > > > >> On Tue, Jun 4, 2024 at 3:03 PM Yuxin Tan <
> > tanyuxinw...@gmail.com>
> > > > > wrote:
> > > > > > >>
> > > > > > >>> Congratulations, Weijie!
> > > > > > >>>
> > > > > > >>> Best,
> > > > > > >>> Yuxin
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> Yuepeng Pan  于2024年6月4日周二 14:57写道：
> > > > > > >>>
> > > > > >  Congratulations !
> > > > > > 
> > > > > > 
> > > > > >  Best,
> > > > > >  Yuepeng Pan
> > > > > > 
> > > > > >  At 2024-06-04 14:45:45, "Xintong Song" <
> tonysong...@gmail.com
> > >
> > > > > wrote:
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > On behalf of the PMC, I'm very happy to announce that
> Weijie
> > > Guo
> > > > > has
> > > > > >  joined
> > > > > > > the Flink PMC!
> > > > > > >
> > > > > > > Weijie has been an active member of the Apache Flink
> > community
> > > > for
> > > > > many
> > > > > > > years. He has made significant contributions in many
> > > components,
> > > > > > >>> including
> > > > > > > runtime, shuffle, sdk, connectors, etc. He has driven /
> > > > > participated in
> > > > > > > many FLIPs, authored and reviewed hundreds of PRs, been
> > > > > consistently
> > > > > >  active
> > > > > > > on mailing lists, and also helped with release management
> of
> > > 1.20
> > > > > and
> > > > > > > several other bugfix releases.
> > > > > > >
> > > > > > > Congratulations and welcome Weijie!
> > > > > > >
> > > > > > > Best,
> > > > > > >
> > > > > > > Xintong (on behalf of the Flink PMC)
> > > > > > 
> > > > > > >>>
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > Best,
> > > Hangxiang.
> > >
> >
>
>
> --
> Best
>
> ConradJam
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

[Lijie Wang]

Congratulations, Rui!

Best,
Lijie

Rodrigo Meneses  于2024年6月5日周三 21:35写道：

> All the best
>
> On Wed, Jun 5, 2024 at 5:56 AM xiangyu feng  wrote:
>
> > Congratulations, Rui!
> >
> > Regards,
> > Xiangyu Feng
> >
> > Feng Jin  于2024年6月5日周三 20:42写道：
> >
> > > Congratulations, Rui!
> > >
> > >
> > > Best,
> > > Feng Jin
> > >
> > > On Wed, Jun 5, 2024 at 8:23 PM Yanfei Lei  wrote:
> > >
> > > > Congratulations, Rui!
> > > >
> > > > Best,
> > > > Yanfei
> > > >
> > > > Luke Chen  于2024年6月5日周三 20:08写道：
> > > > >
> > > > > Congrats, Rui!
> > > > >
> > > > > Luke
> > > > >
> > > > > On Wed, Jun 5, 2024 at 8:02 PM Jiabao Sun 
> > > wrote:
> > > > >
> > > > > > Congrats, Rui. Well-deserved!
> > > > > >
> > > > > > Best,
> > > > > > Jiabao
> > > > > >
> > > > > > Zhanghao Chen  于2024年6月5日周三 19:29写道：
> > > > > >
> > > > > > > Congrats, Rui!
> > > > > > >
> > > > > > > Best,
> > > > > > > Zhanghao Chen
> > > > > > > 
> > > > > > > From: Piotr Nowojski 
> > > > > > > Sent: Wednesday, June 5, 2024 18:01
> > > > > > > To: dev ; rui fan <1996fan...@gmail.com>
> > > > > > > Subject: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui
> > > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > On behalf of the PMC, I'm very happy to announce another new
> > Apache
> > > > Flink
> > > > > > > PMC Member - Fan Rui.
> > > > > > >
> > > > > > > Rui has been active in the community since August 2019. During
> > this
> > > > time
> > > > > > he
> > > > > > > has contributed a lot of new features. Among others:
> > > > > > >   - Decoupling Autoscaler from Kubernetes Operator, and
> > supporting
> > > > > > > Standalone Autoscaler
> > > > > > >   - Improvements to checkpointing, flamegraphs, restart
> > strategies,
> > > > > > > watermark alignment, network shuffles
> > > > > > >   - Optimizing the memory and CPU usage of large operators,
> > greatly
> > > > > > > reducing the risk and probability of TaskManager OOM
> > > > > > >
> > > > > > > He reviewed a significant amount of PRs and has been active
> both
> > on
> > > > the
> > > > > > > mailing lists and in Jira helping to both maintain and grow
> > Apache
> > > > > > Flink's
> > > > > > > community. He is also our current Flink 1.20 release manager.
> > > > > > >
> > > > > > > In the last 12 months, Rui has been the most active contributor
> > in
> > > > the
> > > > > > > Flink Kubernetes Operator project, while being the 2nd most
> > active
> > > > Flink
> > > > > > > contributor at the same time.
> > > > > > >
> > > > > > > Please join me in welcoming and congratulating Fan Rui!
> > > > > > >
> > > > > > > Best,
> > > > > > > Piotrek (on behalf of the Flink PMC)
> > > > > > >
> > > > > >
> > > >
> > >
> >
>

Re: Poor Load Balancing across TaskManagers for Multiple Kafka Sources

[Kevin Lam]

I've also just found https://issues.apache.org/jira/browse/FLINK-31762
which tracks the Kafka specific issue.

On Wed, Jun 5, 2024 at 3:05 PM Kevin Lam  wrote:

> cc. panyuep...@apache.org as related to FLIP-370
>
> On Wed, Jun 5, 2024 at 2:32 PM Kevin Lam  wrote:
>
>> Hey all,
>>
>> I'm seeing an issue with poor load balancing across TaskManagers for
>> Kafka Sources using the Flink SQL API and wondering if FLIP-370 will help
>> with it, or if not, interested in any ideas the community has to mitigate
>> the issue.
>>
>> The Kafka SplitEnumerator uses the following logic to assign split owners
>> (code pointer
>> 
>> ):
>>
>> ```
>>   static int getSplitOwner(TopicPartition tp, int numReaders) {
>> int startIndex = ((tp.topic().hashCode() * 31) & 0x7FFF) %
>> numReaders;
>> return (startIndex + tp.partition()) % numReaders;
>> }
>> ```
>>
>> However this can result in imbalanced distribution of kafka partition
>> consumers across task managers.
>>
>> To illustrate, I created a pipeline that consumes from 2 kafka topics,
>> each with 8 partitions, and just sinks them to a blackhole connector sink.
>> For a parallelism of 16 and 1 task slot per TaskManager, we'd ideally
>> expect each TaskManager to get its own kafka partition. ie. 16 partitions
>> (8 partitions from each topic) split evenly across TaskManagers. However,
>> due the algorithm I linked and how the startIndex is computed, I have
>> observed a bunch of TaskManagers with 2 partitions (one from each topic),
>> and some TaskManager completely idle.
>>
>> I've also run an experiment with the same pipeline where I set
>> parallelism such that each task manager gets exactly 1 partition, and
>> compared it against when each task manager gets exactly 2 partitions (one
>> from each topic). I ensured this was the case by setting an appropriate
>> parallelism, and ran the jobs on an application cluster. Since the
>> partitions are fixed, the extra parallelism if any isn't used. The case
>> where there is exactly 1 partition per TaskManager processes a fixed set of
>> data 20% faster.
>>
>> I was reading FLIP-370
>> 
>> and understand it will improve task scheduling in certain scenarios. Will
>> FLIP-370 help with this KafkaSource scenario? If not any ideas to improve
>> the subtask scheduling for KafkaSources? Ideally we don't need to carefully
>> consider the partition + resulting task distribution when selecting our
>> parallelism values.
>>
>> Thanks for your help!
>>
>

Re: Poor Load Balancing across TaskManagers for Multiple Kafka Sources

[Kevin Lam]

cc. panyuep...@apache.org as related to FLIP-370

On Wed, Jun 5, 2024 at 2:32 PM Kevin Lam  wrote:

> Hey all,
>
> I'm seeing an issue with poor load balancing across TaskManagers for Kafka
> Sources using the Flink SQL API and wondering if FLIP-370 will help with
> it, or if not, interested in any ideas the community has to mitigate the
> issue.
>
> The Kafka SplitEnumerator uses the following logic to assign split owners 
> (code
> pointer
> 
> ):
>
> ```
>   static int getSplitOwner(TopicPartition tp, int numReaders) {
> int startIndex = ((tp.topic().hashCode() * 31) & 0x7FFF) %
> numReaders;
> return (startIndex + tp.partition()) % numReaders;
> }
> ```
>
> However this can result in imbalanced distribution of kafka partition
> consumers across task managers.
>
> To illustrate, I created a pipeline that consumes from 2 kafka topics,
> each with 8 partitions, and just sinks them to a blackhole connector sink.
> For a parallelism of 16 and 1 task slot per TaskManager, we'd ideally
> expect each TaskManager to get its own kafka partition. ie. 16 partitions
> (8 partitions from each topic) split evenly across TaskManagers. However,
> due the algorithm I linked and how the startIndex is computed, I have
> observed a bunch of TaskManagers with 2 partitions (one from each topic),
> and some TaskManager completely idle.
>
> I've also run an experiment with the same pipeline where I set parallelism
> such that each task manager gets exactly 1 partition, and compared it
> against when each task manager gets exactly 2 partitions (one from each
> topic). I ensured this was the case by setting an appropriate parallelism,
> and ran the jobs on an application cluster. Since the partitions are fixed,
> the extra parallelism if any isn't used. The case where there is exactly 1
> partition per TaskManager processes a fixed set of data 20% faster.
>
> I was reading FLIP-370
> 
> and understand it will improve task scheduling in certain scenarios. Will
> FLIP-370 help with this KafkaSource scenario? If not any ideas to improve
> the subtask scheduling for KafkaSources? Ideally we don't need to carefully
> consider the partition + resulting task distribution when selecting our
> parallelism values.
>
> Thanks for your help!
>

Poor Load Balancing across TaskManagers for Multiple Kafka Sources

[Kevin Lam]

Hey all,

I'm seeing an issue with poor load balancing across TaskManagers for Kafka
Sources using the Flink SQL API and wondering if FLIP-370 will help with
it, or if not, interested in any ideas the community has to mitigate the
issue.

The Kafka SplitEnumerator uses the following logic to assign split owners (code
pointer

):

```
  static int getSplitOwner(TopicPartition tp, int numReaders) {
int startIndex = ((tp.topic().hashCode() * 31) & 0x7FFF) %
numReaders;
return (startIndex + tp.partition()) % numReaders;
}
```

However this can result in imbalanced distribution of kafka partition
consumers across task managers.

To illustrate, I created a pipeline that consumes from 2 kafka topics, each
with 8 partitions, and just sinks them to a blackhole connector sink. For a
parallelism of 16 and 1 task slot per TaskManager, we'd ideally expect each
TaskManager to get its own kafka partition. ie. 16 partitions (8 partitions
from each topic) split evenly across TaskManagers. However, due the
algorithm I linked and how the startIndex is computed, I have observed a
bunch of TaskManagers with 2 partitions (one from each topic), and some
TaskManager completely idle.

I've also run an experiment with the same pipeline where I set parallelism
such that each task manager gets exactly 1 partition, and compared it
against when each task manager gets exactly 2 partitions (one from each
topic). I ensured this was the case by setting an appropriate parallelism,
and ran the jobs on an application cluster. Since the partitions are fixed,
the extra parallelism if any isn't used. The case where there is exactly 1
partition per TaskManager processes a fixed set of data 20% faster.

I was reading FLIP-370

and understand it will improve task scheduling in certain scenarios. Will
FLIP-370 help with this KafkaSource scenario? If not any ideas to improve
the subtask scheduling for KafkaSources? Ideally we don't need to carefully
consider the partition + resulting task distribution when selecting our
parallelism values.

Thanks for your help!

[jira] [Created] (FLINK-35532) Prevent Cross-Site Authentication (XSA) attacks on Flink dashboard

[Hong Liang Teoh (Jira)]

Hong Liang Teoh created FLINK-35532:
---

 Summary: Prevent Cross-Site Authentication (XSA) attacks on Flink 
dashboard
 Key: FLINK-35532
 URL: https://issues.apache.org/jira/browse/FLINK-35532
 Project: Flink
  Issue Type: Technical Debt
  Components: Runtime / Web Frontend
Affects Versions: 1.19.0, 1.19.1
Reporter: Hong Liang Teoh
Assignee: Hong Liang Teoh
 Fix For: 1.19.2


As part of FLINK-33325, we introduced a new tab on the Flink dashboard to 
trigger the async profiler on the JobManager and TaskManager.

 

The HTML component introduced links out to async profiler page on Github -> 
[https://github.com/async-profiler/async-profiler/wiki].

However, the anchor element introduced does not follow best practices around 
preventing XSA attacks, by setting up the below:
{code:java}
target="_blank" rel="noopener noreferrer"{code}
We should add these attributes as best practice!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-461: FLIP-461: Synchronize rescaling with checkpoint creation to minimize reprocessing

[Matthias Pohl]

Hi Zakelly,
thanks for your reply. See my inlined responses below:

On Wed, Jun 5, 2024 at 10:26 AM Zakelly Lan  wrote:

> Hi Matthias,
>
> Thanks for your proposal! I have a few questions:
>
> 1. Is it possible a change event observed right after a complete checkpoint
> (or within a specific short time after a checkpoint) that triggers a
> rescale immediately? Sometimes the checkpoint interval is huge and it is
> better to rescale immediately.
>

That's something that could be considered as another optimization. I would
consider this as a possible follow-up. My concern here is that we'd make
the rescaling configuration even more complicated by introducing yet
another parameter.


> 2. Should we introduce `CheckpointLifecycleListener` instead of reusing
> `CheckpointListener`? Is `CheckpointListener` enough for this scenario?
>

Good point, they are serving similar purposes. But I'm hesitant to use
CheckpointListener (which is a public interface) for this internal quite
narrowly scoped runtime-specific use case of FLIP-461.

It might be worth renaming the internal interface into something that
indicates its internal usage to avoid confusion.


> Best,
> Zakelly
>
> On Wed, Jun 5, 2024 at 3:02 PM Matthias Pohl  wrote:
>
> > Hi ConradJam,
> > thanks for your response.
> >
> > The CheckpointStatsTracker gets notified about the checkpoint completion
> > after the checkpoint is finalized, i.e. all its data is persisted and the
> > metadata is written to the CompletedCheckpointStore. At this moment, the
> > checkpoint is considered for restoring a job and, therefore, becomes
> > available for restarts. This workflow also applies to unaligned
> > checkpoints. But I see how this context might be helpful for
> understanding
> > the change. I will add it to the FLIP. So far, I don't see a reason
> > to disable the feature for unaligned checkpoints. Do you see other issues
> > that might make it necessary to disable this feature for this type of
> > checkpoints?
> >
> > Can you elaborate a bit more what you mean by "checkpoints that do not
> > check it"? I do not fully understand what you are referring to with "it"
> > here.
> >
> > Best,
> > Matthias
> >
> > On Wed, Jun 5, 2024 at 4:46 AM ConradJam  wrote:
> >
> > > I have a few questions:
> > > Unaligned checkpoints Do we need to enable this feature? Whether this
> > > feature should be disabled for checkpoints that do not check it
> > >
> > > Matthias Pohl  于2024年6月4日周二 18:03写道：
> > >
> > > > Hi everyone,
> > > > I'd like to discuss FLIP-461 [1]. The FLIP proposes the
> synchronization
> > > of
> > > > rescaling and the completion of checkpoints. The idea is to reduce
> the
> > > > amount of data that needs to be processed after rescaling happened. A
> > > more
> > > > detailed motivation can be found in FLIP-461.
> > > >
> > > > I'm looking forward to feedback and suggestions.
> > > >
> > > > Best,
> > > > Matthias
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-461%3A+Synchronize+rescaling+with+checkpoint+creation+to+minimize+reprocessing
> > > >
> > >
> > >
> > > --
> > > Best
> > >
> > > ConradJam
> > >
> >
>

Re: [DISCUSS] FLIP-461: FLIP-461: Synchronize rescaling with checkpoint creation to minimize reprocessing

[Matthias Pohl]

Thanks Rui for your reply. Find my answers inlined below:

[1]
https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#jobmanager-adaptive-scheduler-resource-stabilization-timeout

On Wed, Jun 5, 2024 at 10:16 AM Rui Fan <1996fan...@gmail.com> wrote:

> Thanks Matthias for driving this proposal!
>
> This proposal can reduce the amount of data that is processed repeatedly
> after rescaling, so this proposal makes sense to me.
>
> I have some questions:
> 1. The public change only includes the "New Configuration Parameters"
> part, right?
>

Correct. I updated the section to make this a bit clearer.


> 2. jobmanager.adaptive-scheduler.rescale-on-failed-checkpoints-count is
> obviously a config option for users. But I'm not sure whether
> jobmanager.adaptive-scheduler.max-delay-for-rescale-trigger is a config
> option or an internal logic? I saw it's computed by
> rescale-on-failed-checkpoints-count.
>

That's a fair point. I wanted the user to be able to go back to the old
implementation even if checkpointing is enabled. One could argue that there
is no need for the parameter being expressed through a Duration. The only
motivation of delaying the rescaling might be waiting for consecutive
change events (which is similar to what we already have with
resource-stabilization-timeout that is utilized in the WaitingForResource
state [1]). Maybe, let's wait for other feedback here.


> 3. I'm not sure if the default value of rescale-on-failed-checkpoints-count
> should be 1 or is greater than 1 better?
>If 1 as the default value, when the checkpoint fails occasionally, and
> rescale happens, flink job will process a series of repeated data as well.
>If 2 as the default value, when the checkpoint fails occasionally, and
> the next checkpoint succeeds, the flink job won't process repeated data.
>

You're right. Using 2 as a default value sounds reasonable to work around
occasional "hiccups". My main motivation to set it to 1 was to be as close
as possible to the current (pre-FLIP-461) behavior where the rescale
happens immediately.

But I start to lean towards following your proposal here. I won't update
the FLIP in this regard for now to see what others have to say.

4. The description of rescale-on-failed-checkpoints-count is
>   "The number of subsequent failed checkpoints that will initiate
> rescaling."
>   IIUC, the "consecutive" is more accurate than subsequent here. WDYT?
>

Good idea. I will update the FLIP accordingly.


> 5. Proposed Changes part is specific implementation, I'm not sure whether
>all internal interfaces are best for the current version. So I cannot
> give any suggestion or feedback for now. But I'm happy to review them when
> your PR is ready if I have time.
>   Feel free to cc me (I'm interested in Adaptive Scheduler)
>

Will do.


> 6. This proposal aims to improve one logic inside of Adaptive Scheduler.
>Would you mind mentioning Adaptive Scheduler in the FLIP title? It will
>be useful for users to understand which component this proposal belongs
> to.
>

Good point. I updated the FLIPs title.


>
> Also, I also don't understand why this proposal needs to care about the
> checkpoint type is unaligned checkpoint or aligned checkpoint.
>
> Please correct me if anything is wrong, thanks.
>
> Best,
> Rui

On Wed, Jun 5, 2024 at 3:01 PM Matthias Pohl  wrote:
>
> > Hi ConradJam,
> > thanks for your response.
> >
> > The CheckpointStatsTracker gets notified about the checkpoint completion
> > after the checkpoint is finalized, i.e. all its data is persisted and the
> > metadata is written to the CompletedCheckpointStore. At this moment, the
> > checkpoint is considered for restoring a job and, therefore, becomes
> > available for restarts. This workflow also applies to unaligned
> > checkpoints. But I see how this context might be helpful for
> understanding
> > the change. I will add it to the FLIP. So far, I don't see a reason
> > to disable the feature for unaligned checkpoints. Do you see other issues
> > that might make it necessary to disable this feature for this type of
> > checkpoints?
> >
> > Can you elaborate a bit more what you mean by "checkpoints that do not
> > check it"? I do not fully understand what you are referring to with "it"
> > here.
> >
> > Best,
> > Matthias
> >
> > On Wed, Jun 5, 2024 at 4:46 AM ConradJam  wrote:
> >
> > > I have a few questions:
> > > Unaligned checkpoints Do we need to enable this feature? Whether this
> > > feature should be disabled for checkpoints that do not check it
> > >
> > > Matthias Pohl  于2024年6月4日周二 18:03写道：
> > >
> > > > Hi everyone,
> > > > I'd like to discuss FLIP-461 [1]. The FLIP proposes the
> synchronization
> > > of
> > > > rescaling and the completion of checkpoints. The idea is to reduce
> the
> > > > amount of data that needs to be processed after rescaling happened. A
> > > more
> > > > detailed motivation can be found in FLIP-461.
> > > >
> > > > I'm looking forward to feedback and 

[jira] [Created] (FLINK-35531) Avoid calling hsync in flush method in BaseHadoopFsRecoverableFsDataOutputStream

[dzcxzl (Jira)]

dzcxzl created FLINK-35531:
--

 Summary: Avoid calling hsync in flush method in 
BaseHadoopFsRecoverableFsDataOutputStream
 Key: FLINK-35531
 URL: https://issues.apache.org/jira/browse/FLINK-35531
 Project: Flink
  Issue Type: Bug
Reporter: dzcxzl






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35530) protobuf-format support discard unknow field Improve deserialization performance

[JingWei Li (Jira)]

JingWei Li created FLINK-35530:
--

 Summary: protobuf-format support discard unknow field Improve 
deserialization performance
 Key: FLINK-35530
 URL: https://issues.apache.org/jira/browse/FLINK-35530
 Project: Flink
  Issue Type: Improvement
  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Reporter: JingWei Li


Add a protobuf option that allows calling 
_CodedStreamHelper.discardUnknownFields_ to save the performance overhead of 
deserializing unknown fields when decoding data.
{code:java}
create table source (...) with (
 'format' = 'protobuf',
 'protobuf.discard-unknown-field' = 'true'
){code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

[Rodrigo Meneses]

All the best

On Wed, Jun 5, 2024 at 5:56 AM xiangyu feng  wrote:

> Congratulations, Rui!
>
> Regards,
> Xiangyu Feng
>
> Feng Jin  于2024年6月5日周三 20:42写道：
>
> > Congratulations, Rui!
> >
> >
> > Best,
> > Feng Jin
> >
> > On Wed, Jun 5, 2024 at 8:23 PM Yanfei Lei  wrote:
> >
> > > Congratulations, Rui!
> > >
> > > Best,
> > > Yanfei
> > >
> > > Luke Chen  于2024年6月5日周三 20:08写道：
> > > >
> > > > Congrats, Rui!
> > > >
> > > > Luke
> > > >
> > > > On Wed, Jun 5, 2024 at 8:02 PM Jiabao Sun 
> > wrote:
> > > >
> > > > > Congrats, Rui. Well-deserved!
> > > > >
> > > > > Best,
> > > > > Jiabao
> > > > >
> > > > > Zhanghao Chen  于2024年6月5日周三 19:29写道：
> > > > >
> > > > > > Congrats, Rui!
> > > > > >
> > > > > > Best,
> > > > > > Zhanghao Chen
> > > > > > 
> > > > > > From: Piotr Nowojski 
> > > > > > Sent: Wednesday, June 5, 2024 18:01
> > > > > > To: dev ; rui fan <1996fan...@gmail.com>
> > > > > > Subject: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui
> > > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > On behalf of the PMC, I'm very happy to announce another new
> Apache
> > > Flink
> > > > > > PMC Member - Fan Rui.
> > > > > >
> > > > > > Rui has been active in the community since August 2019. During
> this
> > > time
> > > > > he
> > > > > > has contributed a lot of new features. Among others:
> > > > > >   - Decoupling Autoscaler from Kubernetes Operator, and
> supporting
> > > > > > Standalone Autoscaler
> > > > > >   - Improvements to checkpointing, flamegraphs, restart
> strategies,
> > > > > > watermark alignment, network shuffles
> > > > > >   - Optimizing the memory and CPU usage of large operators,
> greatly
> > > > > > reducing the risk and probability of TaskManager OOM
> > > > > >
> > > > > > He reviewed a significant amount of PRs and has been active both
> on
> > > the
> > > > > > mailing lists and in Jira helping to both maintain and grow
> Apache
> > > > > Flink's
> > > > > > community. He is also our current Flink 1.20 release manager.
> > > > > >
> > > > > > In the last 12 months, Rui has been the most active contributor
> in
> > > the
> > > > > > Flink Kubernetes Operator project, while being the 2nd most
> active
> > > Flink
> > > > > > contributor at the same time.
> > > > > >
> > > > > > Please join me in welcoming and congratulating Fan Rui!
> > > > > >
> > > > > > Best,
> > > > > > Piotrek (on behalf of the Flink PMC)
> > > > > >
> > > > >
> > >
> >
>

[jira] [Created] (FLINK-35529) protobuf-format compatible protobuf bad indentifier

[JingWei Li (Jira)]

JingWei Li created FLINK-35529:
--

 Summary: protobuf-format compatible protobuf bad indentifier
 Key: FLINK-35529
 URL: https://issues.apache.org/jira/browse/FLINK-35529
 Project: Flink
  Issue Type: Improvement
  Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
Affects Versions: 1.17.2
Reporter: JingWei Li
 Fix For: 2.0.0


The main bug occurs during the decode process. The decode method is a method 
generated by the codegen of Flink at runtime, and in the process of generating 
the decode method, some getter and setter methods of the protobuf object need 
to be used to construct the RowData. Currently, the way to generate the getter 
and setter is through string concatenation, using the "get" prefix and 
camelCase variable names. Some special characters may lead to errors in the 
generated Getter and Setter methods, thus causing bugs.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35528) Skip execution of interruptible mails when yielding

[Piotr Nowojski (Jira)]

Piotr Nowojski created FLINK-35528:
--

 Summary: Skip execution of interruptible mails when yielding
 Key: FLINK-35528
 URL: https://issues.apache.org/jira/browse/FLINK-35528
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Checkpointing, Runtime / Task
Affects Versions: 1.20.0
Reporter: Piotr Nowojski
Assignee: Piotr Nowojski


When operators are yielding, for example waiting for async state access to 
complete before a checkpoint, it would be beneficial to not execute 
interruptible mails. Otherwise continuation mail for firing timers would be 
continuously re-enqeueed. To achieve that MailboxExecutor must be aware which 
mails are interruptible.

The easiest way to achieve this is to set MIN_PRIORITY for interruptible mails.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35527) Polish quickstart guide & clean stale links in docs

[yux (Jira)]

yux created FLINK-35527:
---

 Summary: Polish quickstart guide & clean stale links in docs
 Key: FLINK-35527
 URL: https://issues.apache.org/jira/browse/FLINK-35527
 Project: Flink
  Issue Type: Bug
  Components: Flink CDC
Affects Versions: cdc-3.1.0
Reporter: yux
 Fix For: cdc-3.2.0


Currently, there's still a lot of stale links in Flink CDC docs, including some 
download links pointing to Ververica maven repositories. Need to clean them up 
to avoid user conflicts.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

[xiangyu feng]

Congratulations, Rui!

Regards,
Xiangyu Feng

Feng Jin  于2024年6月5日周三 20:42写道：

> Congratulations, Rui!
>
>
> Best,
> Feng Jin
>
> On Wed, Jun 5, 2024 at 8:23 PM Yanfei Lei  wrote:
>
> > Congratulations, Rui!
> >
> > Best,
> > Yanfei
> >
> > Luke Chen  于2024年6月5日周三 20:08写道：
> > >
> > > Congrats, Rui!
> > >
> > > Luke
> > >
> > > On Wed, Jun 5, 2024 at 8:02 PM Jiabao Sun 
> wrote:
> > >
> > > > Congrats, Rui. Well-deserved!
> > > >
> > > > Best,
> > > > Jiabao
> > > >
> > > > Zhanghao Chen  于2024年6月5日周三 19:29写道：
> > > >
> > > > > Congrats, Rui!
> > > > >
> > > > > Best,
> > > > > Zhanghao Chen
> > > > > 
> > > > > From: Piotr Nowojski 
> > > > > Sent: Wednesday, June 5, 2024 18:01
> > > > > To: dev ; rui fan <1996fan...@gmail.com>
> > > > > Subject: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui
> > > > >
> > > > > Hi everyone,
> > > > >
> > > > > On behalf of the PMC, I'm very happy to announce another new Apache
> > Flink
> > > > > PMC Member - Fan Rui.
> > > > >
> > > > > Rui has been active in the community since August 2019. During this
> > time
> > > > he
> > > > > has contributed a lot of new features. Among others:
> > > > >   - Decoupling Autoscaler from Kubernetes Operator, and supporting
> > > > > Standalone Autoscaler
> > > > >   - Improvements to checkpointing, flamegraphs, restart strategies,
> > > > > watermark alignment, network shuffles
> > > > >   - Optimizing the memory and CPU usage of large operators, greatly
> > > > > reducing the risk and probability of TaskManager OOM
> > > > >
> > > > > He reviewed a significant amount of PRs and has been active both on
> > the
> > > > > mailing lists and in Jira helping to both maintain and grow Apache
> > > > Flink's
> > > > > community. He is also our current Flink 1.20 release manager.
> > > > >
> > > > > In the last 12 months, Rui has been the most active contributor in
> > the
> > > > > Flink Kubernetes Operator project, while being the 2nd most active
> > Flink
> > > > > contributor at the same time.
> > > > >
> > > > > Please join me in welcoming and congratulating Fan Rui!
> > > > >
> > > > > Best,
> > > > > Piotrek (on behalf of the Flink PMC)
> > > > >
> > > >
> >
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

[Feng Jin]

Congratulations, Rui!


Best,
Feng Jin

On Wed, Jun 5, 2024 at 8:23 PM Yanfei Lei  wrote:

> Congratulations, Rui!
>
> Best,
> Yanfei
>
> Luke Chen  于2024年6月5日周三 20:08写道：
> >
> > Congrats, Rui!
> >
> > Luke
> >
> > On Wed, Jun 5, 2024 at 8:02 PM Jiabao Sun  wrote:
> >
> > > Congrats, Rui. Well-deserved!
> > >
> > > Best,
> > > Jiabao
> > >
> > > Zhanghao Chen  于2024年6月5日周三 19:29写道：
> > >
> > > > Congrats, Rui!
> > > >
> > > > Best,
> > > > Zhanghao Chen
> > > > 
> > > > From: Piotr Nowojski 
> > > > Sent: Wednesday, June 5, 2024 18:01
> > > > To: dev ; rui fan <1996fan...@gmail.com>
> > > > Subject: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui
> > > >
> > > > Hi everyone,
> > > >
> > > > On behalf of the PMC, I'm very happy to announce another new Apache
> Flink
> > > > PMC Member - Fan Rui.
> > > >
> > > > Rui has been active in the community since August 2019. During this
> time
> > > he
> > > > has contributed a lot of new features. Among others:
> > > >   - Decoupling Autoscaler from Kubernetes Operator, and supporting
> > > > Standalone Autoscaler
> > > >   - Improvements to checkpointing, flamegraphs, restart strategies,
> > > > watermark alignment, network shuffles
> > > >   - Optimizing the memory and CPU usage of large operators, greatly
> > > > reducing the risk and probability of TaskManager OOM
> > > >
> > > > He reviewed a significant amount of PRs and has been active both on
> the
> > > > mailing lists and in Jira helping to both maintain and grow Apache
> > > Flink's
> > > > community. He is also our current Flink 1.20 release manager.
> > > >
> > > > In the last 12 months, Rui has been the most active contributor in
> the
> > > > Flink Kubernetes Operator project, while being the 2nd most active
> Flink
> > > > contributor at the same time.
> > > >
> > > > Please join me in welcoming and congratulating Fan Rui!
> > > >
> > > > Best,
> > > > Piotrek (on behalf of the Flink PMC)
> > > >
> > >
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

[Yanfei Lei]

Congratulations, Rui!

Best,
Yanfei

Luke Chen  于2024年6月5日周三 20:08写道：
>
> Congrats, Rui!
>
> Luke
>
> On Wed, Jun 5, 2024 at 8:02 PM Jiabao Sun  wrote:
>
> > Congrats, Rui. Well-deserved!
> >
> > Best,
> > Jiabao
> >
> > Zhanghao Chen  于2024年6月5日周三 19:29写道：
> >
> > > Congrats, Rui!
> > >
> > > Best,
> > > Zhanghao Chen
> > > 
> > > From: Piotr Nowojski 
> > > Sent: Wednesday, June 5, 2024 18:01
> > > To: dev ; rui fan <1996fan...@gmail.com>
> > > Subject: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui
> > >
> > > Hi everyone,
> > >
> > > On behalf of the PMC, I'm very happy to announce another new Apache Flink
> > > PMC Member - Fan Rui.
> > >
> > > Rui has been active in the community since August 2019. During this time
> > he
> > > has contributed a lot of new features. Among others:
> > >   - Decoupling Autoscaler from Kubernetes Operator, and supporting
> > > Standalone Autoscaler
> > >   - Improvements to checkpointing, flamegraphs, restart strategies,
> > > watermark alignment, network shuffles
> > >   - Optimizing the memory and CPU usage of large operators, greatly
> > > reducing the risk and probability of TaskManager OOM
> > >
> > > He reviewed a significant amount of PRs and has been active both on the
> > > mailing lists and in Jira helping to both maintain and grow Apache
> > Flink's
> > > community. He is also our current Flink 1.20 release manager.
> > >
> > > In the last 12 months, Rui has been the most active contributor in the
> > > Flink Kubernetes Operator project, while being the 2nd most active Flink
> > > contributor at the same time.
> > >
> > > Please join me in welcoming and congratulating Fan Rui!
> > >
> > > Best,
> > > Piotrek (on behalf of the Flink PMC)
> > >
> >

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

[Luke Chen]

Congrats, Rui!

Luke

On Wed, Jun 5, 2024 at 8:02 PM Jiabao Sun  wrote:

> Congrats, Rui. Well-deserved!
>
> Best,
> Jiabao
>
> Zhanghao Chen  于2024年6月5日周三 19:29写道：
>
> > Congrats, Rui!
> >
> > Best,
> > Zhanghao Chen
> > 
> > From: Piotr Nowojski 
> > Sent: Wednesday, June 5, 2024 18:01
> > To: dev ; rui fan <1996fan...@gmail.com>
> > Subject: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui
> >
> > Hi everyone,
> >
> > On behalf of the PMC, I'm very happy to announce another new Apache Flink
> > PMC Member - Fan Rui.
> >
> > Rui has been active in the community since August 2019. During this time
> he
> > has contributed a lot of new features. Among others:
> >   - Decoupling Autoscaler from Kubernetes Operator, and supporting
> > Standalone Autoscaler
> >   - Improvements to checkpointing, flamegraphs, restart strategies,
> > watermark alignment, network shuffles
> >   - Optimizing the memory and CPU usage of large operators, greatly
> > reducing the risk and probability of TaskManager OOM
> >
> > He reviewed a significant amount of PRs and has been active both on the
> > mailing lists and in Jira helping to both maintain and grow Apache
> Flink's
> > community. He is also our current Flink 1.20 release manager.
> >
> > In the last 12 months, Rui has been the most active contributor in the
> > Flink Kubernetes Operator project, while being the 2nd most active Flink
> > contributor at the same time.
> >
> > Please join me in welcoming and congratulating Fan Rui!
> >
> > Best,
> > Piotrek (on behalf of the Flink PMC)
> >
>

[jira] [Created] (FLINK-35526) Remove deprecated stedolan/jq Docker image from Flink e2e tests

[Robert Metzger (Jira)]

Robert Metzger created FLINK-35526:
--

 Summary: Remove deprecated stedolan/jq Docker image from Flink e2e 
tests
 Key: FLINK-35526
 URL: https://issues.apache.org/jira/browse/FLINK-35526
 Project: Flink
  Issue Type: Bug
  Components: Test Infrastructure
Reporter: Robert Metzger
Assignee: Robert Metzger


Our CI logs contain this warning: 
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=60060=logs=af184cdd-c6d8-5084-0b69-7e9c67b35f7a=0f3adb59-eefa-51c6-2858-3654d9e0749d=3828

{code}
latest: Pulling from stedolan/jq
[DEPRECATION NOTICE] Docker Image Format v1, and Docker Image manifest version 
2, schema 1 support will be removed in an upcoming release. Suggest the author 
of docker.io/stedolan/jq:latest to upgrade the image to the OCI Format, or 
Docker Image manifest v2, schema 2. More information at 
https://docs.docker.com/go/deprecated-image-specs/
{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

[Jiabao Sun]

Congrats, Rui. Well-deserved!

Best,
Jiabao

Zhanghao Chen  于2024年6月5日周三 19:29写道：

> Congrats, Rui!
>
> Best,
> Zhanghao Chen
> 
> From: Piotr Nowojski 
> Sent: Wednesday, June 5, 2024 18:01
> To: dev ; rui fan <1996fan...@gmail.com>
> Subject: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui
>
> Hi everyone,
>
> On behalf of the PMC, I'm very happy to announce another new Apache Flink
> PMC Member - Fan Rui.
>
> Rui has been active in the community since August 2019. During this time he
> has contributed a lot of new features. Among others:
>   - Decoupling Autoscaler from Kubernetes Operator, and supporting
> Standalone Autoscaler
>   - Improvements to checkpointing, flamegraphs, restart strategies,
> watermark alignment, network shuffles
>   - Optimizing the memory and CPU usage of large operators, greatly
> reducing the risk and probability of TaskManager OOM
>
> He reviewed a significant amount of PRs and has been active both on the
> mailing lists and in Jira helping to both maintain and grow Apache Flink's
> community. He is also our current Flink 1.20 release manager.
>
> In the last 12 months, Rui has been the most active contributor in the
> Flink Kubernetes Operator project, while being the 2nd most active Flink
> contributor at the same time.
>
> Please join me in welcoming and congratulating Fan Rui!
>
> Best,
> Piotrek (on behalf of the Flink PMC)
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

[Zhanghao Chen]

Congrats, Rui!

Best,
Zhanghao Chen

From: Piotr Nowojski 
Sent: Wednesday, June 5, 2024 18:01
To: dev ; rui fan <1996fan...@gmail.com>
Subject: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

Hi everyone,

On behalf of the PMC, I'm very happy to announce another new Apache Flink
PMC Member - Fan Rui.

Rui has been active in the community since August 2019. During this time he
has contributed a lot of new features. Among others:
  - Decoupling Autoscaler from Kubernetes Operator, and supporting
Standalone Autoscaler
  - Improvements to checkpointing, flamegraphs, restart strategies,
watermark alignment, network shuffles
  - Optimizing the memory and CPU usage of large operators, greatly
reducing the risk and probability of TaskManager OOM

He reviewed a significant amount of PRs and has been active both on the
mailing lists and in Jira helping to both maintain and grow Apache Flink's
community. He is also our current Flink 1.20 release manager.

In the last 12 months, Rui has been the most active contributor in the
Flink Kubernetes Operator project, while being the 2nd most active Flink
contributor at the same time.

Please join me in welcoming and congratulating Fan Rui!

Best,
Piotrek (on behalf of the Flink PMC)

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

[Yun Tang]

Congratulations, Rui!


Best,
Yun Tang

From: weijie guo 
Sent: Wednesday, June 5, 2024 19:12
To: dev@flink.apache.org 
Subject: Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

Congratulations, Rui. Well-deserved!

Best regards,

Weijie


Zakelly Lan  于2024年6月5日周三 18:05写道：

> Congratulations, Rui!
>
> Best,
> Zakelly
>
> On Wed, Jun 5, 2024 at 6:02 PM Piotr Nowojski 
> wrote:
>
> > Hi everyone,
> >
> > On behalf of the PMC, I'm very happy to announce another new Apache Flink
> > PMC Member - Fan Rui.
> >
> > Rui has been active in the community since August 2019. During this time
> he
> > has contributed a lot of new features. Among others:
> >   - Decoupling Autoscaler from Kubernetes Operator, and supporting
> > Standalone Autoscaler
> >   - Improvements to checkpointing, flamegraphs, restart strategies,
> > watermark alignment, network shuffles
> >   - Optimizing the memory and CPU usage of large operators, greatly
> > reducing the risk and probability of TaskManager OOM
> >
> > He reviewed a significant amount of PRs and has been active both on the
> > mailing lists and in Jira helping to both maintain and grow Apache
> Flink's
> > community. He is also our current Flink 1.20 release manager.
> >
> > In the last 12 months, Rui has been the most active contributor in the
> > Flink Kubernetes Operator project, while being the 2nd most active Flink
> > contributor at the same time.
> >
> > Please join me in welcoming and congratulating Fan Rui!
> >
> > Best,
> > Piotrek (on behalf of the Flink PMC)
> >
>

Re: Flink stateful functions and Agentic Architecture

[Yun Tang]

Hi David,

Since flink stateful has been inactive for some time[1], I'm not sure whether 
you could get some help here. From my understanding, your request to a 
distribute system is event driven processing + actor model. Perhaps you could 
enrich Ray[2] with continuous message processing to achieve this goal, since 
Ray already has the ability of object store.


BTW, since LLM is really popular in the current data + AI era, maybe you could 
also share some insights here after the investigation.


[1] https://lists.apache.org/thread/7cr2bgt91ppk6pz8o0nfbd10gs63nz6t
[2] https://docs.ray.io/en/latest/

Best,
Yun Tang

From: Yunfeng Zhou 
Sent: Monday, May 6, 2024 9:03
To: dev@flink.apache.org 
Subject: Re: Flink stateful functions and Agentic Architecture

Hi David,

I'm not very familiar with stateful functions, but I participated in
Flink ML, a machine learning infrastructure and algorithm library
based on Flink. There we developed functions like iteration based
model-training process and hot updating ML model during online
prediction. You may check if these functions and their corresponding
designs could be of your interest.
https://nightlies.apache.org/flink/flink-ml-docs-release-2.3/docs/development/iteration/
https://github.com/apache/flink-ml

Best,
Yunfeng

On Mon, Apr 29, 2024 at 9:35 PM David Carroll  wrote:
>
> I am a systems architect developing a POC concept for an AI product using
> Agentic Architecture with generative LLMs. It occurred to me that it could
> be possible to use Flink stateful functions to provide the event driven
> communications and execution environment for agents built with a framework
> like langchain or langgraph. Such a system could be multi-tenant and fully
> scalable.
>
> I wonder if anyone has explored using Flink stateful functions i this way
> of has suggestion of examples to look at for similar use cases?
> --
> David Carroll
> Chief of Research & Development
> 
> da...@spotter.la  | 310.569.5103
> 
> From TIME. © 2023 TIME USA LLC. All rights reserved. Used under license.
>
> IMPORTANT: The contents of this email and any attachments are confidential.
> They are intended for the named recipient(s) only. If you have received
> this email by mistake, please notify the sender immediately and do not
> disclose the contents to anyone or make copies thereof.

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

[weijie guo]

Congratulations, Rui. Well-deserved!

Best regards,

Weijie


Zakelly Lan  于2024年6月5日周三 18:05写道：

> Congratulations, Rui!
>
> Best,
> Zakelly
>
> On Wed, Jun 5, 2024 at 6:02 PM Piotr Nowojski 
> wrote:
>
> > Hi everyone,
> >
> > On behalf of the PMC, I'm very happy to announce another new Apache Flink
> > PMC Member - Fan Rui.
> >
> > Rui has been active in the community since August 2019. During this time
> he
> > has contributed a lot of new features. Among others:
> >   - Decoupling Autoscaler from Kubernetes Operator, and supporting
> > Standalone Autoscaler
> >   - Improvements to checkpointing, flamegraphs, restart strategies,
> > watermark alignment, network shuffles
> >   - Optimizing the memory and CPU usage of large operators, greatly
> > reducing the risk and probability of TaskManager OOM
> >
> > He reviewed a significant amount of PRs and has been active both on the
> > mailing lists and in Jira helping to both maintain and grow Apache
> Flink's
> > community. He is also our current Flink 1.20 release manager.
> >
> > In the last 12 months, Rui has been the most active contributor in the
> > Flink Kubernetes Operator project, while being the 2nd most active Flink
> > contributor at the same time.
> >
> > Please join me in welcoming and congratulating Fan Rui!
> >
> > Best,
> > Piotrek (on behalf of the Flink PMC)
> >
>

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

[Zakelly Lan]

Congratulations, Rui!

Best,
Zakelly

On Wed, Jun 5, 2024 at 6:02 PM Piotr Nowojski  wrote:

> Hi everyone,
>
> On behalf of the PMC, I'm very happy to announce another new Apache Flink
> PMC Member - Fan Rui.
>
> Rui has been active in the community since August 2019. During this time he
> has contributed a lot of new features. Among others:
>   - Decoupling Autoscaler from Kubernetes Operator, and supporting
> Standalone Autoscaler
>   - Improvements to checkpointing, flamegraphs, restart strategies,
> watermark alignment, network shuffles
>   - Optimizing the memory and CPU usage of large operators, greatly
> reducing the risk and probability of TaskManager OOM
>
> He reviewed a significant amount of PRs and has been active both on the
> mailing lists and in Jira helping to both maintain and grow Apache Flink's
> community. He is also our current Flink 1.20 release manager.
>
> In the last 12 months, Rui has been the most active contributor in the
> Flink Kubernetes Operator project, while being the 2nd most active Flink
> contributor at the same time.
>
> Please join me in welcoming and congratulating Fan Rui!
>
> Best,
> Piotrek (on behalf of the Flink PMC)
>

[ANNOUNCE] New Apache Flink PMC Member - Fan Rui

[Piotr Nowojski]

Hi everyone,

On behalf of the PMC, I'm very happy to announce another new Apache Flink
PMC Member - Fan Rui.

Rui has been active in the community since August 2019. During this time he
has contributed a lot of new features. Among others:
  - Decoupling Autoscaler from Kubernetes Operator, and supporting
Standalone Autoscaler
  - Improvements to checkpointing, flamegraphs, restart strategies,
watermark alignment, network shuffles
  - Optimizing the memory and CPU usage of large operators, greatly
reducing the risk and probability of TaskManager OOM

He reviewed a significant amount of PRs and has been active both on the
mailing lists and in Jira helping to both maintain and grow Apache Flink's
community. He is also our current Flink 1.20 release manager.

In the last 12 months, Rui has been the most active contributor in the
Flink Kubernetes Operator project, while being the 2nd most active Flink
contributor at the same time.

Please join me in welcoming and congratulating Fan Rui!

Best,
Piotrek (on behalf of the Flink PMC)

Re: [DISCUSS] FLIP-461: FLIP-461: Synchronize rescaling with checkpoint creation to minimize reprocessing

[Zakelly Lan]

Hi Matthias,

Thanks for your proposal! I have a few questions:

1. Is it possible a change event observed right after a complete checkpoint
(or within a specific short time after a checkpoint) that triggers a
rescale immediately? Sometimes the checkpoint interval is huge and it is
better to rescale immediately.
2. Should we introduce `CheckpointLifecycleListener` instead of reusing
`CheckpointListener`? Is `CheckpointListener` enough for this scenario?


Best,
Zakelly

On Wed, Jun 5, 2024 at 3:02 PM Matthias Pohl  wrote:

> Hi ConradJam,
> thanks for your response.
>
> The CheckpointStatsTracker gets notified about the checkpoint completion
> after the checkpoint is finalized, i.e. all its data is persisted and the
> metadata is written to the CompletedCheckpointStore. At this moment, the
> checkpoint is considered for restoring a job and, therefore, becomes
> available for restarts. This workflow also applies to unaligned
> checkpoints. But I see how this context might be helpful for understanding
> the change. I will add it to the FLIP. So far, I don't see a reason
> to disable the feature for unaligned checkpoints. Do you see other issues
> that might make it necessary to disable this feature for this type of
> checkpoints?
>
> Can you elaborate a bit more what you mean by "checkpoints that do not
> check it"? I do not fully understand what you are referring to with "it"
> here.
>
> Best,
> Matthias
>
> On Wed, Jun 5, 2024 at 4:46 AM ConradJam  wrote:
>
> > I have a few questions:
> > Unaligned checkpoints Do we need to enable this feature? Whether this
> > feature should be disabled for checkpoints that do not check it
> >
> > Matthias Pohl  于2024年6月4日周二 18:03写道：
> >
> > > Hi everyone,
> > > I'd like to discuss FLIP-461 [1]. The FLIP proposes the synchronization
> > of
> > > rescaling and the completion of checkpoints. The idea is to reduce the
> > > amount of data that needs to be processed after rescaling happened. A
> > more
> > > detailed motivation can be found in FLIP-461.
> > >
> > > I'm looking forward to feedback and suggestions.
> > >
> > > Best,
> > > Matthias
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-461%3A+Synchronize+rescaling+with+checkpoint+creation+to+minimize+reprocessing
> > >
> >
> >
> > --
> > Best
> >
> > ConradJam
> >
>

Re: [DISCUSS] FLIP-461: FLIP-461: Synchronize rescaling with checkpoint creation to minimize reprocessing

[Rui Fan]

Thanks Matthias for driving this proposal!

This proposal can reduce the amount of data that is processed repeatedly
after rescaling, so this proposal makes sense to me.

I have some questions:
1. The public change only includes the "New Configuration Parameters" part,
right?
2. jobmanager.adaptive-scheduler.rescale-on-failed-checkpoints-count is
obviously
  a config option for users. But I'm not sure whether
  jobmanager.adaptive-scheduler.max-delay-for-rescale-trigger is a config
option
 or an internal logic? I saw it's computed by
rescale-on-failed-checkpoints-count.
3. I'm not sure if the default value of rescale-on-failed-checkpoints-count
should
   be 1 or is greater than 1 better?
   If 1 as the default value, when the checkpoint fails occasionally, and
rescale happens,
   flink job will process a series of repeated data as well.
   If 2 as the default value, when the checkpoint fails occasionally, and
the next
   checkpoint succeeds, the flink job won't process repeated data.
4. The description of rescale-on-failed-checkpoints-count is
  "The number of subsequent failed checkpoints that will initiate
rescaling."
  IIUC, the "consecutive" is more accurate than subsequent here. WDYT?
5. Proposed Changes part is specific implementation, I'm not sure whether
   all internal interfaces are best for the current version. So I cannot
give
  any suggestion or feedback for now. But I'm happy to review them when
  your PR is ready if I have time.
  Feel free to cc me (I'm interested in Adaptive Scheduler)
6. This proposal aims to improve one logic inside of Adaptive Scheduler.
   Would you mind mentioning Adaptive Scheduler in the FLIP title? It will
   be useful for users to understand which component this proposal belongs
to.

Also, I also don't understand why this proposal needs to care about the
checkpoint type is unaligned checkpoint or aligned checkpoint.

Please correct me if anything is wrong, thanks.

Best,
Rui

On Wed, Jun 5, 2024 at 3:01 PM Matthias Pohl  wrote:

> Hi ConradJam,
> thanks for your response.
>
> The CheckpointStatsTracker gets notified about the checkpoint completion
> after the checkpoint is finalized, i.e. all its data is persisted and the
> metadata is written to the CompletedCheckpointStore. At this moment, the
> checkpoint is considered for restoring a job and, therefore, becomes
> available for restarts. This workflow also applies to unaligned
> checkpoints. But I see how this context might be helpful for understanding
> the change. I will add it to the FLIP. So far, I don't see a reason
> to disable the feature for unaligned checkpoints. Do you see other issues
> that might make it necessary to disable this feature for this type of
> checkpoints?
>
> Can you elaborate a bit more what you mean by "checkpoints that do not
> check it"? I do not fully understand what you are referring to with "it"
> here.
>
> Best,
> Matthias
>
> On Wed, Jun 5, 2024 at 4:46 AM ConradJam  wrote:
>
> > I have a few questions:
> > Unaligned checkpoints Do we need to enable this feature? Whether this
> > feature should be disabled for checkpoints that do not check it
> >
> > Matthias Pohl  于2024年6月4日周二 18:03写道：
> >
> > > Hi everyone,
> > > I'd like to discuss FLIP-461 [1]. The FLIP proposes the synchronization
> > of
> > > rescaling and the completion of checkpoints. The idea is to reduce the
> > > amount of data that needs to be processed after rescaling happened. A
> > more
> > > detailed motivation can be found in FLIP-461.
> > >
> > > I'm looking forward to feedback and suggestions.
> > >
> > > Best,
> > > Matthias
> > >
> > > [1]
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-461%3A+Synchronize+rescaling+with+checkpoint+creation+to+minimize+reprocessing
> > >
> >
> >
> > --
> > Best
> >
> > ConradJam
> >
>

[jira] [Created] (FLINK-35525) HDFS delegation token fetched by custom DelegationTokenProvider is not passed to Yarn AM

[Zhen Wang (Jira)]

Zhen Wang created FLINK-35525:
-

 Summary: HDFS  delegation token fetched by custom 
DelegationTokenProvider is not passed to Yarn AM
 Key: FLINK-35525
 URL: https://issues.apache.org/jira/browse/FLINK-35525
 Project: Flink
  Issue Type: Bug
  Components: Deployment / YARN
Affects Versions: 1.18.1, 1.19.0
Reporter: Zhen Wang


I tried running flink with hadoop proxy user by disabling HadoopModuleFactory 
and flink built-in token providers, and implementing a custom token provider.

However, only the hdfs token obtained by hadoopfs provider was added in 
YarnClusterDescriptor, which resulted in Yarn AM submission failure.

Discussion: https://github.com/apache/flink/pull/22009#issuecomment-2132676114



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-35524) Clear connections pools when reader exist.

[Hongshun Wang (Jira)]

Hongshun Wang created FLINK-35524:
-

 Summary: Clear connections pools when reader exist.
 Key: FLINK-35524
 URL: https://issues.apache.org/jira/browse/FLINK-35524
 Project: Flink
  Issue Type: Bug
  Components: Flink CDC
Affects Versions: cdc-3.1.1
Reporter: Hongshun Wang
 Fix For: cdc-3.2.0


Current, inJdbcConnectionPools is static instance, so the datasource pools in 
it won't be recycle when reader close. It will cause memory leak.

```java
public class JdbcConnectionPools implements ConnectionPools {

private static final Logger LOG = 
LoggerFactory.getLogger(JdbcConnectionPools.class);

private static JdbcConnectionPools instance;
private final Map pools = new HashMap<>();
private static final Map POOL_FACTORY_MAP = 
new HashMap<>();
```



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-461: FLIP-461: Synchronize rescaling with checkpoint creation to minimize reprocessing

[Matthias Pohl]

Hi ConradJam,
thanks for your response.

The CheckpointStatsTracker gets notified about the checkpoint completion
after the checkpoint is finalized, i.e. all its data is persisted and the
metadata is written to the CompletedCheckpointStore. At this moment, the
checkpoint is considered for restoring a job and, therefore, becomes
available for restarts. This workflow also applies to unaligned
checkpoints. But I see how this context might be helpful for understanding
the change. I will add it to the FLIP. So far, I don't see a reason
to disable the feature for unaligned checkpoints. Do you see other issues
that might make it necessary to disable this feature for this type of
checkpoints?

Can you elaborate a bit more what you mean by "checkpoints that do not
check it"? I do not fully understand what you are referring to with "it"
here.

Best,
Matthias

On Wed, Jun 5, 2024 at 4:46 AM ConradJam  wrote:

> I have a few questions:
> Unaligned checkpoints Do we need to enable this feature? Whether this
> feature should be disabled for checkpoints that do not check it
>
> Matthias Pohl  于2024年6月4日周二 18:03写道：
>
> > Hi everyone,
> > I'd like to discuss FLIP-461 [1]. The FLIP proposes the synchronization
> of
> > rescaling and the completion of checkpoints. The idea is to reduce the
> > amount of data that needs to be processed after rescaling happened. A
> more
> > detailed motivation can be found in FLIP-461.
> >
> > I'm looking forward to feedback and suggestions.
> >
> > Best,
> > Matthias
> >
> > [1]
> >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-461%3A+Synchronize+rescaling+with+checkpoint+creation+to+minimize+reprocessing
> >
>
>
> --
> Best
>
> ConradJam
>

[jira] [Created] (FLINK-35523) When using the Hive connector to read a Hive table in Parquet format, a null pointer exception is thrown.

[Jichao Wang (Jira)]

Jichao Wang created FLINK-35523:
---

 Summary: When using the Hive connector to read a Hive table in 
Parquet format, a null pointer exception is thrown.
 Key: FLINK-35523
 URL: https://issues.apache.org/jira/browse/FLINK-35523
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Hive
Affects Versions: 1.19.0, 1.16.2
Reporter: Jichao Wang


When using the Hive connector to read a Hive table in Parquet format, a null 
pointer exception is thrown.
The exception stack information is as follows:
{code:text}
java.lang.RuntimeException: SplitFetcher thread 0 received unexpected exception 
while polling the records
at 
org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:150)
 ~[flink-connector-files-1.16.2-HDP-24.06.1.jar:1.16.2-HDP-24.06.1]
at 
org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.run(SplitFetcher.java:105)
 [flink-connector-files-1.16.2-HDP-24.06.1.jar:1.16.2-HDP-24.06.1]
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
[?:1.8.0_342]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
[?:1.8.0_342]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
[?:1.8.0_342]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
[?:1.8.0_342]
at java.lang.Thread.run(Thread.java:750) [?:1.8.0_342]
Caused by: java.io.IOException: java.lang.IndexOutOfBoundsException: Index: 1, 
Size: 1
at 
org.apache.flink.connector.file.src.impl.HdpFileSourceSplitReader.fetch(HdpFileSourceSplitReader.java:40)
 ~[flink-connector-files-1.16.2-HDP-24.06.1.jar:1.16.2-HDP-24.06.1]
at 
org.apache.flink.connector.base.source.reader.fetcher.FetchTask.run(FetchTask.java:58)
 ~[flink-connector-files-1.16.2-HDP-24.06.1.jar:1.16.2-HDP-24.06.1]
at 
org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:142)
 ~[flink-connector-files-1.16.2-HDP-24.06.1.jar:1.16.2-HDP-24.06.1]
... 6 more
Caused by: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
at java.util.ArrayList.rangeCheck(ArrayList.java:659) ~[?:1.8.0_342]
at java.util.ArrayList.get(ArrayList.java:435) ~[?:1.8.0_342]
at 
org.apache.flink.hive.shaded.parquet.schema.GroupType.getType(GroupType.java:216)
 ~[flink-connector-hive_2.12-1.16.2-HDP-24.06.1.jar:1.16.2-HDP-24.06.1]
at 
org.apache.flink.hive.shaded.formats.parquet.vector.ParquetSplitReaderUtil.createWritableColumnVector(ParquetSplitReaderUtil.java:536)
 ~[flink-connector-hive_2.12-1.16.2-HDP-24.06.1.jar:1.16.2-HDP-24.06.1]
at 
org.apache.flink.hive.shaded.formats.parquet.vector.ParquetSplitReaderUtil.createWritableColumnVector(ParquetSplitReaderUtil.java:503)
 ~[flink-connector-hive_2.12-1.16.2-HDP-24.06.1.jar:1.16.2-HDP-24.06.1]
at 
org.apache.flink.hive.shaded.formats.parquet.ParquetVectorizedInputFormat.createWritableVectors(ParquetVectorizedInputFormat.java:277)
 ~[flink-connector-hive_2.12-1.16.2-HDP-24.06.1.jar:1.16.2-HDP-24.06.1]
at 
org.apache.flink.hive.shaded.formats.parquet.ParquetVectorizedInputFormat.createReaderBatch(ParquetVectorizedInputFormat.java:266)
 ~[flink-connector-hive_2.12-1.16.2-HDP-24.06.1.jar:1.16.2-HDP-24.06.1]
at 
org.apache.flink.hive.shaded.formats.parquet.ParquetVectorizedInputFormat.createPoolOfBatches(ParquetVectorizedInputFormat.java:256)
 ~[flink-connector-hive_2.12-1.16.2-HDP-24.06.1.jar:1.16.2-HDP-24.06.1]
at 
org.apache.flink.hive.shaded.formats.parquet.ParquetVectorizedInputFormat.createReader(ParquetVectorizedInputFormat.java:139)
 ~[flink-connector-hive_2.12-1.16.2-HDP-24.06.1.jar:1.16.2-HDP-24.06.1]
at 
org.apache.flink.hive.shaded.formats.parquet.ParquetVectorizedInputFormat.createReader(ParquetVectorizedInputFormat.java:75)
 ~[flink-connector-hive_2.12-1.16.2-HDP-24.06.1.jar:1.16.2-HDP-24.06.1]
at 
org.apache.flink.connectors.hive.read.HiveInputFormat.createReader(HiveInputFormat.java:110)
 ~[flink-connector-hive_2.12-1.16.2-HDP-24.06.1.jar:1.16.2-HDP-24.06.1]
at 
org.apache.flink.connectors.hive.read.HiveInputFormat.createReader(HiveInputFormat.java:65)
 ~[flink-connector-hive_2.12-1.16.2-HDP-24.06.1.jar:1.16.2-HDP-24.06.1]
at 
org.apache.flink.connector.file.src.impl.FileSourceSplitReader.checkSplitOrStartNext(FileSourceSplitReader.java:112)
 ~[flink-connector-files-1.16.2-HDP-24.06.1.jar:1.16.2-HDP-24.06.1]
at 
org.apache.flink.connector.file.src.impl.FileSourceSplitReader.fetch(FileSourceSplitReader.java:65)
 ~[flink-connector-files-1.16.2-HDP-24.06.1.jar:1.16.2-HDP-24.06.1]
at 
org.apache.flink.connector.file.src.impl.HdpFileSourceSplitReader.lambda$fetch$0(HdpFileSourceSplitReader.java:38)
 

Re: flink cdc 是否支持oracle rac

[Leonard Xu]

Hey，please send email to user...@apache.org  address 
to discuss user questions if you’d like to use Chinese, the d...@apache.org 
 is used to discuss development related tasks and 
English is required in this mailing list.

Best,
Leonard 

> 2024年6月3日 下午2:03，bbcca...@163.com 写道：
> 
> 查看debezium 1.9.8文档是支持oracle rac的，但flink cdc oracle 
> rac不管怎么配置都无法获取到源库数据，是我的配置不对吗？应该如何配置？
> debeziumProperties.setProperty("rac.nodes", 
> "172.16.140.254,172.16.140.253,172.16.140.252");
> debeziumProperties.setProperty("debezium.log.mining.strategy", 
> "online_catalog");
> debeziumProperties.setProperty("debezium.log.mining.continuous.mine", 
> "true");DebeziumSourceFunction sourceFunction = 
> OracleSource.builder()
>   .hostname("172.16.140.150")
>   .port(1521)
>   .database("testracdb")
>   .schemaList("kfz") // monitor inventory schema
>   .tableList("kfz.test_fb") // monitor products table
>   .username("sync")
>   .password("123456")
>   .startupOptions(StartupOptions.latest())
>   .debeziumProperties(debeziumProperties)
>   .deserializer(new TestFbDeserializationSchema()) 
>   .build();
> 
> 
> bbcca...@163.com