[Discuss] Non-retriable (partial) errors in Elasticsearch 8 connector

Hi all, Currently the Elasticsearch 8 connector retries all items if the request fails as a whole, and retries failed items if the request has partial failures [1]. I think this infinitely retries might be problematic in some cases when retrying can never eventually succeed. For example, if the

Re: [DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

Thanks Yuxin for your answer. +1 for this proposal. Best, Junrui. Yuxin Tan 于2024年6月6日周四 13:42写道： > Thanks Junrui for your question. > > > I wonder if the current interface design support the > future adaptation for batch job recovery > > I noticed that FLIP-383 supports batch job recovery by

Re: [DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

Thanks Junrui for your question. > I wonder if the current interface design support the future adaptation for batch job recovery I noticed that FLIP-383 supports batch job recovery by introducing some new APIs. These APIs can also be added to the Tier-related interfaces to facilitate the

[jira] [Created] (FLINK-35536) FileSystem sink on S3 produces invalid Avros when compaction is disabled

Juliusz Nadberezny created FLINK-35536: -- Summary: FileSystem sink on S3 produces invalid Avros when compaction is disabled Key: FLINK-35536 URL: https://issues.apache.org/jira/browse/FLINK-35536

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

Congrats, Rui! Best, Paul Lam > 2024年6月6日 11:02，Junrui Lee 写道： > > Congratulations, Rui. > > Best, > Junrui > > Hang Ruan 于2024年6月6日周四 10:35写道： > >> Congratulations, Rui! >> >> Best, >> Hang >> >> Samrat Deb 于2024年6月6日周四 10:28写道： >> >>> Congratulations Rui >>> >>> Bests, >>> Samrat

[jira] [Created] (FLINK-35535) Enable benchmark profiling in daily run

Zakelly Lan created FLINK-35535: --- Summary: Enable benchmark profiling in daily run Key: FLINK-35535 URL: https://issues.apache.org/jira/browse/FLINK-35535 Project: Flink Issue Type:

[jira] [Created] (FLINK-35534) Support profiler for benchmarks

Zakelly Lan created FLINK-35534: --- Summary: Support profiler for benchmarks Key: FLINK-35534 URL: https://issues.apache.org/jira/browse/FLINK-35534 Project: Flink Issue Type: Improvement

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

Congratulations, Rui. Best, Junrui Hang Ruan 于2024年6月6日周四 10:35写道： > Congratulations, Rui! > > Best, > Hang > > Samrat Deb 于2024年6月6日周四 10:28写道： > > > Congratulations Rui > > > > Bests, > > Samrat > > > > On Thu, 6 Jun 2024 at 7:45 AM, Yuxin Tan wrote: > > > > > Congratulations, Rui! > > > >

Re: [DISCUSS] FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

Thanks Yuxin for driving this proposal! I have a question about the public interface compatibility in the context of FLIP-459. As we've supported batch job recovery from jobMaster failures in FLIP-383 which will be released in Flink 1.20. I wonder if the current interface design support the

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

Congratulations ! Best, Yuepeng Pan On 2024/06/05 10:01:58 Piotr Nowojski wrote: > Hi everyone, > > On behalf of the PMC, I'm very happy to announce another new Apache Flink > PMC Member - Fan Rui. > > Rui has been active in the community since August 2019. During this time he > has

Re: Poor Load Balancing across TaskManagers for Multiple Kafka Sources

Hi Kevin, The problem here is about how to evenly distribute partitions from multiple Kafka topics to tasks, while FLIP-370 is only concerned about how to evenly distribute tasks to slots & taskmanagers, so FLIP-370 won't help here. Best, Zhanghao Chen From:

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

Congratulations, Rui! Best, Hang Samrat Deb 于2024年6月6日周四 10:28写道： > Congratulations Rui > > Bests, > Samrat > > On Thu, 6 Jun 2024 at 7:45 AM, Yuxin Tan wrote: > > > Congratulations, Rui! > > > > Best, > > Yuxin > > > > > > Xuannan Su 于2024年6月6日周四 09:58写道： > > > > > Congratulations! > > > >

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

Congratulations Rui Bests, Samrat On Thu, 6 Jun 2024 at 7:45 AM, Yuxin Tan wrote: > Congratulations, Rui! > > Best, > Yuxin > > > Xuannan Su 于2024年6月6日周四 09:58写道： > > > Congratulations! > > > > Best regards, > > Xuannan > > > > On Thu, Jun 6, 2024 at 9:53 AM Hangxiang Yu wrote: > > > > > >

Re: [ANNOUNCE] New Apache Flink PMC Member - Weijie Guo

Congratulations Bests, Samrat On Thu, 6 Jun 2024 at 7:28 AM, Xuannan Su wrote: > Congratulations! > > Best regards, > Xuannan > > On Thu, Jun 6, 2024 at 9:20 AM Jiabao Sun wrote: > > > > Congratulations, Weijie! > > > > Best, > > Jiabao > > > > Lincoln Lee 于2024年6月6日周四 09:17写道： > > > > >

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

Congratulations, Rui! Best, Yuxin Xuannan Su 于2024年6月6日周四 09:58写道： > Congratulations! > > Best regards, > Xuannan > > On Thu, Jun 6, 2024 at 9:53 AM Hangxiang Yu wrote: > > > > Congratulations, Rui ! > > > > On Thu, Jun 6, 2024 at 9:18 AM Lincoln Lee > wrote: > > > > > Congratulations, Rui!

[jira] [Created] (FLINK-35533) FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn

Yuxin Tan created FLINK-35533: - Summary: FLIP-459: Support Flink hybrid shuffle integration with Apache Celeborn Key: FLINK-35533 URL: https://issues.apache.org/jira/browse/FLINK-35533 Project: Flink

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

Congratulations! Best regards, Xuannan On Thu, Jun 6, 2024 at 9:53 AM Hangxiang Yu wrote: > > Congratulations, Rui ! > > On Thu, Jun 6, 2024 at 9:18 AM Lincoln Lee wrote: > > > Congratulations, Rui! > > > > Best, > > Lincoln Lee > > > > > > Lijie Wang 于2024年6月6日周四 09:11写道： > > > > >

Re: [ANNOUNCE] New Apache Flink PMC Member - Weijie Guo

Congratulations! Best regards, Xuannan On Thu, Jun 6, 2024 at 9:20 AM Jiabao Sun wrote: > > Congratulations, Weijie! > > Best, > Jiabao > > Lincoln Lee 于2024年6月6日周四 09:17写道： > > > Congratulations, Weijie! > > > > Best, > > Lincoln Lee > > > > > > ConradJam 于2024年6月5日周三 10:46写道： > > > > >

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

Congratulations, Rui ! On Thu, Jun 6, 2024 at 9:18 AM Lincoln Lee wrote: > Congratulations, Rui! > > Best, > Lincoln Lee > > > Lijie Wang 于2024年6月6日周四 09:11写道： > > > Congratulations, Rui! > > > > Best, > > Lijie > > > > Rodrigo Meneses 于2024年6月5日周三 21:35写道： > > > > > All the best > > > > > >

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

Congratulations, Rui! Best, Wei > 2024年6月6日 09:16，Lincoln Lee 写道： > > Congratulations, Rui! > > Best, > Lincoln Lee > > > Lijie Wang 于2024年6月6日周四 09:11写道： > >> Congratulations, Rui! >> >> Best, >> Lijie >> >> Rodrigo Meneses 于2024年6月5日周三 21:35写道： >> >>> All the best >>> >>> On Wed,

Re: [ANNOUNCE] New Apache Flink PMC Member - Weijie Guo

Congratulations, Weijie! Best, Jiabao Lincoln Lee 于2024年6月6日周四 09:17写道： > Congratulations, Weijie! > > Best, > Lincoln Lee > > > ConradJam 于2024年6月5日周三 10:46写道： > > > Congratulations！ > > > > Biao Geng 于2024年6月5日周三 10:44写道： > > > > > Congratulations, Weijie! > > > Best, > > > Biao Geng > > >

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

Congratulations, Rui! Best, Lincoln Lee Lijie Wang 于2024年6月6日周四 09:11写道： > Congratulations, Rui! > > Best, > Lijie > > Rodrigo Meneses 于2024年6月5日周三 21:35写道： > > > All the best > > > > On Wed, Jun 5, 2024 at 5:56 AM xiangyu feng > wrote: > > > > > Congratulations, Rui! > > > > > > Regards, >

Re: [ANNOUNCE] New Apache Flink PMC Member - Weijie Guo

Congratulations, Weijie! Best, Lincoln Lee ConradJam 于2024年6月5日周三 10:46写道： > Congratulations！ > > Biao Geng 于2024年6月5日周三 10:44写道： > > > Congratulations, Weijie! > > Best, > > Biao Geng > > > > > > Yun Tang 于2024年6月5日周三 10:42写道： > > > > > Congratulations, Weijie! > > > > > > Best > > > Yun

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

Congratulations, Rui! Best, Lijie Rodrigo Meneses 于2024年6月5日周三 21:35写道： > All the best > > On Wed, Jun 5, 2024 at 5:56 AM xiangyu feng wrote: > > > Congratulations, Rui! > > > > Regards, > > Xiangyu Feng > > > > Feng Jin 于2024年6月5日周三 20:42写道： > > > > > Congratulations, Rui! > > > > > > > > >

Re: Poor Load Balancing across TaskManagers for Multiple Kafka Sources

I've also just found https://issues.apache.org/jira/browse/FLINK-31762 which tracks the Kafka specific issue. On Wed, Jun 5, 2024 at 3:05 PM Kevin Lam wrote: > cc. panyuep...@apache.org as related to FLIP-370 > > On Wed, Jun 5, 2024 at 2:32 PM Kevin Lam wrote: > >> Hey all, >> >> I'm seeing an

Re: Poor Load Balancing across TaskManagers for Multiple Kafka Sources

cc. panyuep...@apache.org as related to FLIP-370 On Wed, Jun 5, 2024 at 2:32 PM Kevin Lam wrote: > Hey all, > > I'm seeing an issue with poor load balancing across TaskManagers for Kafka > Sources using the Flink SQL API and wondering if FLIP-370 will help with > it, or if not, interested in

Poor Load Balancing across TaskManagers for Multiple Kafka Sources

Hey all, I'm seeing an issue with poor load balancing across TaskManagers for Kafka Sources using the Flink SQL API and wondering if FLIP-370 will help with it, or if not, interested in any ideas the community has to mitigate the issue. The Kafka SplitEnumerator uses the following logic to

[jira] [Created] (FLINK-35532) Prevent Cross-Site Authentication (XSA) attacks on Flink dashboard

Hong Liang Teoh created FLINK-35532: --- Summary: Prevent Cross-Site Authentication (XSA) attacks on Flink dashboard Key: FLINK-35532 URL: https://issues.apache.org/jira/browse/FLINK-35532 Project:

Re: [DISCUSS] FLIP-461: FLIP-461: Synchronize rescaling with checkpoint creation to minimize reprocessing

Hi Zakelly, thanks for your reply. See my inlined responses below: On Wed, Jun 5, 2024 at 10:26 AM Zakelly Lan wrote: > Hi Matthias, > > Thanks for your proposal! I have a few questions: > > 1. Is it possible a change event observed right after a complete checkpoint > (or within a specific

Re: [DISCUSS] FLIP-461: FLIP-461: Synchronize rescaling with checkpoint creation to minimize reprocessing

Thanks Rui for your reply. Find my answers inlined below: [1] https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#jobmanager-adaptive-scheduler-resource-stabilization-timeout On Wed, Jun 5, 2024 at 10:16 AM Rui Fan <1996fan...@gmail.com> wrote: > Thanks Matthias for

[jira] [Created] (FLINK-35531) Avoid calling hsync in flush method in BaseHadoopFsRecoverableFsDataOutputStream

dzcxzl created FLINK-35531: -- Summary: Avoid calling hsync in flush method in BaseHadoopFsRecoverableFsDataOutputStream Key: FLINK-35531 URL: https://issues.apache.org/jira/browse/FLINK-35531 Project: Flink

[jira] [Created] (FLINK-35530) protobuf-format support discard unknow field Improve deserialization performance

JingWei Li created FLINK-35530: -- Summary: protobuf-format support discard unknow field Improve deserialization performance Key: FLINK-35530 URL: https://issues.apache.org/jira/browse/FLINK-35530

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

All the best On Wed, Jun 5, 2024 at 5:56 AM xiangyu feng wrote: > Congratulations, Rui! > > Regards, > Xiangyu Feng > > Feng Jin 于2024年6月5日周三 20:42写道： > > > Congratulations, Rui! > > > > > > Best, > > Feng Jin > > > > On Wed, Jun 5, 2024 at 8:23 PM Yanfei Lei wrote: > > > > > Congratulations,

[jira] [Created] (FLINK-35529) protobuf-format compatible protobuf bad indentifier

JingWei Li created FLINK-35529: -- Summary: protobuf-format compatible protobuf bad indentifier Key: FLINK-35529 URL: https://issues.apache.org/jira/browse/FLINK-35529 Project: Flink Issue Type:

[jira] [Created] (FLINK-35528) Skip execution of interruptible mails when yielding

Piotr Nowojski created FLINK-35528: -- Summary: Skip execution of interruptible mails when yielding Key: FLINK-35528 URL: https://issues.apache.org/jira/browse/FLINK-35528 Project: Flink

[jira] [Created] (FLINK-35527) Polish quickstart guide & clean stale links in docs

yux created FLINK-35527: --- Summary: Polish quickstart guide & clean stale links in docs Key: FLINK-35527 URL: https://issues.apache.org/jira/browse/FLINK-35527 Project: Flink Issue Type: Bug

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

Congratulations, Rui! Regards, Xiangyu Feng Feng Jin 于2024年6月5日周三 20:42写道： > Congratulations, Rui! > > > Best, > Feng Jin > > On Wed, Jun 5, 2024 at 8:23 PM Yanfei Lei wrote: > > > Congratulations, Rui! > > > > Best, > > Yanfei > > > > Luke Chen 于2024年6月5日周三 20:08写道： > > > > > > Congrats,

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

Congratulations, Rui! Best, Feng Jin On Wed, Jun 5, 2024 at 8:23 PM Yanfei Lei wrote: > Congratulations, Rui! > > Best, > Yanfei > > Luke Chen 于2024年6月5日周三 20:08写道： > > > > Congrats, Rui! > > > > Luke > > > > On Wed, Jun 5, 2024 at 8:02 PM Jiabao Sun wrote: > > > > > Congrats, Rui.

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

Congratulations, Rui! Best, Yanfei Luke Chen 于2024年6月5日周三 20:08写道： > > Congrats, Rui! > > Luke > > On Wed, Jun 5, 2024 at 8:02 PM Jiabao Sun wrote: > > > Congrats, Rui. Well-deserved! > > > > Best, > > Jiabao > > > > Zhanghao Chen 于2024年6月5日周三 19:29写道： > > > > > Congrats, Rui! > > > > > >

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

Congrats, Rui! Luke On Wed, Jun 5, 2024 at 8:02 PM Jiabao Sun wrote: > Congrats, Rui. Well-deserved! > > Best, > Jiabao > > Zhanghao Chen 于2024年6月5日周三 19:29写道： > > > Congrats, Rui! > > > > Best, > > Zhanghao Chen > > > > From: Piotr Nowojski > > Sent:

[jira] [Created] (FLINK-35526) Remove deprecated stedolan/jq Docker image from Flink e2e tests

Robert Metzger created FLINK-35526: -- Summary: Remove deprecated stedolan/jq Docker image from Flink e2e tests Key: FLINK-35526 URL: https://issues.apache.org/jira/browse/FLINK-35526 Project: Flink

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

Congrats, Rui. Well-deserved! Best, Jiabao Zhanghao Chen 于2024年6月5日周三 19:29写道： > Congrats, Rui! > > Best, > Zhanghao Chen > > From: Piotr Nowojski > Sent: Wednesday, June 5, 2024 18:01 > To: dev ; rui fan <1996fan...@gmail.com> > Subject: [ANNOUNCE] New Apache

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

Congrats, Rui! Best, Zhanghao Chen From: Piotr Nowojski Sent: Wednesday, June 5, 2024 18:01 To: dev ; rui fan <1996fan...@gmail.com> Subject: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui Hi everyone, On behalf of the PMC, I'm very happy to announce another

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

Congratulations, Rui! Best, Yun Tang From: weijie guo Sent: Wednesday, June 5, 2024 19:12 To: dev@flink.apache.org Subject: Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui Congratulations, Rui. Well-deserved! Best regards, Weijie Zakelly Lan

Re: Flink stateful functions and Agentic Architecture

Hi David, Since flink stateful has been inactive for some time[1], I'm not sure whether you could get some help here. From my understanding, your request to a distribute system is event driven processing + actor model. Perhaps you could enrich Ray[2] with continuous message processing to

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

Congratulations, Rui. Well-deserved! Best regards, Weijie Zakelly Lan 于2024年6月5日周三 18:05写道： > Congratulations, Rui! > > Best, > Zakelly > > On Wed, Jun 5, 2024 at 6:02 PM Piotr Nowojski > wrote: > > > Hi everyone, > > > > On behalf of the PMC, I'm very happy to announce another new Apache

Re: [ANNOUNCE] New Apache Flink PMC Member - Fan Rui

Congratulations, Rui! Best, Zakelly On Wed, Jun 5, 2024 at 6:02 PM Piotr Nowojski wrote: > Hi everyone, > > On behalf of the PMC, I'm very happy to announce another new Apache Flink > PMC Member - Fan Rui. > > Rui has been active in the community since August 2019. During this time he > has

[ANNOUNCE] New Apache Flink PMC Member - Fan Rui

Hi everyone, On behalf of the PMC, I'm very happy to announce another new Apache Flink PMC Member - Fan Rui. Rui has been active in the community since August 2019. During this time he has contributed a lot of new features. Among others: - Decoupling Autoscaler from Kubernetes Operator, and

Re: [DISCUSS] FLIP-461: FLIP-461: Synchronize rescaling with checkpoint creation to minimize reprocessing

Hi Matthias, Thanks for your proposal! I have a few questions: 1. Is it possible a change event observed right after a complete checkpoint (or within a specific short time after a checkpoint) that triggers a rescale immediately? Sometimes the checkpoint interval is huge and it is better to

Re: [DISCUSS] FLIP-461: FLIP-461: Synchronize rescaling with checkpoint creation to minimize reprocessing

Thanks Matthias for driving this proposal! This proposal can reduce the amount of data that is processed repeatedly after rescaling, so this proposal makes sense to me. I have some questions: 1. The public change only includes the "New Configuration Parameters" part, right? 2.

[jira] [Created] (FLINK-35525) HDFS delegation token fetched by custom DelegationTokenProvider is not passed to Yarn AM

Zhen Wang created FLINK-35525: - Summary: HDFS delegation token fetched by custom DelegationTokenProvider is not passed to Yarn AM Key: FLINK-35525 URL: https://issues.apache.org/jira/browse/FLINK-35525

[jira] [Created] (FLINK-35524) Clear connections pools when reader exist.

Hongshun Wang created FLINK-35524: - Summary: Clear connections pools when reader exist. Key: FLINK-35524 URL: https://issues.apache.org/jira/browse/FLINK-35524 Project: Flink Issue Type: Bug

Re: [DISCUSS] FLIP-461: FLIP-461: Synchronize rescaling with checkpoint creation to minimize reprocessing

Hi ConradJam, thanks for your response. The CheckpointStatsTracker gets notified about the checkpoint completion after the checkpoint is finalized, i.e. all its data is persisted and the metadata is written to the CompletedCheckpointStore. At this moment, the checkpoint is considered for

[jira] [Created] (FLINK-35523) When using the Hive connector to read a Hive table in Parquet format, a null pointer exception is thrown.

Jichao Wang created FLINK-35523: --- Summary: When using the Hive connector to read a Hive table in Parquet format, a null pointer exception is thrown. Key: FLINK-35523 URL:

Re: flink cdc 是否支持oracle rac

Hey，please send email to user...@apache.org address to discuss user questions if you’d like to use Chinese, the d...@apache.org is used to discuss development related tasks and English is required in this mailing list. Best, Leonard >