Best Practices? Fault Isolation for Processing Large Number of Same-Shaped Input Kafka Topics in a Big Flink Job

Hi everyone, I'm currently prototyping on a project where we need to process a large number of Kafka input topics (say, a couple of hundred), all of which share the same DataType/Schema. Our objective is to run the same Flink SQL on all of the input topics, but I am concerned about doing this in

Re: [RESULT][VOTE] FLIP-454: New Apicurio Avro format

Thanks David for driving the FLIP forward, but we need 3 +1(binding) votes according Flink Bylaws[1] before community accepted it. Best, Leonard [1] https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws > 2024年5月8日 下午11:05，David Radley 写道： > > Hi everyone, > I am happy to say that

Re: [RESULT][VOTE] FLIP-454: New Apicurio Avro format

Hi David, thanks for the update We should wait to reach threshold of "binding" votes as per the process[1] 1- https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws#FlinkBylaws-Approvals Best Regards Ahmed Hamdy On Wed, 8 May 2024 at 16:06, David Radley wrote: > Hi everyone, > I am

Re: [DISCUSS] Flink CDC 3.2 Release Planning

+1 for the proposal code freeze date and RM candidate. Best, Leonard > 2024年5月8日 下午10:27，gongzhongqiang 写道： > > Hi Qingsheng > > Thank you for driving the release. > Agree with the goal and I'm willing to help. > > Best, > Zhongqiang Gong > > Qingsheng Ren 于2024年5月8日周三 14:22写道： > >> Hi

[jira] [Created] (FLINK-35312) Insufficient number of arguments were supplied for the procedure or function cdc.fn_cdc_get_all_changes_

yux created FLINK-35312: --- Summary: Insufficient number of arguments were supplied for the procedure or function cdc.fn_cdc_get_all_changes_ Key: FLINK-35312 URL: https://issues.apache.org/jira/browse/FLINK-35312

Re: [VOTE] FLIP-452: Allow Skipping Invocation of Function Calls While Constant-folding

+1 (binding) Timo Walther 于2024年5月8日周三 17:15写道： > > +1 (binding) > > Thanks, > Timo > > On 08.05.24 11:10, Stefan Richter wrote: > > Hi Alan, > > > > Thanks for this proposal, the ability to exclude functions from constant > > folding makes sense to me. > > > > +1 (binding) > > > > Best, > >

[jira] [Created] (FLINK-35314) Add Flink CDC pipeline transform user document

Wenkai Qi created FLINK-35314: - Summary: Add Flink CDC pipeline transform user document Key: FLINK-35314 URL: https://issues.apache.org/jira/browse/FLINK-35314 Project: Flink Issue Type: New

[jira] [Created] (FLINK-35315) MemoryManagerConcurrentModReleaseTest executes more than 15 minutes

Rui Fan created FLINK-35315: --- Summary: MemoryManagerConcurrentModReleaseTest executes more than 15 minutes Key: FLINK-35315 URL: https://issues.apache.org/jira/browse/FLINK-35315 Project: Flink

Re: [VOTE] FLIP-447: Upgrade FRocksDB from 6.20.3 to 8.10.0

+1 (non-binding) ConradJam 于2024年5月7日周二 11:35写道： > +1 (no-binding) > > Piotr Nowojski 于2024年5月6日周一 20:17写道： > > > +1 (binding) > > > > Piotrek > > > > pon., 6 maj 2024 o 12:35 Roman Khachatryan > napisał(a): > > > > > +1 (binding) > > > > > > Regards, > > > Roman > > > > > > > > > On Mon, May

[RESULT][VOTE] FLIP-447: Upgrade FRocksDB from 6.20.3 to 8.10.0

Hi everyone, Thanks for your review and the votes! I am happy to announce that FLIP-447: Upgrade FRocksDB from 6.20.3 to 8.10.0 [1]. has been accepted. The proposal has been accepted with 15 approving votes (9 binding) and there is no disapproval: - Zakelly Lan (binding) - Yanfei Lei (binding)

[VOTE] FLIP-448: Introduce Pluggable Workflow Scheduler Interface for Materialized Table

Hi Dev, Thank you to everyone for the feedback on FLIP-448: Introduce Pluggable Workflow Scheduler Interface for Materialized Table[1][2]. I'd like to start a vote for it. The vote will be open for at least 72 hours unless there is an objection or not enough votes. [1]

flink-connector-kafka weekly CI job failing

Hi, I noticed the flink-connector-kafka weekly CI job is failing: https://github.com/apache/flink-connector-kafka/actions/runs/8954222477 Looks like flink-connector-kafka main has a compile error against flink 1.20-SNAPSHOT, I tried locally and get a different compile failure

[jira] [Created] (FLINK-35313) Add upsert changelog mode to avoid UPDATE_BEFORE records push down

ude created FLINK-35313: --- Summary: Add upsert changelog mode to avoid UPDATE_BEFORE records push down Key: FLINK-35313 URL: https://issues.apache.org/jira/browse/FLINK-35313 Project: Flink

Re: [VOTE] FLIP-448: Introduce Pluggable Workflow Scheduler Interface for Materialized Table

Sorry for the re-post, just to format this email content. Hi Dev Thank you to everyone for the feedback on FLIP-448: Introduce Pluggable Workflow Scheduler Interface for Materialized Table[1][2]. I'd like to start a vote for it. The vote will be open for at least 72 hours unless there is an

Re: [VOTE] Apache Flink CDC Release 3.1.0, release candidate #1

-1 Thanks Qingsheng for preparing this RC. If you bundle third-party dependencies (non Flink dependencies) in your published jar, you need to write them in the NOTICE file. I recommend adding a test using flink-ci-tools to verify if the NOTICE file is correct. Of course, you cannot rely too

[DISCUSS] Flink CDC 3.2 Release Planning

Hi devs, As we are in the midst of the release voting process for Flink CDC 3.1.0, I think it's a good time to kick off the upcoming Flink CDC 3.2 release cycle. In this release cycle I would like to focus on the stability of Flink CDC, especially for the newly introduced YAML-based data

Re: [VOTE] Apache Flink CDC Release 3.1.0, release candidate #1

Thanks everyone for your verification! I'll investigate the issue and prepare another RC once it is resolved. Much appreciated to Jingsong! This release candidate is now canceled. Best, Qingsheng On Wed, May 8, 2024 at 2:20 PM Jingsong Li wrote: > -1 > > Thanks Qingsheng for preparing this

[jira] [Created] (FLINK-35308) StarRocks sink maps TINYINT type to BOOLEAN incorrectly

yux created FLINK-35308: --- Summary: StarRocks sink maps TINYINT type to BOOLEAN incorrectly Key: FLINK-35308 URL: https://issues.apache.org/jira/browse/FLINK-35308 Project: Flink Issue Type: Bug

Re: [DISCUSS] FLIP-448: Introduce Pluggable Workflow Scheduler Interface for Materialized Table

Hi, Dev Thank you all for joining this thread and giving your comments and suggestions, they have helped improve this proposal and I look forward to further feedback. If there are no further comments, I'd like to close the discussion and start the voting one day later. Best, Ron Ron Liu

Re: [DISCUSS] Flink CDC 3.2 Release Planning

Thanks Qingsheng, Improving stability is crucial for Flink CDC, looking forward to this release. If assistance is needed, I am happy to help with it. Best, Jiabao Qingsheng Ren 于2024年5月8日周三 14:22写道： > Hi devs, > > As we are in the midst of the release voting process for Flink CDC 3.1.0, I >

Re: [DISCUSS] Flink CDC 3.2 Release Planning

Thanks Qinsheng for driving, +1 for the feature freeze date. I am happy to assist with any release duties. Best Regards Ahmed Hamdy On Wed, 8 May 2024 at 10:50, Jiabao Sun wrote: > Thanks Qingsheng, > > Improving stability is crucial for Flink CDC, looking forward to this > release. > If

Re: [VOTE] Apache Flink CDC Release 3.1.0, release candidate #1

Hi Qingsheng, I open a block issue[1] to add ci for NOTICE check and fix NOTICE file. Best, Zhongqiang Gong [1] https://issues.apache.org/jira/browse/FLINK-35309 Qingsheng Ren 于2024年5月8日周三 14:28写道： > Thanks everyone for your verification! > > I'll investigate the issue and prepare another RC

Re: [VOTE] FLIP-452: Allow Skipping Invocation of Function Calls While Constant-folding

+1 (binding) Thanks, Timo On 08.05.24 11:10, Stefan Richter wrote: Hi Alan, Thanks for this proposal, the ability to exclude functions from constant folding makes sense to me. +1 (binding) Best, Stefan On 8. May 2024, at 02:01, Alan Sheinberg wrote: Hi everyone, I'd like to start a

[jira] [Created] (FLINK-35309) Enable Notice file ci check and fix Notice

Zhongqiang Gong created FLINK-35309: --- Summary: Enable Notice file ci check and fix Notice Key: FLINK-35309 URL: https://issues.apache.org/jira/browse/FLINK-35309 Project: Flink Issue

Re: [VOTE] FLIP-452: Allow Skipping Invocation of Function Calls While Constant-folding

Hi Alan, Thanks for this proposal, the ability to exclude functions from constant folding makes sense to me. +1 (binding) Best, Stefan > On 8. May 2024, at 02:01, Alan Sheinberg > wrote: > > Hi everyone, > > I'd like to start a vote on FLIP-452 [1]. It covers adding a new method >

Re: [DISCUSS] Flink CDC 3.2 Release Planning

Hey Qingsheng, Thanks for your efforts, agreed! I would be happy to help. Best, Muhammet On 2024-05-08 06:21, Qingsheng Ren wrote: Hi devs, As we are in the midst of the release voting process for Flink CDC 3.1.0, I think it's a good time to kick off the upcoming Flink CDC 3.2 release

Re: [DISCUSS] FLIP-444: Native file copy support

Hi Piotr +1 for the proposal, it seems to have a lot of gains. Best Regards Ahmed Hamdy On Mon, 6 May 2024 at 12:06, Zakelly Lan wrote: > Hi Piotrek, > > Thanks for your answers! > > Good question. The intention and use case behind `DuplicatingFileSystem` is > > different. It marks if

Re: [DISCUSS] Flink CDC 3.2 Release Planning

Hi Qingsheng Thank you for driving the release. Agree with the goal and I'm willing to help. Best, Zhongqiang Gong Qingsheng Ren 于2024年5月8日周三 14:22写道： > Hi devs, > > As we are in the midst of the release voting process for Flink CDC 3.1.0, I > think it's a good time to kick off the upcoming

[jira] [Created] (FLINK-35310) Replace RBAC verb wildcards with actual verbs

Tim created FLINK-35310: --- Summary: Replace RBAC verb wildcards with actual verbs Key: FLINK-35310 URL: https://issues.apache.org/jira/browse/FLINK-35310 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-35311) FLIP-454: New Apicurio Avro format

david radley created FLINK-35311: Summary: FLIP-454: New Apicurio Avro format Key: FLINK-35311 URL: https://issues.apache.org/jira/browse/FLINK-35311 Project: Flink Issue Type: Improvement

[RESULT][VOTE] FLIP-454: New Apicurio Avro format

Hi everyone, I am happy to say that FLIP-454: New Apicurio Avro format [1] has been accepted and voted through this thread [2]. The proposal has been accepted with 4 approving votes and there are no vetos: - Ahmed Hamdy (non-binding) - Jeyhun Karimov (non-binding) - Mark Nuttall (non-binding) -