[jira] [Created] (FLINK-34556) Migrate EnumerableToLogicalTableScan

2024-02-29 Thread Jacky Lau (Jira)
Jacky Lau created FLINK-34556: - Summary: Migrate EnumerableToLogicalTableScan Key: FLINK-34556 URL: https://issues.apache.org/jira/browse/FLINK-34556 Project: Flink Issue Type: Sub-task

Re: [DISCUSS] FLIP-423 ~FLIP-428: Introduce Disaggregated State Storage and Management in Flink 2.0

2024-02-29 Thread Zakelly Lan
Hi Piotr, Thanks for your thoughtful feedback! In fact, we had a lot of internal debates on this topic :). The proposal you mentioned is one of several possible alternatives we've considered. It works well for simple use cases of ValueState or ListState, where users typically read state by

[jira] [Created] (FLINK-34555) Migrate JoinConditionTypeCoerceRule

2024-02-29 Thread Jacky Lau (Jira)
Jacky Lau created FLINK-34555: - Summary: Migrate JoinConditionTypeCoerceRule Key: FLINK-34555 URL: https://issues.apache.org/jira/browse/FLINK-34555 Project: Flink Issue Type: Sub-task

[jira] [Created] (FLINK-34554) Using EXACTLY_ONCE with KafkaSink cause broker's OOM due to newly created transactionalId per checkpoint

2024-02-29 Thread Hilmi Al Fatih (Jira)
Hilmi Al Fatih created FLINK-34554: -- Summary: Using EXACTLY_ONCE with KafkaSink cause broker's OOM due to newly created transactionalId per checkpoint Key: FLINK-34554 URL:

[jira] [Created] (FLINK-34553) Time travel support by Flink catalogs

2024-02-29 Thread Mehmet Aktas (Jira)
Mehmet Aktas created FLINK-34553: Summary: Time travel support by Flink catalogs Key: FLINK-34553 URL: https://issues.apache.org/jira/browse/FLINK-34553 Project: Flink Issue Type: Technical

Re: [DISCUSS] FLINK-34440 Support Debezium Protobuf Confluent Format

2024-02-29 Thread Kevin Lam
Hey Robert, Awesome thanks, that timeline works for me. Sounds good re: deciding on FLIP once we have the PR, and thanks for looking into the field ids. Looking forward to it! On Thu, Feb 29, 2024 at 5:09 AM Robert Metzger wrote: > Hey Kevin, > > Thanks a lot. Then let's contribute the

[jira] [Created] (FLINK-34552) Support message deduplication for input data sources

2024-02-29 Thread Sergey Anokhovskiy (Jira)
Sergey Anokhovskiy created FLINK-34552: -- Summary: Support message deduplication for input data sources Key: FLINK-34552 URL: https://issues.apache.org/jira/browse/FLINK-34552 Project: Flink

[jira] [Created] (FLINK-34551) Align retry mechanisms of FutureUtils

2024-02-29 Thread Matthias Pohl (Jira)
Matthias Pohl created FLINK-34551: - Summary: Align retry mechanisms of FutureUtils Key: FLINK-34551 URL: https://issues.apache.org/jira/browse/FLINK-34551 Project: Flink Issue Type:

Re: [DISCUSS] FLIP-423 ~FLIP-428: Introduce Disaggregated State Storage and Management in Flink 2.0

2024-02-29 Thread Piotr Nowojski
Hi! Thanks for this proposal. It looks like it will be a great improvement! I've only started reading the FLIP's, but I already have some questions about the FLIP-425, the async execution. What's the motivation behind splitting execution of a single element into multiple independent steps in

[ANNOUNCE] Apache flink-connector-parent 1.1.0 released

2024-02-29 Thread Etienne Chauchot
The Apache Flink community is very happy to announce the release of Apache flink-connector-parent 1.1.0. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications. The release is available for

[jira] [Created] (FLINK-34550) attempted task still report metric of currentEmitEventTimeLag

2024-02-29 Thread siwei.gao (Jira)
siwei.gao created FLINK-34550: - Summary: attempted task still report metric of currentEmitEventTimeLag Key: FLINK-34550 URL: https://issues.apache.org/jira/browse/FLINK-34550 Project: Flink

[jira] [Created] (FLINK-34549) FLIP-410: Config, Context and Processing Timer Service of DataStream API V2

2024-02-29 Thread Weijie Guo (Jira)
Weijie Guo created FLINK-34549: -- Summary: FLIP-410: Config, Context and Processing Timer Service of DataStream API V2 Key: FLINK-34549 URL: https://issues.apache.org/jira/browse/FLINK-34549 Project:

[jira] [Created] (FLINK-34548) FLIP-409: DataStream V2 Building Blocks: DataStream, Partitioning and ProcessFunction

2024-02-29 Thread Weijie Guo (Jira)
Weijie Guo created FLINK-34548: -- Summary: FLIP-409: DataStream V2 Building Blocks: DataStream, Partitioning and ProcessFunction Key: FLINK-34548 URL: https://issues.apache.org/jira/browse/FLINK-34548

[jira] [Created] (FLINK-34547) Flink DataStream API V2

2024-02-29 Thread Weijie Guo (Jira)
Weijie Guo created FLINK-34547: -- Summary: Flink DataStream API V2 Key: FLINK-34547 URL: https://issues.apache.org/jira/browse/FLINK-34547 Project: Flink Issue Type: New Feature

[RESULT][VOTE] FLIP-410: Config, Context and Processing Timer Service of DataStream API V2

2024-02-29 Thread weijie guo
Hi devs, I'm happy to announce that FLIP-410: Config, Context and Processing Timer Service of DataStream API V2 [1] has been accepted with 5 approving votes (4 binding) [2]: - Xintong Song (binding) - Weijie Guo (binding) - Rui Fan (binding) - Xuannan Su (non-binding) - Guowei Ma (binding)

[RESULT][VOTE] FLIP-409: DataStream V2 Building Blocks: DataStream, Partitioning and ProcessFunction

2024-02-29 Thread weijie guo
Hi devs, I'm happy to announce that FLIP-409: DataStream V2 Building Blocks: DataStream, Partitioning and ProcessFunction [1] has been accepted with 5 approving votes (4 binding) [2]: - Xintong Song (binding) - Weijie Guo (binding) - Rui Fan (binding) - Guowei Ma (binding) - Xuannan Su

[RESULT][VOTE] FLIP-408: [Umbrella] Introduce DataStream API V2

2024-02-29 Thread weijie guo
Hi devs, I'm happy to announce that FLIP-408: [Umbrella] Introduce DataStream API V2 [1] has been accepted with 5 approving votes (4 binding) [2]: - Xintong Song (binding) - Weijie Guo (binding) - Xuannan Su (non-binding) - Rui Fan (binding) - Guowei Ma (binding) There are no

Re: Question around Flink's AdaptiveBatchScheduler

2024-02-29 Thread Junrui Lee
Hi Venkat, As Lijie mentioned, in Flink, the parallelism is required to be less than or equal to the maximum parallelism. The config option jobmanager.adaptive-batch-scheduler.max-parallelism and jobmanager.adaptive-batch-scheduler.default-source-parallelism will be set as the source's

Re: [DISCUSS] FLINK-34440 Support Debezium Protobuf Confluent Format

2024-02-29 Thread Robert Metzger
Hey Kevin, Thanks a lot. Then let's contribute the Confluent implementation to apache/flink. We can't start working on this immediately because of a team event next week, but within the next two weeks, we will start working on this. It probably makes sense for us to open a pull request of what we

Re: Question around Flink's AdaptiveBatchScheduler

2024-02-29 Thread Lijie Wang
Hi Venkat, >> default-source-parallelism config should be independent from the max-parallelism Actually, it's not. Firstly, it's obvious that the parallelism should be less than or equal to the max parallelism(both literally and execution). The

[jira] [Created] (FLINK-34546) Emit span with failure labels on failure

2024-02-29 Thread Stefan Richter (Jira)
Stefan Richter created FLINK-34546: -- Summary: Emit span with failure labels on failure Key: FLINK-34546 URL: https://issues.apache.org/jira/browse/FLINK-34546 Project: Flink Issue Type:

Re: Flink kinesis connector v4.2.0-1.18 has issue when consuming from EFO consumer

2024-02-29 Thread Xiaolong Wang
Thanks, this looks similar. I'll do some work around. On Thu, Feb 29, 2024 at 5:15 PM Aleksandr Pilipenko wrote: > Based on the stacktrace, this looks like an issue described here: > https://issues.apache.org/jira/browse/FLINK-32964 > Is your configuration similar to the one described in the

Re: Flink kinesis connector v4.2.0-1.18 has issue when consuming from EFO consumer

2024-02-29 Thread Aleksandr Pilipenko
Based on the stacktrace, this looks like an issue described here: https://issues.apache.org/jira/browse/FLINK-32964 Is your configuration similar to the one described in the ticket? If so, you can work around this issue by explicitly specifying the credentials provider for connector, by doing so

Re: [DISCUSS] Apache Bahir retired

2024-02-29 Thread Ferenc Csaky
Thank you Marton and Martijn, I will proceed with the FLIP then. Best, Ferenc On Wednesday, February 28th, 2024 at 21:15, Martijn Visser wrote: > > > Hi all, > > +1 to have a connector FLIP to propose a Kudu connector. I'm +0 overall > because I don't see a lot of activity happening in

[jira] [Created] (FLINK-34545) Add OceanBase pipeline connector to Flink CDC

2024-02-29 Thread He Wang (Jira)
He Wang created FLINK-34545: --- Summary: Add OceanBase pipeline connector to Flink CDC Key: FLINK-34545 URL: https://issues.apache.org/jira/browse/FLINK-34545 Project: Flink Issue Type: New Feature