[jira] [Created] (FLINK-35387) PG CDC source support heart beat

2024-05-16 Thread Hongshun Wang (Jira)
Hongshun Wang created FLINK-35387: - Summary: PG CDC source support heart beat Key: FLINK-35387 URL: https://issues.apache.org/jira/browse/FLINK-35387 Project: Flink Issue Type: Bug

Re: [DISCUSS] Merge "flink run" and "flink run-application" in Flink 2.0

2024-05-16 Thread Xintong Song
AFAIK, the main purpose of having `run-application` was to make sure the user is aware that application mode is used, which executes the main method of the user program in JM rather than in client. This was important at the time application mode was first introduced, but maybe not that important an

[jira] [Created] (FLINK-35386) Build Flink CDC 3.1 Documentation and mark it as stable

2024-05-16 Thread Qingsheng Ren (Jira)
Qingsheng Ren created FLINK-35386: - Summary: Build Flink CDC 3.1 Documentation and mark it as stable Key: FLINK-35386 URL: https://issues.apache.org/jira/browse/FLINK-35386 Project: Flink Iss

Re: [DISCUSSION] FLIP-457: Improve Table/SQL Configuration for Flink 2.0

2024-05-16 Thread Yubin Li
Hi Jane, Thank Jane for driving this proposal ! This makes sense for users, +1 for that. Best, Yubin On Thu, May 16, 2024 at 3:17 PM Jark Wu wrote: > > Hi Jane, > > Thanks for the proposal. +1 from my side. > > > Best, > Jark > > On Thu, 16 May 2024 at 10:28, Xuannan Su wrote: > > > Hi Jane,

[jira] [Created] (FLINK-35385) upgrader flink dependency version to 1.19

2024-05-16 Thread Hongshun Wang (Jira)
Hongshun Wang created FLINK-35385: - Summary: upgrader flink dependency version to 1.19 Key: FLINK-35385 URL: https://issues.apache.org/jira/browse/FLINK-35385 Project: Flink Issue Type: Impro

Re:Re: Confirmation on Lambda Support for UDFs in FlinkSQL / Table API

2024-05-16 Thread Xuyang
Hi, Tucker IIUC, this way is unsupported. You may try other way to work around it. -- Best! Xuyang At 2024-05-17 00:22:57, "Tucker Harvey" wrote: >Hi Xuyang, > >Thanks for your response. I can provide some better context, we are migrating >a use case to FlinkSQL and they wa

Re: [VOTE] FLIP-449: Reorganization of flink-connector-jdbc

2024-05-16 Thread Jiabao Sun
Thanks for driving this proposal! +1 (binding) Best, Jiabao On 2024/05/10 22:18:04 Jeyhun Karimov wrote: > Thanks for driving this! > > +1 (non-binding) > > Regards, > Jeyhun > > On Fri, May 10, 2024 at 12:50 PM Muhammet Orazov > wrote: > > > Thanks João for your efforts and driving this!

[RESULT][VOTE] Apache Flink CDC Release 3.1.0, release candidate #3

2024-05-16 Thread Qingsheng Ren
Hi everyone, I'm happy to announce that we have unanimously approved this release. There are 7 approving votes, 3 of which are binding: - Hang Ruan (non-binding) - Yanquan Lv (non-binding) - Leonard Xu (binding) - Jiabao Sun (non-binding) - Zhongqiang Gong (non-binding) - Jark Wu (binding) - Jin

Re: [DISCUSS] FLINK-35369: Improve `Table API and SQL` page or add new page to guide new users to right Flink SQL option

2024-05-16 Thread Martijn Visser
Hi Keith Lee, Yeah, I think it makes sense that large changes to documentation should be FLIP worthy. We've done it before when we reorganized the project website as well. >From a strict point of view, we can't retrospectively change an accepted FLIP: we can create a new FLIP that refers to the o

Re: [DISCUSS] FLINK-35369: Improve `Table API and SQL` page or add new page to guide new users to right Flink SQL option

2024-05-16 Thread Keith Lee
Hi Martijn Thank you for pointing me to this FLIP. I initially thought of making limited scoped improvement hence the Jira. Given that the Flip was voted on more than four years ago. Would it make sense to for me to update the FLIP, reopen discussion and redrive a vote for it? On a side note, sh

Re: [DISCUSS] Connector releases for Flink 1.19

2024-05-16 Thread Sergey Nuyanzin
>, it looks like opensearch-2.0.0 has been created now, all good. yep, thanks to Martijn I've created RCs for Opensearch connector On Tue, May 14, 2024 at 12:38 PM Danny Cranmer wrote: > Hello, > > @Sergey Nuyanzin , it looks like opensearch-2.0.0 > has been created now, all good. > > @Hongshun

[Discuss] expose TaskIOMetricGroup to custom Partitioner

2024-05-16 Thread Steven Wu
Hi, I am trying to implement a custom range partitioner in the Flink Iceberg sink. Want to publish some counter metrics for certain scenarios. This is like the network metrics exposed in `TaskIOMetricGroup`. This requires adding a new setup method to the custom `Partitioner` interface. Like to ge

[jira] [Created] (FLINK-35384) Expose metrics group to custom Partitioner

2024-05-16 Thread Steven Zhen Wu (Jira)
Steven Zhen Wu created FLINK-35384: -- Summary: Expose metrics group to custom Partitioner Key: FLINK-35384 URL: https://issues.apache.org/jira/browse/FLINK-35384 Project: Flink Issue Type: Im

[jira] [Created] (FLINK-35383) Update compatibility matrix to include 1.19 release

2024-05-16 Thread Aleksandr Pilipenko (Jira)
Aleksandr Pilipenko created FLINK-35383: --- Summary: Update compatibility matrix to include 1.19 release Key: FLINK-35383 URL: https://issues.apache.org/jira/browse/FLINK-35383 Project: Flink

Re: Confirmation on Lambda Support for UDFs in FlinkSQL / Table API

2024-05-16 Thread Tucker Harvey
Hi Xuyang, Thanks for your response. I can provide some better context, we are migrating a use case to FlinkSQL and they want to do some query that has a UDF like SELECT user_id, ARRAY_MAP(transactions, x -> x * 1.1) AS taxed_transactions FROM users; Based on my understanding of th

[jira] [Created] (FLINK-35382) ChangelogCompatibilityITCase.testRestore fails with an NPE

2024-05-16 Thread Ryan Skraba (Jira)
Ryan Skraba created FLINK-35382: --- Summary: ChangelogCompatibilityITCase.testRestore fails with an NPE Key: FLINK-35382 URL: https://issues.apache.org/jira/browse/FLINK-35382 Project: Flink Issu

[DISCUSS] Merge "flink run" and "flink run-application" in Flink 2.0

2024-05-16 Thread Ferenc Csaky
Hello devs, I saw quite some examples when customers were confused about run​, and run- application​ in the Flink CLI and I was wondering about the necessity of deploying Application Mode (AM) jobs with a different command, than Session and Per-Job mode jobs. I can see a point that YarnDeployme

[jira] [Created] (FLINK-35381) LocalRecoveryITCase failure on deleting directory

2024-05-16 Thread Ryan Skraba (Jira)
Ryan Skraba created FLINK-35381: --- Summary: LocalRecoveryITCase failure on deleting directory Key: FLINK-35381 URL: https://issues.apache.org/jira/browse/FLINK-35381 Project: Flink Issue Type: B

[jira] [Created] (FLINK-35380) ResumeCheckpointManuallyITCase hanging on tests

2024-05-16 Thread Ryan Skraba (Jira)
Ryan Skraba created FLINK-35380: --- Summary: ResumeCheckpointManuallyITCase hanging on tests Key: FLINK-35380 URL: https://issues.apache.org/jira/browse/FLINK-35380 Project: Flink Issue Type: Bu

[jira] [Created] (FLINK-35379) File merging manager is not properly notified about checkpoint

2024-05-16 Thread Zakelly Lan (Jira)
Zakelly Lan created FLINK-35379: --- Summary: File merging manager is not properly notified about checkpoint Key: FLINK-35379 URL: https://issues.apache.org/jira/browse/FLINK-35379 Project: Flink

Re: [RESULT][VOTE] FLIP-449: Reorganization of flink-connector-jdbc

2024-05-16 Thread João Boto
Hmm sorry, I will try to search consensus.. I read 3 Votes but miss the binding part.. On 2024/05/16 01:36:46 Rui Fan wrote: > Thanks Joao Boto for driving the FLIP. We need 3 +1(binding) > votes according to Flink Bylaws[1] before the community accepts it. > > You can search "Consensus" and

[jira] [Created] (FLINK-35378) [FLIP-453] Promote Unified Sink API V2 to Public and Deprecate SinkFunc

2024-05-16 Thread Martijn Visser (Jira)
Martijn Visser created FLINK-35378: -- Summary: [FLIP-453] Promote Unified Sink API V2 to Public and Deprecate SinkFunc Key: FLINK-35378 URL: https://issues.apache.org/jira/browse/FLINK-35378 Project:

Re: [DISCUSS] FLINK-35369: Improve `Table API and SQL` page or add new page to guide new users to right Flink SQL option

2024-05-16 Thread Martijn Visser
Hi Keith Lee, There is actually an open and accepted FLIP [1] that was just never implemented. I think that needs to be taken into account as well, since it was voted on. I think it also makes more sense to have a proposal as a FLIP, instead of under a Jira. Best regards, Martijn [1] https://cw

[jira] [Created] (FLINK-35377) debezium-json support soft deletion.

2024-05-16 Thread Tongkai Yin (Jira)
Tongkai Yin created FLINK-35377: --- Summary: debezium-json support soft deletion. Key: FLINK-35377 URL: https://issues.apache.org/jira/browse/FLINK-35377 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-35376) When flink submits the job by calling the rest api, the dependent jar package generated to the tmp is not removed

2024-05-16 Thread Jira
中国无锡周良 created FLINK-35376: -- Summary: When flink submits the job by calling the rest api, the dependent jar package generated to the tmp is not removed Key: FLINK-35376 URL: https://issues.apache.org/jira/browse/FLINK-35

Re: [DISCUSS] FLIP-443: Interruptible watermark processing

2024-05-16 Thread Rui Fan
Hi Piotr, > we checked in the firing timers benchmark [1] and we didn't observe any > performance regression. Thanks for the feedback, it's good news to hear that. I didn't notice we already have fireProcessingTimers benchmark. If so, we can follow it after this FLIP is merged. +1 for this FLIP

Re: [DISCUSS] FLIP-443: Interruptible watermark processing

2024-05-16 Thread Piotr Nowojski
Hi Zakelly, > I'm suggesting skipping the continuation mail during draining of async state access. I see. That makes sense to me now. I will later update the FLIP. > the code path will become more complex after this FLIP due to the addition of shouldIntterupt() checks, right? Yes, that's correc

Re: [DISCUSS] FLIP-444: Native file copy support

2024-05-16 Thread Piotr Nowojski
Hi Lorenzo, > • concerns about memory and CPU used out of Flink's control Please note that using AWS SDKv2 would have the same concerns. In both cases Flink can control only to a certain extent what either the SDKv2 library does under the hood or the s5cmd process, via configuration parameters. S

Re: [VOTE] Apache Flink CDC Release 3.1.0, release candidate #3

2024-05-16 Thread Jingsong Li
+1 (binding) - Verified signature and checksum hash - Verified that no binaries exist in the source archive - Build source code successful - Reviewed the release web PR - Check tag and branch exist Best, Jingsong On Thu, May 16, 2024 at 4:25 PM Jark Wu wrote: > > +1 (binding) > > - checked sign

Re: [VOTE] Apache Flink CDC Release 3.1.0, release candidate #3

2024-05-16 Thread Jark Wu
+1 (binding) - checked signatures - checked hashes - checked release notes - reviewed the release web PR - checked the jars in the staging repo - build and compile the source code locally with jdk8 Best, Jark On Wed, 15 May 2024 at 16:05, gongzhongqiang wrote: > +1 (non-binding) > > - Verified

[jira] [Created] (FLINK-35375) About a question report "Failed to deserialize data of EventHeaderV4" of flink cdc2.4.2

2024-05-16 Thread Jira
田元志 created FLINK-35375: --- Summary: About a question report "Failed to deserialize data of EventHeaderV4" of flink cdc2.4.2 Key: FLINK-35375 URL: https://issues.apache.org/jira/browse/FLINK-35375 Project: Flink

[jira] [Created] (FLINK-35374) Flink 1.14 kafka connector Demo Error

2024-05-16 Thread hongxu han (Jira)
hongxu han created FLINK-35374: -- Summary: Flink 1.14 kafka connector Demo Error Key: FLINK-35374 URL: https://issues.apache.org/jira/browse/FLINK-35374 Project: Flink Issue Type: Bug C

[jira] [Created] (FLINK-35373) Beautify Flink CDC Paimon Sink web ui

2024-05-16 Thread JunboWang (Jira)
JunboWang created FLINK-35373: - Summary: Beautify Flink CDC Paimon Sink web ui Key: FLINK-35373 URL: https://issues.apache.org/jira/browse/FLINK-35373 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-35372) flink-cdc from mysql to doris, gives an error Unsupported type:TIME(0) when mysql data type is TIME and synchronise table structure or data.

2024-05-16 Thread joeb (Jira)
joeb created FLINK-35372: Summary: flink-cdc from mysql to doris, gives an error Unsupported type:TIME(0) when mysql data type is TIME and synchronise table structure or data. Key: FLINK-35372 URL: https://issues.apache.

Re: [RESULT][VOTE] FLIP-449: Reorganization of flink-connector-jdbc

2024-05-16 Thread lorenzo . affetti
Well, I give my +1 (non-binding), but we need more binding votes :) Looking forward to see this approved and see the changes. On May 16, 2024 at 03:37 +0200, Rui Fan <1996fan...@gmail.com>, wrote: > Thanks Joao Boto for driving the FLIP. We need 3 +1(binding) > votes according to Flink Bylaws[1] b

Re: [DISCUSSION] FLIP-457: Improve Table/SQL Configuration for Flink 2.0

2024-05-16 Thread Jark Wu
Hi Jane, Thanks for the proposal. +1 from my side. Best, Jark On Thu, 16 May 2024 at 10:28, Xuannan Su wrote: > Hi Jane, > > Thanks for driving this effort! And +1 for the proposed changes. > > I have one comment on the migration plan. > > For options to be moved to another module/package, I

Re: [DISCUSS] FLIP-453: Promote Unified Sink API V2 to Public and Deprecate SinkFunction

2024-05-16 Thread lorenzo . affetti
Also reviewed your PubSub PR, thanks you for the great job! On May 15, 2024 at 17:28 +0200, Muhammet Orazov , wrote: > Thanks Ahmed! > > I have reviewed your PubSub and RabbitMQ PRs, please have a look. > > Best, > Muhammet

Re: [DISCUSS] FLIP-444: Native file copy support

2024-05-16 Thread lorenzo . affetti
Hello Piotr and thanks for this proposal! The idea sounds smart and very well grounded thank you! Also here, as others, I have some doubts about invoking an external binary (namely s5cmd): • concerns about memory and CPU used out of Flink's control • deployment concerns from the usability perspe