Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

2023-12-06 Thread Jing Ge
Awesome! +1 Best regards, Jing On Thu, Dec 7, 2023 at 8:34 AM Sergey Nuyanzin wrote: > thanks for working on this and driving it > > +1 > > On Thu, Dec 7, 2023 at 7:26 AM Feng Jin wrote: > > > This is incredibly exciting news, a big +1 for this. > > > > Thank you for the fantastic work on

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

2023-12-06 Thread Sergey Nuyanzin
thanks for working on this and driving it +1 On Thu, Dec 7, 2023 at 7:26 AM Feng Jin wrote: > This is incredibly exciting news, a big +1 for this. > > Thank you for the fantastic work on Flink CDC. We have created thousands of > real-time integration jobs using Flink CDC connectors. > > >

SQL return type change from 1.17 to 1.18

2023-12-06 Thread Péter Váry
Hi Team, We are working on upgrading the Iceberg-Flink connector from 1.17 to 1.18, and found that some of our tests are failing. Prabhu Joseph created a jira [1] to discuss this issue, along with short example code. In a nutshell: - Create a table with an 'ARRAY' column - Run a select which

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

2023-12-06 Thread Feng Jin
This is incredibly exciting news, a big +1 for this. Thank you for the fantastic work on Flink CDC. We have created thousands of real-time integration jobs using Flink CDC connectors. Best, Feng On Thu, Dec 7, 2023 at 1:45 PM gongzhongqiang wrote: > It's very exciting to hear the news. > +1

Re: [DISCUSS] FLIP-400: AsyncScalarFunction for asynchronous scalar function support

2023-12-06 Thread Alan Sheinberg
> > Nicely written and makes sense. The only feedback I have is around the > naming of the generalization, e.g. "Specifically, PythonCalcSplitRuleBase > will be generalized into RemoteCalcSplitRuleBase." This naming seems to > imply/suggest that all Async functions are remote. I wonder if we

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

2023-12-06 Thread gongzhongqiang
It's very exciting to hear the news. +1 for adding CDC Connectors to Apache Flink ! Best, Zhongqiang Leonard Xu 于2023年12月7日周四 11:25写道: > Dear Flink devs, > > > As you may have heard, we at Alibaba (Ververica) are planning to donate CDC > Connectors for the Apache Flink project > *[1]* to

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

2023-12-06 Thread Jiabao Sun
Very excited to hear the news, +1 for this proposal. Best, Jiabao > 2023年12月7日 13:08,Rui Fan <1996fan...@gmail.com> 写道: > > It's cool! > > Thanks for the great work of CDC, > +1 for this donation. > > Best, > Rui > > On Thu, Dec 7, 2023 at 12:28 PM Hongshun Wang > wrote: > >> So cool,

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

2023-12-06 Thread Rui Fan
It's cool! Thanks for the great work of CDC, +1 for this donation. Best, Rui On Thu, Dec 7, 2023 at 12:28 PM Hongshun Wang wrote: > So cool, Big +1 for this exciting work. > > Best > Hongshun > > On Thu, Dec 7, 2023 at 12:20 PM Qingsheng Ren wrote: > > > Thanks for kicking this off, Leonard!

[DISCUSS] FLIP-399: Flink Connector Doris

2023-12-06 Thread wudi
Hi all, As discussed in the previous email [1], about contributing the Flink Doris Connector to the Flink community. Apache Doris[2] is a high-performance, real-time analytical database based on MPP architecture, for scenarios where Flink is used for data analysis, processing, or real-time

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

2023-12-06 Thread Hongshun Wang
So cool, Big +1 for this exciting work. Best Hongshun On Thu, Dec 7, 2023 at 12:20 PM Qingsheng Ren wrote: > Thanks for kicking this off, Leonard! > > As one of the contributors of the CDC project, I'm truly honored to be part > of the community and so excited to hear the news. CDC project was

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

2023-12-06 Thread Qingsheng Ren
Thanks for kicking this off, Leonard! As one of the contributors of the CDC project, I'm truly honored to be part of the community and so excited to hear the news. CDC project was born from and developed together with Apache Flink, and we are so proud to be accepted by more and more users around

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

2023-12-06 Thread Jark Wu
+1 for adding this to Apache Flink! I think this can further extend the ability of Apache Flink and a lot of users would be interested to try this out. Best, Jark On Thu, 7 Dec 2023 at 12:06, Samrat Deb wrote: > That's really cool :) > +1 for the great addition > > Bests, > Samrat > > On Thu,

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

2023-12-06 Thread Xintong Song
A big +1 for this proposal. Thanks Leonard and the Flink CDC community. While there are many further details to be discussed, I believe this proposal is aligned with the long-term interests for both communities. Best, Xintong On Thu, Dec 7, 2023 at 12:06 PM Samrat Deb wrote: > That's

Re:Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

2023-12-06 Thread Xuyang
Big +1 for this exciting work. -- Best! Xuyang 在 2023-12-07 12:06:07,"Samrat Deb" 写道: >That's really cool :) >+1 for the great addition > >Bests, >Samrat > >On Thu, 7 Dec 2023 at 9:20 AM, Jingsong Li wrote: > >> Wow, Cool, Nice >> >> CDC is playing an increasingly important

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

2023-12-06 Thread Samrat Deb
That's really cool :) +1 for the great addition Bests, Samrat On Thu, 7 Dec 2023 at 9:20 AM, Jingsong Li wrote: > Wow, Cool, Nice > > CDC is playing an increasingly important role. > > +1 > > Best, > Jingsong > > On Thu, Dec 7, 2023 at 11:25 AM Leonard Xu wrote: > > > > Dear Flink devs, > > >

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

2023-12-06 Thread Shengkai Fang
Thanks for all Flink CDC maintainers great work! Big +1. Best, Shengkai tison 于2023年12月7日周四 12:01写道: > This is very cool! +1 from my side. > > Best, > tison. > > Benchao Li 于2023年12月7日周四 11:56写道: > > > > Thank you, Leonard and all the Flink CDC maintainers. > > > > Big big +1 from me. As a

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

2023-12-06 Thread tison
This is very cool! +1 from my side. Best, tison. Benchao Li 于2023年12月7日周四 11:56写道: > > Thank you, Leonard and all the Flink CDC maintainers. > > Big big +1 from me. As a heavy user of both Flink and Flink CDC, I've > already taken them as a whole project. > > Leonard Xu 于2023年12月7日周四 11:25写道:

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

2023-12-06 Thread Benchao Li
Thank you, Leonard and all the Flink CDC maintainers. Big big +1 from me. As a heavy user of both Flink and Flink CDC, I've already taken them as a whole project. Leonard Xu 于2023年12月7日周四 11:25写道: > > Dear Flink devs, > > As you may have heard, we at Alibaba (Ververica) are planning to donate

Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

2023-12-06 Thread Jingsong Li
Wow, Cool, Nice CDC is playing an increasingly important role. +1 Best, Jingsong On Thu, Dec 7, 2023 at 11:25 AM Leonard Xu wrote: > > Dear Flink devs, > > As you may have heard, we at Alibaba (Ververica) are planning to donate CDC > Connectors for the Apache Flink project[1] to the Apache

[PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink

2023-12-06 Thread Leonard Xu
Dear Flink devs, As you may have heard, we at Alibaba (Ververica) are planning to donate CDC Connectors for the Apache Flink project[1] to the Apache Flink community. CDC Connectors for Apache Flink comprise a collection of source connectors designed specifically for Apache Flink. These

[DISCUSS] FLIP-398: Improve Serialization Configuration And Usage In Flink

2023-12-06 Thread Yong Fang
Hi devs, I'd like to start a discussion about FLIP-398: Improve Serialization Configuration And Usage In Flink [1]. Currently, users can register custom data types and serializers in Flink jobs through various methods, including registration in code, configuration, and annotations. These lead to

[jira] [Created] (FLINK-33768) Support dynamic source parallelism inference for batch jobs

2023-12-06 Thread xingbe (Jira)
xingbe created FLINK-33768: -- Summary: Support dynamic source parallelism inference for batch jobs Key: FLINK-33768 URL: https://issues.apache.org/jira/browse/FLINK-33768 Project: Flink Issue Type:

Re: [DISCUSS] FLIP-400: AsyncScalarFunction for asynchronous scalar function support

2023-12-06 Thread Jim Hughes
Hi Alan, Nicely written and makes sense. The only feedback I have is around the naming of the generalization, e.g. "Specifically, PythonCalcSplitRuleBase will be generalized into RemoteCalcSplitRuleBase." This naming seems to imply/suggest that all Async functions are remote. I wonder if we

[DISCUSS] FLIP-400: AsyncScalarFunction for asynchronous scalar function support

2023-12-06 Thread Alan Sheinberg
I'd like to start a discussion of FLIP-400: AsyncScalarFunction for asynchronous scalar function support [1] This feature proposes adding a new UDF type AsyncScalarFunction which is invoked just like a normal ScalarFunction, but is implemented with an asynchronous eval method. I had brought this

[jira] [Created] (FLINK-33767) Implement restore tests for TemporalJoin node

2023-12-06 Thread Jim Hughes (Jira)
Jim Hughes created FLINK-33767: -- Summary: Implement restore tests for TemporalJoin node Key: FLINK-33767 URL: https://issues.apache.org/jira/browse/FLINK-33767 Project: Flink Issue Type:

[jira] [Created] (FLINK-33766) Support MockEnvironment in BroadcastOperatorTestHarness

2023-12-06 Thread Koala Lam (Jira)
Koala Lam created FLINK-33766: - Summary: Support MockEnvironment in BroadcastOperatorTestHarness Key: FLINK-33766 URL: https://issues.apache.org/jira/browse/FLINK-33766 Project: Flink Issue

[jira] [Created] (FLINK-33765) Flink SQL to support COLLECTLIST

2023-12-06 Thread Zhenzhong Xu (Jira)
Zhenzhong Xu created FLINK-33765: Summary: Flink SQL to support COLLECTLIST Key: FLINK-33765 URL: https://issues.apache.org/jira/browse/FLINK-33765 Project: Flink Issue Type: Improvement

Re: [jira] [Created] (FLINK-33753) ContinuousFileReaderOperator consume records as mini batch

2023-12-06 Thread Prabhu Joseph
I understand the operator wouldn't be causing the checkpoint issue based on your comments. I need to debug further, but the symptoms are not showing up again for my Cx. I will close this as not a problem. Thanks for the detailed explanations. On Wed, Dec 6, 2023 at 8:22 PM Darin Amos wrote: >

[jira] [Created] (FLINK-33764) Incorporate GC / Heap metrics in autoscaler decisions

2023-12-06 Thread Gyula Fora (Jira)
Gyula Fora created FLINK-33764: -- Summary: Incorporate GC / Heap metrics in autoscaler decisions Key: FLINK-33764 URL: https://issues.apache.org/jira/browse/FLINK-33764 Project: Flink Issue

Re: Flink Operator - Supporting Recovery from Snapshot

2023-12-06 Thread Gyula Fóra
Hi All! Based on some continuous feedback and experience, we feel that it may be a good time to introduce this functionality in a way that doesn't accidentally affect existing users in an unexpected way. Please see: https://issues.apache.org/jira/browse/FLINK-33763 for details and review.

[jira] [Created] (FLINK-33763) Support manual savepoint redeploy for jobs and deployments

2023-12-06 Thread Gyula Fora (Jira)
Gyula Fora created FLINK-33763: -- Summary: Support manual savepoint redeploy for jobs and deployments Key: FLINK-33763 URL: https://issues.apache.org/jira/browse/FLINK-33763 Project: Flink Issue

Re: [jira] [Created] (FLINK-33753) ContinuousFileReaderOperator consume records as mini batch

2023-12-06 Thread Darin Amos
Just to confirm, I ran a test this morning using a large file with only a single file split that took over 10 minutes to process. My checkpoints still executed as expected every minute with minimal latency (sub second). Though I ran this with 1.15.4. What evidence do you have that this operator

[jira] [Created] (FLINK-33762) Versioned release of flink-connector-shared-utils python scripts

2023-12-06 Thread Peter Vary (Jira)
Peter Vary created FLINK-33762: -- Summary: Versioned release of flink-connector-shared-utils python scripts Key: FLINK-33762 URL: https://issues.apache.org/jira/browse/FLINK-33762 Project: Flink

Re: [DISCUSS] REST API response parsing throws exception on new fields

2023-12-06 Thread Gyula Fóra
Thanks G, I think this is a reasonable proposal which will increase compatibility between different Flink clients and versions such as SQL Gateway, CLI, Flink operator etc. I don't really see any harm in ignoring unknown json fields globally, but this probably warrants a FLIP and a proper vote.

[DISCUSS] REST API response parsing throws exception on new fields

2023-12-06 Thread Gabor Somogyi
Hi All, Since the possible solution can have effect on all the REST response deserialization I would like ask for opinions. *Problem statement:* At the moment Flink is not ignoring unknown fields when parsing REST responses. An example for such a class is JobDetailsInfo but this applies to all

Re: [jira] [Created] (FLINK-33753) ContinuousFileReaderOperator consume records as mini batch

2023-12-06 Thread Prabhu Joseph
Thanks Darin for the details. Below is the problematic StreamTask, which has not processed the priority event (checkpoint barrier) from the mailbox, and has been there throughout the checkpoint timeout interval. And it is reading and collecting all the records from the split. Not sure why the

Re:Re: Re: [DISCUSS] FLIP-392: Deprecate the Legacy Group Window Aggregation

2023-12-06 Thread Xuyang
Hi, Shengkai. Thanks to share your thought. Let me answer related questions。 > Could you give an example about the pass-through column. A session window may > contain multiple rows, which value is selected by the windowoperator? The table function make the entire inpyt row available in the