Re: Is there any options to control the file names in file sink

2024-03-20 Thread Yanfei Lei
Hi Lasse, If the datastream job is used, you can try setting `OutputFileConfig` for file sink, something like[1]: ``` OutputFileConfig config = OutputFileConfig .builder() .withPartPrefix("prefix") .withPartSuffix(".ext") .build(); FileSink> sink = FileSink .forRowFormat((new Path(outputPath)

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-20 Thread Zakelly Lan
Congratulations! Best, Zakelly On Thu, Mar 21, 2024 at 12:05 PM weijie guo wrote: > Congratulations! Well done. > > > Best regards, > > Weijie > > > Feng Jin 于2024年3月21日周四 11:40写道： > >> Congratulations! >> >> >> Best, >> Feng >> >> >> On Thu, Mar 21, 2024 at 11:37 AM Ron liu wrote: >> >> > C

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-20 Thread weijie guo
Congratulations! Well done. Best regards, Weijie Feng Jin 于2024年3月21日周四 11:40写道： > Congratulations! > > > Best, > Feng > > > On Thu, Mar 21, 2024 at 11:37 AM Ron liu wrote: > > > Congratulations! > > > > Best, > > Ron > > > > Jark Wu 于2024年3月21日周四 10:46写道： > > > > > Congratulations and wel

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-20 Thread Feng Jin
Congratulations! Best, Feng On Thu, Mar 21, 2024 at 11:37 AM Ron liu wrote: > Congratulations! > > Best, > Ron > > Jark Wu 于2024年3月21日周四 10:46写道： > > > Congratulations and welcome! > > > > Best, > > Jark > > > > On Thu, 21 Mar 2024 at 10:35, Rui Fan <1996fan...@gmail.com> wrote: > > > > > Co

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-20 Thread Ron liu
Congratulations! Best, Ron Jark Wu 于2024年3月21日周四 10:46写道： > Congratulations and welcome! > > Best, > Jark > > On Thu, 21 Mar 2024 at 10:35, Rui Fan <1996fan...@gmail.com> wrote: > > > Congratulations! > > > > Best, > > Rui > > > > On Thu, Mar 21, 2024 at 10:25 AM Hang Ruan > wrote: > > > > > C

Indefinite hanging when querying batch transactional data from kafka

2024-03-20 Thread Dongwoo Kim
Hello community, I reported a potential bug with the flink kafka connector on Jira a few weeks ago. But I haven't received feedback yet. Could anyone kindly review this issue? Any feedback would be much appreciated. Jira issue: https://issues.apache.org/jira/projects/FLINK/issues/FLINK-34470 Than

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-20 Thread Jark Wu
Congratulations and welcome! Best, Jark On Thu, 21 Mar 2024 at 10:35, Rui Fan <1996fan...@gmail.com> wrote: > Congratulations! > > Best, > Rui > > On Thu, Mar 21, 2024 at 10:25 AM Hang Ruan wrote: > > > Congrattulations! > > > > Best, > > Hang > > > > Lincoln Lee 于2024年3月21日周四 09:54写道： > > > >

Re:Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-20 Thread Xuyang
Cheers! -- Best！ Xuyang 在 2024-03-21 10:28:45，"Rui Fan" <1996fan...@gmail.com> 写道： >Congratulations! > >Best, >Rui > >On Thu, Mar 21, 2024 at 10:25 AM Hang Ruan wrote: > >> Congrattulations! >> >> Best, >> Hang >> >> Lincoln Lee 于2024年3月21日周四 09:54写道： >> >>> >>> Congrats, thanks for

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-20 Thread Rui Fan
Congratulations! Best, Rui On Thu, Mar 21, 2024 at 10:25 AM Hang Ruan wrote: > Congrattulations! > > Best, > Hang > > Lincoln Lee 于2024年3月21日周四 09:54写道： > >> >> Congrats, thanks for the great work! >> >> >> Best, >> Lincoln Lee >> >> >> Peter Huang 于2024年3月20日周三 22:48写道： >> >>> Congratulation

Re:Is there any options to control the file names in file sink

2024-03-20 Thread Xuyang
Hi, Nedergaard. IIUC, currently, filesystem sink is not supported to custom the name of the output files, and only custom naming of directories should be supported. [1] https://github.com/apache/flink/blob/4142c4386a92f1ec5016583f4832f8869782765e/flink-connectors/flink-connector-files/src/main/

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-20 Thread Hang Ruan
Congrattulations! Best, Hang Lincoln Lee 于2024年3月21日周四 09:54写道： > > Congrats, thanks for the great work! > > > Best, > Lincoln Lee > > > Peter Huang 于2024年3月20日周三 22:48写道： > >> Congratulations >> >> >> Best Regards >> Peter Huang >> >> On Wed, Mar 20, 2024 at 6:56 AM Huajie Wang wrote: >> >>>

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-20 Thread Lincoln Lee
Congrats, thanks for the great work! Best, Lincoln Lee Peter Huang 于2024年3月20日周三 22:48写道： > Congratulations > > > Best Regards > Peter Huang > > On Wed, Mar 20, 2024 at 6:56 AM Huajie Wang wrote: > >> >> Congratulations >> >> >> >> Best, >> Huajie Wang >> >> >> >> Leonard Xu 于2024年3月20日周三 2

Re: End-to-end lag spikes when closing a large number of panes

2024-03-20 Thread Caio Camatta via user
Hey Asimansu, The inputDataStream is a KeyedStream, I forgot to mention that. Caio On Wed, Mar 20, 2024 at 6:56 PM Asimansu Bera wrote: > Hello Caio, > > Based on the pseudocode, there is no keyed function present. Hence, the > window will not be processed parallely . Please check again and re

Re: End-to-end lag spikes when closing a large number of panes

2024-03-20 Thread Asimansu Bera
Hello Caio, Based on the pseudocode, there is no keyed function present. Hence, the window will not be processed parallely . Please check again and respond back. val windowDataStream = inputDataStream .window(TumblingEventTimeWindows of 1 hour) .trigger(custom trigger) .aggregat

Re: Need help in understanding PojoSerializer

2024-03-20 Thread Alexis Sarda-Espinosa
Hi Sachin, Check the last few comments I wrote in this thread: https://lists.apache.org/thread/l71d1cqo9xv8rsw0gfjo19kb1pct2xj1 Regards, Alexis. On Wed, 20 Mar 2024, 18:51 Sachin Mittal, wrote: > Hi, > I saw the post but I did not understand how I would configure these fields > to use those s

Re: Need help in understanding PojoSerializer

2024-03-20 Thread Sachin Mittal
Hi, I saw the post but I did not understand how I would configure these fields to use those serializers. (I can change the set type to a list type for now). As per the docs I see that we can annotate fields with @TypeInfo But what I did not get is how using this annotation I can use ListSerializer

Re: Need help in understanding PojoSerializer

2024-03-20 Thread Ken Krugler
Flink doesn’t have built-in support for serializing Sets. See this (stale) issue about the same: https://issues.apache.org/jira/browse/FLINK-16729 You could create a custom serializer for sets, see https://stackoverflow.com/questions/59800851/flink-serialization-of-java-util-list-and-java-util-

Need help in understanding PojoSerializer

2024-03-20 Thread Sachin Mittal
Hi, I have a Pojo class like this public class A { public String str; public Set aSet; public Map dMap; } However when I start the flink program I get this message: org.apache.flink.api.java.typeutils.TypeExtractor[] - Field A# dMap will be processed as GenericType. Please read t

End-to-end lag spikes when closing a large number of panes

2024-03-20 Thread Caio Camatta via user
We run a large-scale Flink 1.16 cluster that uses windowed aggregations and we’re seeing lag spikes on window closure. I’m curious if others have encountered similar issues before and if anyone has suggestions for how to tackle this problem (other than simply increasing parallelism). ContextLag def

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-20 Thread Peter Huang
Congratulations Best Regards Peter Huang On Wed, Mar 20, 2024 at 6:56 AM Huajie Wang wrote: > > Congratulations > > > > Best, > Huajie Wang > > > > Leonard Xu 于2024年3月20日周三 21:36写道： > >> Hi devs and users, >> >> We are thrilled to announce that the donation of Flink CDC as a >> sub-project of

Announcing the Community Over Code 2024 Streaming Track

2024-03-20 Thread James Hughes
Hi all, Community Over Code , the ASF conference, will be held in Denver, Colorado, October 7-10, 2024. The call for presentations is open now

Re: [ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-20 Thread Huajie Wang
Congratulations Best, Huajie Wang Leonard Xu 于2024年3月20日周三 21:36写道： > Hi devs and users, > > We are thrilled to announce that the donation of Flink CDC as a > sub-project of Apache Flink has completed. We invite you to explore the new > resources available: > > - GitHub Repository: https://

[ANNOUNCE] Donation Flink CDC into Apache Flink has Completed

2024-03-20 Thread Leonard Xu
Hi devs and users, We are thrilled to announce that the donation of Flink CDC as a sub-project of Apache Flink has completed. We invite you to explore the new resources available: - GitHub Repository: https://github.com/apache/flink-cdc - Flink CDC Documentation: https://nightlies.apache.org/f

Is there any options to control the file names in file sink

2024-03-20 Thread Lasse Nedergaard
Hi. Anyone know if it’s possible to control the file names eg change the uuid file names and extensions to something else. I know I can control the path but I would like to be able to set the file name and the extension based on the data in the stream. So I can’t use any general pre or postfix

Re: can we use Scan Newly Added Tables without restarting the existing job ?

2024-03-20 Thread Hang Ruan
Hi, 3pang zhu. This `Scan Newly added tables` feature requires restarting the job from the savepoint. We cannot add new tables to the running job without restarting by now. Best, Hang 3pang zhu 于2024年3月20日周三 15:22写道： > this link has describe the usage for [Scan Newly Added Tables] > https://ni