Re: Table API and ProcessWindowFunction

Thanks Hequn, I'll give it a try! Best, Flavio On Thu, Jul 11, 2019 at 3:38 AM Hequn Cheng wrote: > Hi, > > > Can you provide a pseudo-code example of how to implement this? > Processing time > If you use a TumblingProcessingTimeWindows.of(Time.seconds(1)), for each > record, you get the timest

Re: Table API and ProcessWindowFunction

Only one proposal here: many times it happens that when working with streaming sources you need to define which field is the processing/row. Right now you could define the processing or event time field implementingthe DefinedProctimeAttribute or DefinedRowtimeAttribute at source. But this is only

Re: How are kafka consumer offsets handled if sink fails?

Hi John, let's say Flink performed a checkpoint after the 2nd record (by injecting a checkpoint marker into the data flow) and the sink fails on the 5th record. When Flink restarts the application, it resets the offset after the 2nd record (it will read the 3rd record first). Hence, the 3rd and 4t

Re: Cannot catch exception throws by kafka consumer with JSONKeyValueDeserializationSchema

Hi, I'd suggest to implement your own custom deserialization schema for example by extending JSONKeyValueDeserializationSchema. Then you can implement whatever logic you need to handle incorrectly formatted messages. Best, Fabian Am Mi., 10. Juli 2019 um 04:29 Uhr schrieb Zhechao Ma < mazhechaom

Re: new user does not run job use flink cli

Hi, Do you mean job submission is OK with local user name "flink", but not for other users? Have you ever checked the authorization of "hdfs://user/flink/recovery"? I guess other users do not have the access right. &#38;#38;#38;#38;#10084; <799326...@qq.com> 于2019年7月11日周四 上午11:55写道： > flink

Re: Apache Flink - Relation between stream time characteristic and timer triggers

Hi Mans, IngestionTime is uses the same internal mechanisms as EventTime (record timestamps and watermarks). The difference is that instead of extracting a timestamp from the record (using a custom timestamp extractor & wm assigner), Flink will assign timestamps based on the machine clock of the

CEP Pattern limit

Hello, I'm using CEP to match a stream against around 1000 different patterns. To do this I create de patterns and then iterate and call CEP.pattern() for each. Later on, I merge the PatternStreams into one using datastream.union(). The problem is that i'm getting this exception: AstTimeoutExcep

[ANNOUNCE] Rong Rong becomes a Flink committer

Hi everyone, I'm very happy to announce that Rong Rong accepted the offer of the Flink PMC to become a committer of the Flink project. Rong has been contributing to Flink for many years, mainly working on SQL and Yarn security features. He's also frequently helping out on the user@f.a.o mailing l

Re: [ANNOUNCE] Rong Rong becomes a Flink committer

Congratulations Rong Rong! Welcome on board! On Thu, 11 Jul 2019 at 22:25, Fabian Hueske wrote: > Hi everyone, > > I'm very happy to announce that Rong Rong accepted the offer of the Flink > PMC to become a committer of the Flink project. > > Rong has been contributing to Flink for many years, m

Re: [ANNOUNCE] Rong Rong becomes a Flink committer

Congratulations Rong! On Thu, Jul 11, 2019 at 4:40 PM Jark Wu wrote: > Congratulations Rong Rong! > Welcome on board! > > On Thu, 11 Jul 2019 at 22:25, Fabian Hueske wrote: > >> Hi everyone, >> >> I'm very happy to announce that Rong Rong accepted the offer of the Flink >> PMC to become a commi

Re: [ANNOUNCE] Rong Rong becomes a Flink committer

Congratulations Rong! Best, Kurt On Thu, Jul 11, 2019 at 10:53 PM Kostas Kloudas wrote: > Congratulations Rong! > > On Thu, Jul 11, 2019 at 4:40 PM Jark Wu wrote: > >> Congratulations Rong Rong! >> Welcome on board! >> >> On Thu, 11 Jul 2019 at 22:25, Fabian Hueske wrote: >> >>> Hi everyone,

Re: [ANNOUNCE] Rong Rong becomes a Flink committer

Congratulations Rong! On Thu, Jul 11, 2019 at 10:40 PM Jark Wu wrote: > Congratulations Rong Rong! > Welcome on board! > > On Thu, 11 Jul 2019 at 22:25, Fabian Hueske wrote: > >> Hi everyone, >> >> I'm very happy to announce that Rong Rong accepted the offer of the Flink >> PMC to become a com

Re: [ANNOUNCE] Rong Rong becomes a Flink committer

Congratulations Rong! Best, Zhijiang -- From:Kurt Young Send Time:2019年7月11日(星期四) 22:54 To:Kostas Kloudas Cc:Jark Wu ; Fabian Hueske ; dev ; user Subject:Re: [ANNOUNCE] Rong Rong becomes a Flink committer Congratulations Rong! B

Re: How are kafka consumer offsets handled if sink fails?

Ok cool. I will try to make my stored proc idempotent. So there no chance that there's a checkpoint happens after the 5th record and the 5th record is missed? On Thu, 11 Jul 2019 at 05:20, Fabian Hueske wrote: > Hi John, > > let's say Flink performed a checkpoint after the 2nd record (by injecti

Re: [ANNOUNCE] Rong Rong becomes a Flink committer

Congratulations Rong! Best Regards, Yu On Thu, 11 Jul 2019 at 22:54, zhijiang wrote: > Congratulations Rong! > > Best, > Zhijiang > > -- > From:Kurt Young > Send Time:2019年7月11日(星期四) 22:54 > To:Kostas Kloudas > Cc:Jark Wu ; Fabi

Re: Apache Flink - Relation between stream time characteristic and timer triggers

Thanks Fabian for your response. Just to clarify then - regardless of the time characteristics, if a processor or window trigger registers with a ProcessingTime  and EventTime  timers - they will all fire when the appropriate watermarks arrive. Thanks again. On Thursday, July 11, 2019, 05:41

Re: Apache Flink - Relation between stream time characteristic and timer triggers

Hi, ProcessingTime timers are always supported EventTime timers are only supported for EventTime and IngestionTime Best, Fabian Am Do., 11. Juli 2019 um 17:44 Uhr schrieb M Singh : > Thanks Fabian for your response. > > Just to clarify then - regardless of the time characteristics, if a > proce

Issue starting Flink job with with Avro class

Hello! We are working on a Flink application and came across this error. The "Record" class is a class generated from an Avro Schema. It's actually used by a second "Operation" class which doesn't seem to have this problem. Has anyone seen this before? org.apache.flink.streaming.runtime.tasks.Str

Re: [ANNOUNCE] Rong Rong becomes a Flink committer

Congratulations, Rong! On Thu, Jul 11, 2019 at 8:26 AM Yu Li wrote: > Congratulations Rong! > > Best Regards, > Yu > > > On Thu, 11 Jul 2019 at 22:54, zhijiang wrote: > >> Congratulations Rong! >> >> Best, >> Zhijiang >> >> -- >> F

Re: [ANNOUNCE] Rong Rong becomes a Flink committer

Congrats Rong! Best, Xingcan > On Jul 11, 2019, at 1:08 PM, Shuyi Chen wrote: > > Congratulations, Rong! > > On Thu, Jul 11, 2019 at 8:26 AM Yu Li > wrote: > Congratulations Rong! > > Best Regards, > Yu > > > On Thu, 11 Jul 2019 at 22:54, zhijiang

Re: [ANNOUNCE] Rong Rong becomes a Flink committer

Congrats, Rong! On Fri, Jul 12, 2019 at 1:13 AM Xingcan Cui wrote: > Congrats Rong! > > Best, > Xingcan > > On Jul 11, 2019, at 1:08 PM, Shuyi Chen wrote: > > Congratulations, Rong! > > On Thu, Jul 11, 2019 at 8:26 AM Yu Li wrote: > >> Congratulations Rong! >> >> Best Regards, >> Yu >> >> >> O

Re: [ANNOUNCE] Rong Rong becomes a Flink committer

Congrats Rong! On Thu, Jul 11, 2019 at 10:40 AM Becket Qin wrote: > Congrats, Rong! > > On Fri, Jul 12, 2019 at 1:13 AM Xingcan Cui wrote: > >> Congrats Rong! >> >> Best, >> Xingcan >> >> On Jul 11, 2019, at 1:08 PM, Shuyi Chen wrote: >> >> Congratulations, Rong! >> >> On Thu, Jul 11, 2019 at

Re: [ANNOUNCE] Rong Rong becomes a Flink committer

Congratulations Rong! --- Oytun Tez *M O T A W O R D* The World's Fastest Human Translation Platform. oy...@motaword.com — www.motaword.com On Thu, Jul 11, 2019 at 1:44 PM Peter Huang wrote: > Congrats Rong! > > On Thu, Jul 11, 2019 at 10:40 AM Becket Qin wrote: > >> Congrats, Rong! >> >> On

Question in the tutorial

Hey Flink team, Novice here. I just about started using Flink. I have a slight issue with respect to the following instruction in the tutorial. https://ci.apache.org/projects/flink/flink-docs-release-1.8/tutorials/local_setup.html In this, when I try to 'Start a Local Flink Cluster' and then conn

Re: [ANNOUNCE] Rong Rong becomes a Flink committer

Congrats, Rong! On Thu, Jul 11, 2019 at 10:48 AM Oytun Tez wrote: > Congratulations Rong! > > --- > Oytun Tez > > *M O T A W O R D* > The World's Fastest Human Translation Platform. > oy...@motaword.com — www.motaword.com > > > On Thu, Jul 11, 2019 at 1:44 PM Peter Huang > wrote: > >> Congrats

Re: Apache Flink - Relation between stream time characteristic and timer triggers

Thanks Fabian/Xingcan/Yun for all your help.  Mans On Thursday, July 11, 2019, 11:46:42 AM EDT, Fabian Hueske wrote: Hi, ProcessingTime timers are always supportedEventTime timers are only supported for EventTime and IngestionTime Best, Fabian Am Do., 11. Juli 2019 um 17:44 Uhr schrie

Re: [ANNOUNCE] Rong Rong becomes a Flink committer

Congratulations, Rong! On Thu, Jul 11, 2019 at 10:59 AM Bowen Li wrote: > Congrats, Rong! > > > On Thu, Jul 11, 2019 at 10:48 AM Oytun Tez wrote: > > > Congratulations Rong! > > > > --- > > Oytun Tez > > > > *M O T A W O R D* > > The World's Fastest Human Translation Platform. > > oy...@motawor

Flink SQL API: Extra columns added from order by

Hi Flink folks, We have a custom date formatting function that we use to format the output of columns containing dates. Ideally what we want is to format the output in the select statement but be able to order by the underlying datetime (so that and output with formatted dates "February 2019" and

Graceful Task Manager Termination and Replacement

Hello, Is there a way to gracefully terminate a Task Manager beyond just killing it (this seems to be what `./taskmanager.sh stop` does)? Specifically I'm interested in a way to replace a Task Manager that has currently-running tasks. It would be great if it was possible to terminate a Task Manage

Re: Graceful Task Manager Termination and Replacement

I have a common interest in this topic. My k8s recycle hosts, and I am facing the same issue. Flink can tolerate this situation, but I am wondering if I can do better On Thu, Jul 11, 2019, 12:39 Aaron Levin wrote: > Hello, > > Is there a way to gracefully terminate a Task Manager beyond just kil

Re: [ANNOUNCE] Rong Rong becomes a Flink committer

Congratulations Rong. On Thu, Jul 11, 2019, 11:39 Xuefu Z wrote: > Congratulations, Rong! > > On Thu, Jul 11, 2019 at 10:59 AM Bowen Li wrote: > >> Congrats, Rong! >> >> >> On Thu, Jul 11, 2019 at 10:48 AM Oytun Tez wrote: >> >> > Congratulations Rong! >> > >> > --- >> > Oytun Tez >> > >> > *M

Re: [ANNOUNCE] Rong Rong becomes a Flink committer

Thank you all for the warm welcome! It's my honor to become an Apache Flink committer. I will continue to work on this great project and contribute more to the community. Cheers, Rong On Thu, Jul 11, 2019 at 1:05 PM Hao Sun wrote: > Congratulations Rong. > > On Thu, Jul 11, 2019, 11:39 Xuefu Z

Re: [ANNOUNCE] Rong Rong becomes a Flink committer

congratulations! 获取 Outlook for iOS 发件人: Rong Rong 发送时间: 星期五, 七月 12, 2019 8:09 上午 收件人: Hao Sun 抄送: Xuefu Z; dev; Flink ML 主题: Re: [ANNOUNCE] Rong Rong becomes a Flink committer Thank you all for the warm welcome! It's my honor to become a

Re: [ANNOUNCE] Rong Rong becomes a Flink committer

Congrats Rong! > 在 2019年7月12日，上午8:47，Chen YuZhao 写道： > > congratulations! > > 获取 Outlook for iOS > > 发件人: Rong Rong > 发送时间: 星期五, 七月 12, 2019 8:09 上午 > 收件人: Hao Sun > 抄送: Xuefu Z; dev; Flink ML > 主题: Re: [ANNOUNCE] Rong Rong becomes a Flink committer > > Thank you

Re: [ANNOUNCE] Rong Rong becomes a Flink committer

Congratulations Rong, Well deserved! Cheers, Jincheng Dian Fu 于2019年7月12日周五 上午9:06写道： > > Congrats Rong! > > > 在 2019年7月12日，上午8:47，Chen YuZhao 写道： > > congratulations! > > 获取 Outlook for iOS > > -- > *发件人:* Rong Rong > *发送时间:* 星期五, 七月 12, 20

Re: [ANNOUNCE] Rong Rong becomes a Flink committer

Congratulations Rong. Rong Rong has done a lot of nice work in the past time to the flink community. Best, JingsongLee -- From:Rong Rong Send Time:2019年7月12日(星期五) 08:09 To:Hao Sun Cc:Xuefu Z ; dev ; Flink ML Subject:Re: [ANNOUN

Re:Re: [ANNOUNCE] Rong Rong becomes a Flink committer

Congrats Rong!Best, Haibo At 2019-07-12 09:40:26, "JingsongLee" wrote: Congratulations Rong. Rong Rong has done a lot of nice work in the past time to the flink community. Best, JingsongLee -- From:Rong Rong Send Time:2019年7月1

Re: [ANNOUNCE] Rong Rong becomes a Flink committer

Congrats to Rong! Rong has contributed a lot to the community and well deserves it. Best, Paul Lam > 在 2019年7月12日，09:40，JingsongLee 写道： > > Congratulations Rong. > Rong Rong has done a lot of nice work in the past time to the flink community. > > Best, JingsongLee > > --

Re: [ANNOUNCE] Rong Rong becomes a Flink committer

congratulations Rong Rong! Fabian Hueske 于2019年7月11日周四 下午10:25写道： > Hi everyone, > > I'm very happy to announce that Rong Rong accepted the offer of the Flink > PMC to become a committer of the Flink project. > > Rong has been contributing to Flink for many years, mainly working on SQL > and Yar

Re: Graceful Task Manager Termination and Replacement

Hi, Maybe region restart strategy can help. It restarts minimum required tasks. Note that it’s recommended to use only after 1.9 release, see [1], unless you’re running a stateless job. [1] https://issues.apache.org/jira/browse/FLINK-10712 B

Re: Question in the tutorial

Hi Karthik, I think more information is needed for diagnosing the problem, and I would suggest you to do the followings: - Check whether the task managers are configured in the file 'conf/slaves'. - If configured, you should see some hosts in the file, each for one task manager. - Check whether

Re: [ANNOUNCE] Rong Rong becomes a Flink committer

Congrats, Rong! vino yang 于2019年7月12日周五 上午10:08写道： > congratulations Rong Rong! > > Fabian Hueske 于2019年7月11日周四 下午10:25写道： > >> Hi everyone, >> >> I'm very happy to announce that Rong Rong accepted the offer of the Flink >> PMC to become a committer of the Flink project. >> >> Rong has been co

Re: [ANNOUNCE] Rong Rong becomes a Flink committer

Congratulations Rong! Best, Hequn On Fri, Jul 12, 2019 at 12:19 PM Jeff Zhang wrote: > Congrats, Rong! > > > vino yang 于2019年7月12日周五 上午10:08写道： > >> congratulations Rong Rong! >> >> Fabian Hueske 于2019年7月11日周四 下午10:25写道： >> >>> Hi everyone, >>> >>> I'm very happy to announce that Rong Rong ac

Re: Table API and ProcessWindowFunction

Hi Flavio, I think the reason that we don't have interfaces like EventTimeObject and ProcessingTimeObject is we don't want to define time attributes anywhere. It is considered to define your time attributes in the source. If we add an interface like EventTimeObject and ProcessingTimeObject in Flin

Re: Question in the tutorial

It seems to me that your task manager is started and then failed before registering to the cluster. I think we need the entire log file instead of the result of tail command. Could you just clear the log dir, start the cluster again and wait for a little while (1-2 min should be enough), and post a