Re: Yarn Kerberos issue

Hello Rong, Chesnay, Thank you for your answer, the way we are trying to launch the job is through a scheduler (similar to oozie) where we have a keytab for the scheduler user and with that keytab we get delegation tokens impersonating the right user (owner of the job). But the only way I was a

Re: Checkpoints issue and job failing

Hi Navneeth, Since the file still exists, this exception is very strange. I want to ask, does it happen by accident or frequently? Another concern is that since the 1.4 version is very far away, all maintenance and response are not as timely as the recent versions. I personally recommend upgradi

Re: How long Flink state default TTL，if I don't config the state ttl config?

Ok, got it ,thank you Zhu Zhu 于2020年1月6日周一 上午10:30写道： > Yes. State TTL is by default disabled. > > Thanks, > Zhu Zhu > > LakeShen 于2020年1月6日周一 上午10:09写道： > >> I saw the flink source code, I find the flink state ttl default is >> never expire,is it right? >> >> LakeShen 于2020年1月6日周一 上午9:58写道： >

Re: Flink logging issue with logback

Hi Bajaj, >> Logs from main method(outside of job graph) do not show up in jobmanager logs. IMO, it's normal phenomena. Other ideas, please check the JVM options mentioned by Yang. Best, Vino Yang Wang 于2020年1月6日周一 上午11:18写道： > Hi Bajaj, Abhinav, > > Could you share the start-command of job

Re: [DISCUSS] Set default planner for SQL Client to Blink planner in 1.10 release

+1 for making the blink planner as default planner, the blink planner becomes more stable since 1.10 Dian Fu 于2020年1月6日周一 上午11:51写道： > +1 to set blink planner as the default planner for SQL client considering > that so many features added since 1.10 are only available in the blink > planner. > >

Re: Need guidance on a use case

Hi Reva, I'm glad to see it can help you. Quick answers for your questions: 1) Yes, it works. You can deduplicate Task table in the same way using ROW_NUMBER(). 2) Yes. It is a stream-stream join which will be triggered for new messages from both sides. Best, Jark On Sat, 28 Dec 2019 at 01:02,

Re: [DISCUSS] Set default planner for SQL Client to Blink planner in 1.10 release

+1 to set blink planner as the default planner for SQL client considering that so many features added since 1.10 are only available in the blink planner. > 在 2020年1月6日，上午11:04，Rui Li 写道： > > +1. I think it improves user experience. > > On Mon, Jan 6, 2020 at 10:18 AM Zhenghua Gao

Re: Flink logging issue with logback

Hi Bajaj, Abhinav, Could you share the start-command of jobmanager and taskmanager. If it is started correctly, we will have a the following jvm options. -Dlog.file=/path/of/taskmanager.log -Dlogback.configurationFile=file:///path/of/logback.xml Best, Yang Bajaj, Abhinav 于2020年1月4日周六 上午7:23写

Re: [DISCUSS] Set default planner for SQL Client to Blink planner in 1.10 release

+1. I think it improves user experience. On Mon, Jan 6, 2020 at 10:18 AM Zhenghua Gao wrote: > +1 for making blink planner as the default planner for SQL Client since we > have made a huge improvement in 1.10. > > *Best Regards,* > *Zhenghua Gao* > > > On Sun, Jan 5, 2020 at 2:42 PM Benchao Li

Re: How long Flink state default TTL，if I don't config the state ttl config?

Yes. State TTL is by default disabled. Thanks, Zhu Zhu LakeShen 于2020年1月6日周一 上午10:09写道： > I saw the flink source code, I find the flink state ttl default is > never expire,is it right? > > LakeShen 于2020年1月6日周一 上午9:58写道： > >> Hi community，I have a question about flink state ttl.If I don't conf

Re: [DISCUSS] Set default planner for SQL Client to Blink planner in 1.10 release

+1 for making blink planner as the default planner for SQL Client since we have made a huge improvement in 1.10. *Best Regards,* *Zhenghua Gao* On Sun, Jan 5, 2020 at 2:42 PM Benchao Li wrote: > +1 > > We have used blink planner since 1.9.0 release in our production > environment, and it behav

Re: How long Flink state default TTL，if I don't config the state ttl config?

I saw the flink source code, I find the flink state ttl default is never expire,is it right? LakeShen 于2020年1月6日周一 上午9:58写道： > Hi community，I have a question about flink state ttl.If I don't config the > flink state ttl config, > How long the flink state retain？Is it forever retain in hdfs? > Th

How long Flink state default TTL，if I don't config the state ttl config?

Hi community，I have a question about flink state ttl.If I don't config the flink state ttl config, How long the flink state retain？Is it forever retain in hdfs? Thanks your replay.

Re: Controlling the Materialization of JOIN updates

Good to hear that the patch resolved your issue, looking forward to hearing more feedback from you! Best, Kurt On Mon, Jan 6, 2020 at 5:56 AM Benoît Paris < benoit.pa...@centraliens-lille.org> wrote: > Hi Kurt, > > Thank you for your answer. > > Yes both fact tables and dimension tables are cha

Re: Duplicate tasks for the same query

Another common skew case we've seen is null handling, the value of the join key is NULL. We will shuffle the NULL value into one task even if the join condition won't stand by definition. For DeDuplication, I just want to make sure this behavior meets your requirement. Because for some other usage

Re: Duplicate tasks for the same query

Hi Kurt, I understand what you mean, some userIds may appear more frequently than the others but this distribution doesn't look in proportionate with the data skew. Do you think of any other possible reasons or anything I can try out to investigate this more? For DeDuplication, I query for the la

Re: Controlling the Materialization of JOIN updates

Hi Kurt, Thank you for your answer. Yes both fact tables and dimension tables are changing over time; it was to illustrate that they could change at the same time but that we could still make a JOIN basically ignore updates from one specified side. The SQL is not the actual one I'm using, and as

Re: Stateful functions and modules

Ok, good. Thanks for your response. /Dan Den sön 5 jan. 2020 11:52Igal Shilman skrev: > Hi Dan, > > Having a class that defines only the function types indeed makes sense, > this would lower the coupling between the (maven) module that contains the > function implementation and the (maven) modu

Re: Stateful functions and modules

Hi Dan, Having a class that defines only the function types indeed makes sense, this would lower the coupling between the (maven) module that contains the function implementation and the (maven) module that uses it. You can peek here for example: https://github.com/ververica/stateful-functions/bl