Re: Flink Conf "yarn.flink-dist-jar" Question

Could you sync >>>>>>>> with Tison on FLINK-14964 on how to proceed. I'm not super deep into >>>>>>>> these >>>>>>>> issues but they seem to be somewhat related and Tison already did some >>>

Re: Flink Conf "yarn.flink-dist-jar" Question

;>>> >>>>>>>> Hi All, thanks a lot for reviving this discussion. >>>>>>>> >>>>>>>> I think we could unify the FLINK-13938 and FLINK-14964 since they >>>>>>>> have the similar >>>>

Re: Flink Conf "yarn.flink-dist-jar" Question

t;>>>>>> directory only, while >>>>>>> FLINK-14964 is trying to support arbitrary pre-uloaded >>>>>>> jars(including user and system jars). >>>>>>> >>>>>>> >>>>>>> So i suggest to d

Re: Flink Conf "yarn.flink-dist-jar" Question

;>> "hdfs://hdpdev/flink/release/flink-1.x" >>>>>> "hdfs://hdpdev/user/someone/mylib" >>>>>> 2. Use the -ypl argument to specify the shared lib, multiple >>>>>> directories could be specif

Re: Flink Conf "yarn.flink-dist-jar" Question

buted >>>>> cache in the YARN nodemanager could be reused by multiple >>>>> applications. This is to avoid >>>>> unnecessary downloading, especially for the "flink-dist-*.jar". For >>>>> the user shared lib, the >>>>> visib

Re: Flink Conf "yarn.flink-dist-jar" Question

d lib, the >>>> visibility is still set to "APPLICATION" level. >>>> >>>> >>>> For our past internal use case, the shared lib could help with >>>> accelerating the submission a lot. >>>> Also it helps to reduce the pressur

Re: Flink Conf "yarn.flink-dist-jar" Question

essure of HDFS when we want to launch many >>> applications together. >>> >>> @tison @Till Rohrmann @Hailu, Andreas >>> If you guys thinks the suggestion makes sense. I >>> will try to find some time to work on this and hope it could catch up >>

Re: Flink Conf "yarn.flink-dist-jar" Question

this and hope it could catch up >> with release-1.1 cycle. >> >> >> Best, >> Yang >> >> Hailu, Andreas [Engineering] 于2020年4月16日周四 >> 上午8:47写道： >> >>> Okay, I’ll continue to watch the JIRAs. Thanks for the update, Till. >>&

Re: Flink Conf "yarn.flink-dist-jar" Question

th > release-1.1 cycle. > > > Best, > Yang > > Hailu, Andreas [Engineering] 于2020年4月16日周四 > 上午8:47写道： > >> Okay, I’ll continue to watch the JIRAs. Thanks for the update, Till. >> >> >> >> *// *ah >> >> >> >> *From:* T

Re: Flink Conf "yarn.flink-dist-jar" Question

t; *To:* Hailu, Andreas [Engineering] > *Cc:* Yang Wang ; tison ; > user@flink.apache.org > *Subject:* Re: Flink Conf "yarn.flink-dist-jar" Question > > > > Hi Andreas, > > > > it looks as if FLINK-13938 and FLINK-14964 won't make it into th

RE: Flink Conf "yarn.flink-dist-jar" Question

Okay, I’ll continue to watch the JIRAs. Thanks for the update, Till. // ah From: Till Rohrmann Sent: Wednesday, April 15, 2020 10:51 AM To: Hailu, Andreas [Engineering] Cc: Yang Wang ; tison ; user@flink.apache.org Subject: Re: Flink Conf "yarn.flink-dist-jar" Question Hi Andreas,

Re: Flink Conf "yarn.flink-dist-jar" Question

l be in a 1.10.x version? > > > > *// *ah > > > > *From:* Hailu, Andreas [Engineering] > *Sent:* Friday, March 20, 2020 9:19 AM > *To:* 'Yang Wang' > *Cc:* tison ; user@flink.apache.org > *Subject:* RE: Flink Conf "yarn.flink-dist-jar" Question >

RE: Flink Conf "yarn.flink-dist-jar" Question

lt;mailto:user@flink.apache.org> Subject: Re: Flink Conf "yarn.flink-dist-jar" Question Hi Hailu, Sorry for the late response. If the Flink cluster(e.g. Yarn application) is stopped directly by `yarn application -kill`, then the staging directory will be left behind. Since the j

RE: Flink Conf "yarn.flink-dist-jar" Question

the two tickets. Thank you. // ah From: Yang Wang Sent: Monday, March 16, 2020 9:37 PM To: Hailu, Andreas [Engineering] Cc: tison ; user@flink.apache.org Subject: Re: Flink Conf "yarn.flink-dist-jar" Question Hi Hailu, Sorry for the late response. If the Flink cluster(e.g. Yarn a

Re: Flink Conf "yarn.flink-dist-jar" Question

ay, March 9, 2020 1:20 PM > *To:* 'Yang Wang' > *Cc:* tison ; user@flink.apache.org > *Subject:* RE: Flink Conf "yarn.flink-dist-jar" Question > > > > Hi Yang, > > > > Yes, a combination of these two would be very helpful for us. We have a > single

RE: Flink Conf "yarn.flink-dist-jar" Question

, Andreas [Engineering] Sent: Monday, March 9, 2020 1:20 PM To: 'Yang Wang' Cc: tison ; user@flink.apache.org Subject: RE: Flink Conf "yarn.flink-dist-jar" Question Hi Yang, Yes, a combination of these two would be very helpful for us. We have a single shaded binary which we use to run all o

RE: Flink Conf "yarn.flink-dist-jar" Question

a general public cache solution is what’s being called for? // ah From: Yang Wang Sent: Sunday, March 8, 2020 10:52 PM To: Hailu, Andreas [Engineering] Cc: tison ; user@flink.apache.org Subject: Re: Flink Conf "yarn.flink-dist-jar" Question Hi Hailu, tison, I created a very similar tic

Re: Flink Conf "yarn.flink-dist-jar" Question

t:* Friday, March 6, 2020 1:40 PM > *To:* Hailu, Andreas [Engineering] > *Cc:* user@flink.apache.org > *Subject:* Re: Flink Conf "yarn.flink-dist-jar" Question > > > > FLINK-13938 seems a bit different than your requirement. The one totally > matches

RE: Flink Conf "yarn.flink-dist-jar" Question

Hi Tison, thanks for the reply. I’ve replied to the ticket. I’ll be watching it as well. // ah From: tison Sent: Friday, March 6, 2020 1:40 PM To: Hailu, Andreas [Engineering] Cc: user@flink.apache.org Subject: Re: Flink Conf "yarn.flink-dist-jar" Question FLINK-13938 seems a bit

Re: Flink Conf "yarn.flink-dist-jar" Question

FLINK-13938 seems a bit different than your requirement. The one totally matches is FLINK-14964 . I'll appreciate it if you can share you opinion on the JIRA ticket. Best, tison. tison 于2020年3月7日周六 上午2:35写道： > Yes your requirement is exactly

Re: Flink Conf "yarn.flink-dist-jar" Question

Yes your requirement is exactly taken into consideration by the community. We currently have an open JIRA ticket for the specific feature[1] and works for loosing the constraint of flink-jar schema to support DFS location should happen. Best, tison. [1]