There is a related use case (not specific to HDFS) that I came across: It would be nice if the jar upload endpoint could accept the URL of a jar file as alternative to the jar file itself. Such URL could point to an artifactory or distributed file system.
Thomas On Mon, Nov 18, 2019 at 7:40 PM Yang Wang <danrtsey...@gmail.com> wrote: > Hi tison, > > Thanks for your starting this discussion. > * For user customized flink-dist jar, it is an useful feature. Since it > could avoid to upload the flink-dist jar > every time. Especially in production environment, it could accelerate the > submission process. > * For the standard flink-dist jar, FLINK-13938[1] could solve > the problem.Upload a official flink release > binary to distributed storage(hdfs) first, and then all the submission > could benefit from it. Users could > also upload the customized flink-dist jar to accelerate their submission. > > If the flink-dist jar could be specified to a remote path, maybe the user > jar have the same situation. > > [1]. https://issues.apache.org/jira/browse/FLINK-13938 > > tison <wander4...@gmail.com> 于2019年11月19日周二 上午11:17写道： > > > Hi forks, > > > > Recently, our customers ask for a feature configuring remote flink jar. > > I'd like to reach to you guys > > to see whether or not it is a general need. > > > > ATM Flink only supports configures local file as flink jar via `-yj` > > option. If we pass a HDFS file > > path, due to implementation detail it will fail with > > IllegalArgumentException. In the story we support > > configure remote flink jar, this limitation is eliminated. We also make > > use of YARN locality so that > > reducing uploading overhead, instead, asking YARN to localize the jar on > > AM container started. > > > > Besides, it possibly has overlap with FLINK-13938. I'd like to put the > > discussion on our > > mailing list first. > > > > Are you looking forward to such a feature? > > > > @Yang Wang: this feature is different from that we discussed offline, it > > only focuses on flink jar, not > > all ship files. > > > > Best, > > tison. > > >
