Hi tison, Thanks for your starting this discussion. * For user customized flink-dist jar, it is an useful feature. Since it could avoid to upload the flink-dist jar every time. Especially in production environment, it could accelerate the submission process. * For the standard flink-dist jar, FLINK-13938[1] could solve the problem.Upload a official flink release binary to distributed storage(hdfs) first, and then all the submission could benefit from it. Users could also upload the customized flink-dist jar to accelerate their submission.
If the flink-dist jar could be specified to a remote path, maybe the user jar have the same situation. [1]. https://issues.apache.org/jira/browse/FLINK-13938 tison <wander4...@gmail.com> 于2019年11月19日周二 上午11:17写道： > Hi forks, > > Recently, our customers ask for a feature configuring remote flink jar. > I'd like to reach to you guys > to see whether or not it is a general need. > > ATM Flink only supports configures local file as flink jar via `-yj` > option. If we pass a HDFS file > path, due to implementation detail it will fail with > IllegalArgumentException. In the story we support > configure remote flink jar, this limitation is eliminated. We also make > use of YARN locality so that > reducing uploading overhead, instead, asking YARN to localize the jar on > AM container started. > > Besides, it possibly has overlap with FLINK-13938. I'd like to put the > discussion on our > mailing list first. > > Are you looking forward to such a feature? > > @Yang Wang: this feature is different from that we discussed offline, it > only focuses on flink jar, not > all ship files. > > Best, > tison. >
