[
https://issues.apache.org/jira/browse/FLINK-18362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17146080#comment-17146080
]
gramo lee edited comment on FLINK-18362 at 6/26/20, 7:15 AM:
-------------------------------------------------------------
[~fly_in_gis] My company policy review was just passed . So now the
patch[^support-yarn.ship-archives.patch]) is able to be attached. As you
mentioned, "yarn.ship-archives" is added as an option for this. at first, my
code had been based on flink 1.10.1 release. After that, I realized an
implementation of "YarnLocalResourceDescriptor" is already existed on master
branch(afebdc2f19a7e54). Therefore it took more time to reimplement by using
it, but anyway it made the code simpler than before. Please give me your review
about the code.
was (Author: gramo):
My company policy review was just passed . So now the
patch[^support-yarn.ship-archives.patch]) is able to be attached. As you
mentioned, "yarn.ship-archives" is added as an option for this. at first, my
code had been based on flink 1.10.1 release. After that, I realized an
implementation of "YarnLocalResourceDescriptor" is already existed on master
branch(afebdc2f19a7e54). Therefore it took more time to reimplement by using
it, but anyway it made the code simpler than before. Please give me your review
about the code.
> Shipping jdk by shipping archive
> --------------------------------
>
> Key: FLINK-18362
> URL: https://issues.apache.org/jira/browse/FLINK-18362
> Project: Flink
> Issue Type: Wish
> Affects Versions: 1.10.1
> Reporter: Noah
> Priority: Minor
> Attachments: support-yarn.ship-archives.patch
>
>
> Hello,
> Our team are running flink cluster on YARN, and it works so well 👍
> h4. Functional requirements
> Is there any option to ship archive to YARN applications?
> h4. Backgrounds
> Recently, one of job has been shut down with jdk8 version related issues.
> https://bugs.java.com/bugdatabase/view_bug.do?bug_id=6675699
> It's easy problem if we could set latest jdk on
> `containerized.taskmanager.env.JAVA_HOME`.
> However, cluster administrator said it's difficult to install the latest jdk
> on all cluster machines.
>
> So, we planned to run a job on latest jdk that is shipped via shared
> resources.
> There's an option `yarn.ship-directories` but it's quite slow because jdk has
> large number of files.
> If Flink supports to ship archive such as `yarn.ship-archive`, we can ship
> jdk archive to remote machines and use shipped jdk location as JAVA_HOME
> (using `yarn.container-start-command-template` )
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
