[
https://issues.apache.org/jira/browse/MAPREDUCE-4421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13410390#comment-13410390
]
Kihwal Lee commented on MAPREDUCE-4421:
---------------------------------------
bq. How do we handle the native lib dependencies though? Do we ship that as
well, or keep a static resource at the NM?
There are two parts to this issue:
# Specifying dependency: App-level dependencies on native libs should be
specified within the app, not by YARN. Apps can also allow each job to specify
additional dependencies. A proper merging of LD_LIBRARY_PATH from job, app and
hadoop must be done. (was -Djava.library.path in mrv1) Who merges what needs to
be made clear (a section in the app writer's guide?).
# Making libs available: The simplest way is to let the app ship them for each
job. But admins may choose to host app-level dependencies (in a multiversion
aware manner) and even some of popular job-level dependencies. YARN should
never automatically shove everything to apps. There are pros and cons in both
approaches. A well designed app will support both.
YARN should not remove or override legitimate app/job-level dependencies. I
think YARN already satisfies this, but there might be some areas that need
improvement. We should also provide a clear guide on how to manage dependencies
for app writers and admins.
This is quite similar to jar dependency management (this jira) in principle.
> Remove dependency on deployed MR jars
> -------------------------------------
>
> Key: MAPREDUCE-4421
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4421
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 2.0.0-alpha
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
>
> Currently MR AM depends on MR jars being deployed on all nodes via implicit
> dependency on YARN_APPLICATION_CLASSPATH.
> We should stop adding mapreduce jars to YARN_APPLICATION_CLASSPATH and,
> probably, just rely on adding a shaded MR jar along with job.jar to the
> dist-cache.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
