[
https://issues.apache.org/jira/browse/HADOOP-9639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13681790#comment-13681790
]
Jason Lowe commented on HADOOP-9639:
------------------------------------
bq. The way MR currently uses it is that it sets up these files under each
job's staging directory, so it's not ready for sharing.
job jars are not required to be uploaded to the staging directory, see
JobSubmitter.copyAndConfigureFiles. If the job jar being submitted looks like
it's already on the same filesystem as the staging directory then it will not
upload it to staging since it's already accessible by the nodes for
localization. Yes, many jobs simply call setJarByClass which will always find
a local filesystem jar and therefore always gets uploaded. However a client
can configure the job jar to point to a jar already in HDFS (or even specify no
jar, implying the code is already sitting in the classpath on all the nodes).
bq. However, the reality is that our MR jobs are a combination of all types of
different MR apps (pig, scalding, handwritten MR apps, ...). Every framework
would need to be retrofitted to do this for it to be useful. While it is not
impossible, it may not be the most effective way.
Yes, it's not a turnkey solution given most jobs want to submit local files by
default. However I think there will be a number of issues trying to make this a
generalized solution via a couple of config options for a job. As you mention,
it will be easy to trash public files with the wrong versions (think two or
more versions of pig running on the cluster, for example). Seems a lot simpler
to target some main offenders (i.e.: frameworks like pig, scalding, etc. and
large custom jobs with big dependencies shipped with the job). The changes
there are probably going to be very straightforward -- e.g.: add a pig-specific
config to tell pig where it can pick up the pig jar from HDFS rather than
always shipping it with each job, etc. Then we can place the pig jar in HDFS
and set that config to avoid shipping the same pig jar for every job. Another
pig version? Point that config to a different pig jar in HDFS.
There might be a way to solve the problems you mentioned above in some general
way, but I think it will complicated. Discerning the proper public/private
intent from local files submitted to the dist cache is going to be tricky to do
automatically if it can be done at all, combined with the whole management
issues of when to clobber/purge the retention area in HDFS.
So while we're hashing out if and how this could be done at the MR framework
level, your custom jobs with large jars should be able to reap the desired
performance benefits with relatively small changes to their job setup code.
> truly shared cache for jars (jobjar/libjar)
> -------------------------------------------
>
> Key: HADOOP-9639
> URL: https://issues.apache.org/jira/browse/HADOOP-9639
> Project: Hadoop Common
> Issue Type: New Feature
> Components: filecache
> Affects Versions: 2.0.4-alpha
> Reporter: Sangjin Lee
>
> Currently there is the distributed cache that enables you to cache jars and
> files so that attempts from the same job can reuse them. However, sharing is
> limited with the distributed cache because it is normally on a per-job basis.
> On a large cluster, sometimes copying of jobjars and libjars becomes so
> prevalent that it consumes a large portion of the network bandwidth, not to
> speak of defeating the purpose of "bringing compute to where data is". This
> is wasteful because in most cases code doesn't change much across many jobs.
> I'd like to propose and discuss feasibility of introducing a truly shared
> cache so that multiple jobs from multiple users can share and cache jars.
> This JIRA is to open the discussion.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
