[
https://issues.apache.org/jira/browse/MAPREDUCE-7172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16719260#comment-16719260
]
Wangda Tan commented on MAPREDUCE-7172:
---------------------------------------
Thanks [~templedf] for trying this.
Per my understanding, the code flow is:
Inside org.apache.hadoop.mapreduce.JobResourceUploader#uploadLibJars
1) Mkdir for the job submission libjar dir (on remote FS path):
{code}
Path libjarsDir = JobSubmissionFiles.getJobDistCacheLibjars(submitJobDir);
{code}
2) Copy remote files:
{code}
if (newURI == null) {
Path newPath =
copyRemoteFiles(libjarsDir, tmp, conf, submitReplication);
{code}
Important: shared cache libjar should be disabled:
{code}
if (scConfig.isSharedCacheLibjarsEnabled()) {
{code}
3) Inside copyRemoteFiles:
{code}
if (FileUtil.compareFs(remoteFs, jtFs)) {
return originalPath;
}
{code}
If remoteFS == jtFS, nothing will be copied. And returns originalPath. (So the
returned newPath is same as originalPath).
4) Call addFileToClasspath, but last parameter will be false. (So file will be
added to classpath, but won't be downloaded by NM).
{code}
DistributedCache.addFileToClassPath(new Path(newURI.getPath()), conf,
jtFs, false);
{code}
5) When wildcard is enabled, and no fragment (please make sure no fragment).
{code}
if (useWildcard && !foundFragment) {
// Add the whole directory to the cache using a wild card
Path libJarsDirWildcard =
jtFs.makeQualified(new Path(libjarsDir, DistributedCache.WILDCARD));
DistributedCache.addCacheFile(libJarsDirWildcard.toUri(), conf);
}
{code}
The whole remote libjar directory will be localized. But because of 3), files
won't be copied to remote libjar.
I will try to spend some time today to reproduce this issue on a local cluster,
if you have time, could u check my comment as well? :).
> Wildcard functionality of -libjar is broken when jars are located in same
> remote FS
> -----------------------------------------------------------------------------------
>
> Key: MAPREDUCE-7172
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7172
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Wangda Tan
> Priority: Critical
>
> We recently found that when -libjar specified jars on the same remote FS,
> jars will not be properly added to classpath.
> The reason is MAPREDUCE-6719 added the wildcard functionality, but the follow
> logic assumes files are all placed under job's submission directory. (Inside
> JobResourceUploader)
> {code:java}
> if (useWildcard && !foundFragment) {
> // Add the whole directory to the cache using a wild card
> Path libJarsDirWildcard =
> jtFs.makeQualified(new Path(libjarsDir, DistributedCache.WILDCARD));
> DistributedCache.addCacheFile(libJarsDirWildcard.toUri(), conf);
> }{code}
> However, in the same method, specified resources will be only uploaded when
> two FSes are different, see copyRemoteFiles:
> {code:java}
> if (FileUtil.compareFs(remoteFs, jtFs)) {
> return originalPath;
> } {code}
> Workaround of this issue is pass:
> mapreduce.client.libjars.wildcard = false.
> When the MR job got launched.
> Example commandline to reproduce this issue is:
> {code:java}
> hadoop jar abc.jar org.ABC -libjars
> "wasb://host/path1/jar1,wasb://host/path2/jar2..."{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
