[
https://issues.apache.org/jira/browse/HBASE-10830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13948652#comment-13948652
]
stack commented on HBASE-10830:
-------------------------------
Rat hole.
The jobtracker filesystem is the localfs so comes time to figure whether to
copy up the supporting jars in the JobSubmitter, we think the target and the
source filesystems the same so we do not copy them (even though 'default' fs is
the mini hdfs this test sets up). The jobtracker picks up its fs from the
setting of mapred.system.dir. Making mapred.system.dir be up in mini hdfs then
has this issue whenever we ask Cluster its fs:
{code}
java.lang.IllegalArgumentException: Wrong FS:
hdfs://localhost:50007/mapred.system.dir, expected: file:///
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:642)
at org.apache.hadoop.fs.FileSystem.makeQualified(FileSystem.java:462)
at
org.apache.hadoop.fs.FilterFileSystem.makeQualified(FilterFileSystem.java:114)
at
org.apache.hadoop.mapred.LocalJobRunner.getSystemDir(LocalJobRunner.java:741)
at org.apache.hadoop.mapreduce.Cluster$1.run(Cluster.java:163)
at org.apache.hadoop.mapreduce.Cluster$1.run(Cluster.java:161)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapreduce.Cluster.getFileSystem(Cluster.java:161)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1264)
...
{code}
If I banged my head harder I should be able to figure out some workaround but
putting aside for a now so maybe a [~ndimiduk] or someone else who has been in
here of late has chance to give input if they have any.
> Integration test MR jobs attempt to load htrace jars from the wrong location
> ----------------------------------------------------------------------------
>
> Key: HBASE-10830
> URL: https://issues.apache.org/jira/browse/HBASE-10830
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.98.1
> Reporter: Andrew Purtell
> Priority: Minor
> Fix For: 0.99.0, 0.98.2
>
>
> The MapReduce jobs submitted by IntegrationTestImportTsv want to load the
> htrace JAR from the local Maven cache but get confused and use a HDFS URI.
> {noformat}
> Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 8.489 sec <<<
> FAILURE!
> testGenerateAndLoad(org.apache.hadoop.hbase.mapreduce.IntegrationTestImportTsv)
> Time elapsed: 0.488 sec <<< ERROR!
> java.io.FileNotFoundException: File does not exist:
> hdfs://localhost:37548/home/apurtell/.m2/repository/org/cloudera/htrace/htrace-core/2.04/htrace-core-2.04.jar
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1110)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:224)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:93)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:57)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:264)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:300)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:387)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1286)
> at org.apache.hadoop.hbase.mapreduce.ImportTsv.run(ImportTsv.java:603)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.mapreduce.TestImportTsv.doMROnTableTest(TestImportTsv.java:270)
> at
> org.apache.hadoop.hbase.mapreduce.TestImportTsv.doMROnTableTest(TestImportTsv.java:232)
> at
> org.apache.hadoop.hbase.mapreduce.IntegrationTestImportTsv.testGenerateAndLoad(IntegrationTestImportTsv.java:206)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
