[jira] [Commented] (MAPREDUCE-6027) mr jobs with relative paths can fail
[
https://issues.apache.org/jira/browse/MAPREDUCE-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15070367#comment-15070367
]
Hadoop QA commented on MAPREDUCE-6027:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s {color}
| {color:red} MAPREDUCE-6027 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12660521/MAPREDUCE-6027.patch |
| JIRA Issue | MAPREDUCE-6027 |
| Powered by | Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6230/console |
This message was automatically generated.
> mr jobs with relative paths can fail
>
>
> Key: MAPREDUCE-6027
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6027
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission
>Reporter: Wing Yew Poon
>Assignee: Wing Yew Poon
> Labels: BB2015-05-TBR
> Attachments: MAPREDUCE-6027.patch
>
>
> I built hadoop from branch-2 and tried to run terasort as follows:
> {noformat}
> wypoon$ bin/hadoop jar
> share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-SNAPSHOT.jar terasort
> sort-input sort-output
> 14/08/07 08:57:55 INFO terasort.TeraSort: starting
> 2014-08-07 08:57:56.229 java[36572:1903] Unable to load realm info from
> SCDynamicStore
> 14/08/07 08:57:56 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/08/07 08:57:57 INFO input.FileInputFormat: Total input paths to process : 2
> Spent 156ms computing base-splits.
> Spent 2ms computing TeraScheduler splits.
> Computing input splits took 159ms
> Sampling 2 splits of 2
> Making 1 from 10 sampled records
> Computing parititions took 626ms
> Spent 789ms computing partitions.
> 14/08/07 08:57:57 INFO client.RMProxy: Connecting to ResourceManager at
> localhost/127.0.0.1:8032
> 14/08/07 08:57:58 INFO mapreduce.JobSubmitter: Cleaning up the staging area
> /tmp/hadoop-yarn/staging/wypoon/.staging/job_1407426900134_0001
> java.lang.IllegalArgumentException: Can not create a Path from an empty URI
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:140)
> at org.apache.hadoop.fs.Path.(Path.java:192)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.checkPermissionOfOther(ClientDistributedCacheManager.java:275)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.ancestorsHaveExecutePermissions(ClientDistributedCacheManager.java:256)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.isPublic(ClientDistributedCacheManager.java:243)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineCacheVisibilities(ClientDistributedCacheManager.java:162)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:58)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:265)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:301)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:389)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
> at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:316)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:325)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDesc
[jira] [Commented] (MAPREDUCE-6027) mr jobs with relative paths can fail
[
https://issues.apache.org/jira/browse/MAPREDUCE-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14607526#comment-14607526
]
Hadoop QA commented on MAPREDUCE-6027:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 1s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12660521/MAPREDUCE-6027.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / d3797f9 |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5860/console |
This message was automatically generated.
> mr jobs with relative paths can fail
>
>
> Key: MAPREDUCE-6027
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6027
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission
>Reporter: Wing Yew Poon
>Assignee: Wing Yew Poon
> Labels: BB2015-05-TBR
> Attachments: MAPREDUCE-6027.patch
>
>
> I built hadoop from branch-2 and tried to run terasort as follows:
> {noformat}
> wypoon$ bin/hadoop jar
> share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-SNAPSHOT.jar terasort
> sort-input sort-output
> 14/08/07 08:57:55 INFO terasort.TeraSort: starting
> 2014-08-07 08:57:56.229 java[36572:1903] Unable to load realm info from
> SCDynamicStore
> 14/08/07 08:57:56 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/08/07 08:57:57 INFO input.FileInputFormat: Total input paths to process : 2
> Spent 156ms computing base-splits.
> Spent 2ms computing TeraScheduler splits.
> Computing input splits took 159ms
> Sampling 2 splits of 2
> Making 1 from 10 sampled records
> Computing parititions took 626ms
> Spent 789ms computing partitions.
> 14/08/07 08:57:57 INFO client.RMProxy: Connecting to ResourceManager at
> localhost/127.0.0.1:8032
> 14/08/07 08:57:58 INFO mapreduce.JobSubmitter: Cleaning up the staging area
> /tmp/hadoop-yarn/staging/wypoon/.staging/job_1407426900134_0001
> java.lang.IllegalArgumentException: Can not create a Path from an empty URI
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:140)
> at org.apache.hadoop.fs.Path.(Path.java:192)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.checkPermissionOfOther(ClientDistributedCacheManager.java:275)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.ancestorsHaveExecutePermissions(ClientDistributedCacheManager.java:256)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.isPublic(ClientDistributedCacheManager.java:243)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineCacheVisibilities(ClientDistributedCacheManager.java:162)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:58)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:265)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:301)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:389)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
> at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:316)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:325)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145)
> at org.apache.hadoo
[jira] [Commented] (MAPREDUCE-6027) mr jobs with relative paths can fail
[
https://issues.apache.org/jira/browse/MAPREDUCE-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14525576#comment-14525576
]
Hadoop QA commented on MAPREDUCE-6027:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 38s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 30s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 37s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 27s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 2 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 13s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 1m 35s | Tests passed in
hadoop-mapreduce-client-core. |
| | | 38m 31s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12660521/MAPREDUCE-6027.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 6ae2a0d |
| whitespace |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5612/artifact/patchprocess/whitespace.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5612/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5612/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5612/console |
This message was automatically generated.
> mr jobs with relative paths can fail
>
>
> Key: MAPREDUCE-6027
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6027
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission
>Reporter: Wing Yew Poon
>Assignee: Wing Yew Poon
> Attachments: MAPREDUCE-6027.patch
>
>
> I built hadoop from branch-2 and tried to run terasort as follows:
> {noformat}
> wypoon$ bin/hadoop jar
> share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-SNAPSHOT.jar terasort
> sort-input sort-output
> 14/08/07 08:57:55 INFO terasort.TeraSort: starting
> 2014-08-07 08:57:56.229 java[36572:1903] Unable to load realm info from
> SCDynamicStore
> 14/08/07 08:57:56 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/08/07 08:57:57 INFO input.FileInputFormat: Total input paths to process : 2
> Spent 156ms computing base-splits.
> Spent 2ms computing TeraScheduler splits.
> Computing input splits took 159ms
> Sampling 2 splits of 2
> Making 1 from 10 sampled records
> Computing parititions took 626ms
> Spent 789ms computing partitions.
> 14/08/07 08:57:57 INFO client.RMProxy: Connecting to ResourceManager at
> localhost/127.0.0.1:8032
> 14/08/07 08:57:58 INFO mapreduce.JobSubmitter: Cleaning up the staging area
> /tmp/hadoop-yarn/staging/wypoon/.staging/job_1407426900134_0001
> java.lang.IllegalArgumentException: Can not create a Path from an empty URI
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:140)
> at org.apache.hadoop.fs.Path.(Path.java:192)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.checkPermissionOfOther(ClientDistributedCacheManager.java:275)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.ancestorsHaveExecutePermissions(ClientDistributedCacheManager.java:256)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.isPublic(ClientDistributedCacheManager.java:243)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineCacheVisibilities(ClientDistributedCacheManager.java:162)
> at
> org.apache.hadoop.mapred
[jira] [Commented] (MAPREDUCE-6027) mr jobs with relative paths can fail
[
https://issues.apache.org/jira/browse/MAPREDUCE-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093058#comment-14093058
]
Wing Yew Poon commented on MAPREDUCE-6027:
--
Karthik, it is possible I misread your first point. I'm not sure what you
intended then.
> mr jobs with relative paths can fail
>
>
> Key: MAPREDUCE-6027
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6027
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission
>Reporter: Wing Yew Poon
>Assignee: Wing Yew Poon
> Attachments: MAPREDUCE-6027.patch
>
>
> I built hadoop from branch-2 and tried to run terasort as follows:
> {noformat}
> wypoon$ bin/hadoop jar
> share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-SNAPSHOT.jar terasort
> sort-input sort-output
> 14/08/07 08:57:55 INFO terasort.TeraSort: starting
> 2014-08-07 08:57:56.229 java[36572:1903] Unable to load realm info from
> SCDynamicStore
> 14/08/07 08:57:56 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/08/07 08:57:57 INFO input.FileInputFormat: Total input paths to process : 2
> Spent 156ms computing base-splits.
> Spent 2ms computing TeraScheduler splits.
> Computing input splits took 159ms
> Sampling 2 splits of 2
> Making 1 from 10 sampled records
> Computing parititions took 626ms
> Spent 789ms computing partitions.
> 14/08/07 08:57:57 INFO client.RMProxy: Connecting to ResourceManager at
> localhost/127.0.0.1:8032
> 14/08/07 08:57:58 INFO mapreduce.JobSubmitter: Cleaning up the staging area
> /tmp/hadoop-yarn/staging/wypoon/.staging/job_1407426900134_0001
> java.lang.IllegalArgumentException: Can not create a Path from an empty URI
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:140)
> at org.apache.hadoop.fs.Path.(Path.java:192)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.checkPermissionOfOther(ClientDistributedCacheManager.java:275)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.ancestorsHaveExecutePermissions(ClientDistributedCacheManager.java:256)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.isPublic(ClientDistributedCacheManager.java:243)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineCacheVisibilities(ClientDistributedCacheManager.java:162)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:58)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:265)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:301)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:389)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
> at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:316)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:325)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145)
> at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> {noformat}
> If I used absolute paths for the input and out di
[jira] [Commented] (MAPREDUCE-6027) mr jobs with relative paths can fail
[
https://issues.apache.org/jira/browse/MAPREDUCE-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14093050#comment-14093050
]
Wing Yew Poon commented on MAPREDUCE-6027:
--
Karthik,
in ClientDistributedCacheManager.java before my change. there was a
getFileStatus static method with signature getFileStatus(Configuration, URI,
Map). It was called in one method, determineTimestamps (where
it's called 4 times) and it called getFileStatus(FileSystem, URI, Map) to do the work. In addition, getFileStatus(FileSystem, URI,
Map) was also called by checkPermissionOfOther (once).
In determineTimestamps, there are only URIs, not Paths. So it is not natural to
convert the URIs to Paths (by doing new Path(uri)) and then call a
getFileStatus method with a Path parameter, where it then converts the Path
back to a URI (path.toUri()) before looking it up in the cache. In
checkPermissionOfOther, however, there is a Path to begin with, hence it made
sense to call a getFileStatus method with a Path parameter.
I thought hard about how to make my change without duplicating code, but I
didn't see any easy way to do it.
I can implement your suggestion, so that getFileStatus(Configuration, URI,
Map) calls getFileStatus(FileSystem, Path, Map) - getting rid of getFileStatus(FileSystem, URI, Map) - but that means converting URIs to Paths and then back to URIs
for the cache lookup.
As to your second point, in practice, the way the project is built (from
maven), TEST_ROOT_DIR is always CURRENT_DIR + "target/test-dir/". So,
secondCacheFile is always a relative Path,. In case the test is run
differently, e.g., the system property test.build.data is not set by the build
(so defaults to /tmp) or set to something else, a valid secondCacheFile is
still constructed, and both the tests are still valid; they just won't test a
relative Path. I didn't want to just do
{noformat}
secondCacheFile = new Path("secondcachefile");
{noformat}
as I didn't want to pollute the project directory.
> mr jobs with relative paths can fail
>
>
> Key: MAPREDUCE-6027
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6027
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission
>Reporter: Wing Yew Poon
>Assignee: Wing Yew Poon
> Attachments: MAPREDUCE-6027.patch
>
>
> I built hadoop from branch-2 and tried to run terasort as follows:
> {noformat}
> wypoon$ bin/hadoop jar
> share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-SNAPSHOT.jar terasort
> sort-input sort-output
> 14/08/07 08:57:55 INFO terasort.TeraSort: starting
> 2014-08-07 08:57:56.229 java[36572:1903] Unable to load realm info from
> SCDynamicStore
> 14/08/07 08:57:56 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/08/07 08:57:57 INFO input.FileInputFormat: Total input paths to process : 2
> Spent 156ms computing base-splits.
> Spent 2ms computing TeraScheduler splits.
> Computing input splits took 159ms
> Sampling 2 splits of 2
> Making 1 from 10 sampled records
> Computing parititions took 626ms
> Spent 789ms computing partitions.
> 14/08/07 08:57:57 INFO client.RMProxy: Connecting to ResourceManager at
> localhost/127.0.0.1:8032
> 14/08/07 08:57:58 INFO mapreduce.JobSubmitter: Cleaning up the staging area
> /tmp/hadoop-yarn/staging/wypoon/.staging/job_1407426900134_0001
> java.lang.IllegalArgumentException: Can not create a Path from an empty URI
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:140)
> at org.apache.hadoop.fs.Path.(Path.java:192)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.checkPermissionOfOther(ClientDistributedCacheManager.java:275)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.ancestorsHaveExecutePermissions(ClientDistributedCacheManager.java:256)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.isPublic(ClientDistributedCacheManager.java:243)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineCacheVisibilities(ClientDistributedCacheManager.java:162)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:58)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:265)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:301)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:389)
> at org.apache.h
[jira] [Commented] (MAPREDUCE-6027) mr jobs with relative paths can fail
[
https://issues.apache.org/jira/browse/MAPREDUCE-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14092213#comment-14092213
]
Karthik Kambatla commented on MAPREDUCE-6027:
-
Looked more closely this time. I propose we keep the test and a modified fix.
Comments on the patch:
# For the fix, don't think we need another method. Why not change the call to
getFileStatus as follows:
{code}
FileStatus status = getFileStatus(fs, path, statCache); // change the
second parameter to path.toUri()
{code}
# IIUC, the test verifies the behavior only for cases where TEST_ROOT_DIR
begins with CURRENT_DIR. In other cases, we don't verify the behavior we want
to verify. Is that right? If so, is there a way to verify it always? May be,
create a custom TEST_ROOT_DIR under CURRENT_DIR and delete it in a cleanup
method?
> mr jobs with relative paths can fail
>
>
> Key: MAPREDUCE-6027
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6027
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission
>Reporter: Wing Yew Poon
>Assignee: Wing Yew Poon
> Attachments: MAPREDUCE-6027.patch
>
>
> I built hadoop from branch-2 and tried to run terasort as follows:
> {noformat}
> wypoon$ bin/hadoop jar
> share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-SNAPSHOT.jar terasort
> sort-input sort-output
> 14/08/07 08:57:55 INFO terasort.TeraSort: starting
> 2014-08-07 08:57:56.229 java[36572:1903] Unable to load realm info from
> SCDynamicStore
> 14/08/07 08:57:56 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/08/07 08:57:57 INFO input.FileInputFormat: Total input paths to process : 2
> Spent 156ms computing base-splits.
> Spent 2ms computing TeraScheduler splits.
> Computing input splits took 159ms
> Sampling 2 splits of 2
> Making 1 from 10 sampled records
> Computing parititions took 626ms
> Spent 789ms computing partitions.
> 14/08/07 08:57:57 INFO client.RMProxy: Connecting to ResourceManager at
> localhost/127.0.0.1:8032
> 14/08/07 08:57:58 INFO mapreduce.JobSubmitter: Cleaning up the staging area
> /tmp/hadoop-yarn/staging/wypoon/.staging/job_1407426900134_0001
> java.lang.IllegalArgumentException: Can not create a Path from an empty URI
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:140)
> at org.apache.hadoop.fs.Path.(Path.java:192)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.checkPermissionOfOther(ClientDistributedCacheManager.java:275)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.ancestorsHaveExecutePermissions(ClientDistributedCacheManager.java:256)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.isPublic(ClientDistributedCacheManager.java:243)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineCacheVisibilities(ClientDistributedCacheManager.java:162)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:58)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:265)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:301)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:389)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
> at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:316)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:325)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDrive
[jira] [Commented] (MAPREDUCE-6027) mr jobs with relative paths can fail
[
https://issues.apache.org/jira/browse/MAPREDUCE-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14091131#comment-14091131
]
Wing Yew Poon commented on MAPREDUCE-6027:
--
Yes, the change to ClientDistributedCacheManager itself does not hurt anything.
Let me know if you'd like me to submit a new patch with just the test.
> mr jobs with relative paths can fail
>
>
> Key: MAPREDUCE-6027
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6027
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission
>Reporter: Wing Yew Poon
>Assignee: Wing Yew Poon
> Attachments: MAPREDUCE-6027.patch
>
>
> I built hadoop from branch-2 and tried to run terasort as follows:
> {noformat}
> wypoon$ bin/hadoop jar
> share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-SNAPSHOT.jar terasort
> sort-input sort-output
> 14/08/07 08:57:55 INFO terasort.TeraSort: starting
> 2014-08-07 08:57:56.229 java[36572:1903] Unable to load realm info from
> SCDynamicStore
> 14/08/07 08:57:56 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/08/07 08:57:57 INFO input.FileInputFormat: Total input paths to process : 2
> Spent 156ms computing base-splits.
> Spent 2ms computing TeraScheduler splits.
> Computing input splits took 159ms
> Sampling 2 splits of 2
> Making 1 from 10 sampled records
> Computing parititions took 626ms
> Spent 789ms computing partitions.
> 14/08/07 08:57:57 INFO client.RMProxy: Connecting to ResourceManager at
> localhost/127.0.0.1:8032
> 14/08/07 08:57:58 INFO mapreduce.JobSubmitter: Cleaning up the staging area
> /tmp/hadoop-yarn/staging/wypoon/.staging/job_1407426900134_0001
> java.lang.IllegalArgumentException: Can not create a Path from an empty URI
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:140)
> at org.apache.hadoop.fs.Path.(Path.java:192)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.checkPermissionOfOther(ClientDistributedCacheManager.java:275)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.ancestorsHaveExecutePermissions(ClientDistributedCacheManager.java:256)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.isPublic(ClientDistributedCacheManager.java:243)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineCacheVisibilities(ClientDistributedCacheManager.java:162)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:58)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:265)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:301)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:389)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
> at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:316)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:325)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145)
> at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> {
[jira] [Commented] (MAPREDUCE-6027) mr jobs with relative paths can fail
[
https://issues.apache.org/jira/browse/MAPREDUCE-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14091110#comment-14091110
]
Karthik Kambatla commented on MAPREDUCE-6027:
-
Agree that the test is useful. Looking at the patch again, I am actually
tempted to leave the fix in there as well since it is benign.
> mr jobs with relative paths can fail
>
>
> Key: MAPREDUCE-6027
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6027
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission
>Reporter: Wing Yew Poon
>Assignee: Wing Yew Poon
> Attachments: MAPREDUCE-6027.patch
>
>
> I built hadoop from branch-2 and tried to run terasort as follows:
> {noformat}
> wypoon$ bin/hadoop jar
> share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-SNAPSHOT.jar terasort
> sort-input sort-output
> 14/08/07 08:57:55 INFO terasort.TeraSort: starting
> 2014-08-07 08:57:56.229 java[36572:1903] Unable to load realm info from
> SCDynamicStore
> 14/08/07 08:57:56 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/08/07 08:57:57 INFO input.FileInputFormat: Total input paths to process : 2
> Spent 156ms computing base-splits.
> Spent 2ms computing TeraScheduler splits.
> Computing input splits took 159ms
> Sampling 2 splits of 2
> Making 1 from 10 sampled records
> Computing parititions took 626ms
> Spent 789ms computing partitions.
> 14/08/07 08:57:57 INFO client.RMProxy: Connecting to ResourceManager at
> localhost/127.0.0.1:8032
> 14/08/07 08:57:58 INFO mapreduce.JobSubmitter: Cleaning up the staging area
> /tmp/hadoop-yarn/staging/wypoon/.staging/job_1407426900134_0001
> java.lang.IllegalArgumentException: Can not create a Path from an empty URI
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:140)
> at org.apache.hadoop.fs.Path.(Path.java:192)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.checkPermissionOfOther(ClientDistributedCacheManager.java:275)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.ancestorsHaveExecutePermissions(ClientDistributedCacheManager.java:256)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.isPublic(ClientDistributedCacheManager.java:243)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineCacheVisibilities(ClientDistributedCacheManager.java:162)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:58)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:265)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:301)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:389)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
> at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:316)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:325)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145)
> at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> {noformat}
[jira] [Commented] (MAPREDUCE-6027) mr jobs with relative paths can fail
[
https://issues.apache.org/jira/browse/MAPREDUCE-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14091101#comment-14091101
]
Wing Yew Poon commented on MAPREDUCE-6027:
--
[~kkambatl], with the revert of HADOOP-10876, the change to
ClientDistributedCacheManager is no longer needed (although works fine with the
revert). However, the unit test could still be useful, since there was no test
for ClientDistributedCacheManager.determineCacheVisibilities() before; the test
is very simple and does not do much, but it could detect the problem caused by
something like HADOOP-10876. I can submit a new patch with just the test, if
you think that is still of interest.
> mr jobs with relative paths can fail
>
>
> Key: MAPREDUCE-6027
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6027
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission
>Reporter: Wing Yew Poon
>Assignee: Wing Yew Poon
> Attachments: MAPREDUCE-6027.patch
>
>
> I built hadoop from branch-2 and tried to run terasort as follows:
> {noformat}
> wypoon$ bin/hadoop jar
> share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-SNAPSHOT.jar terasort
> sort-input sort-output
> 14/08/07 08:57:55 INFO terasort.TeraSort: starting
> 2014-08-07 08:57:56.229 java[36572:1903] Unable to load realm info from
> SCDynamicStore
> 14/08/07 08:57:56 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/08/07 08:57:57 INFO input.FileInputFormat: Total input paths to process : 2
> Spent 156ms computing base-splits.
> Spent 2ms computing TeraScheduler splits.
> Computing input splits took 159ms
> Sampling 2 splits of 2
> Making 1 from 10 sampled records
> Computing parititions took 626ms
> Spent 789ms computing partitions.
> 14/08/07 08:57:57 INFO client.RMProxy: Connecting to ResourceManager at
> localhost/127.0.0.1:8032
> 14/08/07 08:57:58 INFO mapreduce.JobSubmitter: Cleaning up the staging area
> /tmp/hadoop-yarn/staging/wypoon/.staging/job_1407426900134_0001
> java.lang.IllegalArgumentException: Can not create a Path from an empty URI
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:140)
> at org.apache.hadoop.fs.Path.(Path.java:192)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.checkPermissionOfOther(ClientDistributedCacheManager.java:275)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.ancestorsHaveExecutePermissions(ClientDistributedCacheManager.java:256)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.isPublic(ClientDistributedCacheManager.java:243)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineCacheVisibilities(ClientDistributedCacheManager.java:162)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:58)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:265)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:301)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:389)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
> at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:316)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:325)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145)
> at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
> at sun.reflect.NativeMethodAccessor
[jira] [Commented] (MAPREDUCE-6027) mr jobs with relative paths can fail
[
https://issues.apache.org/jira/browse/MAPREDUCE-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090817#comment-14090817
]
Karthik Kambatla commented on MAPREDUCE-6027:
-
HADOOP-10876 is reverted.
> mr jobs with relative paths can fail
>
>
> Key: MAPREDUCE-6027
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6027
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission
>Reporter: Wing Yew Poon
>Assignee: Wing Yew Poon
> Attachments: MAPREDUCE-6027.patch
>
>
> I built hadoop from branch-2 and tried to run terasort as follows:
> {noformat}
> wypoon$ bin/hadoop jar
> share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-SNAPSHOT.jar terasort
> sort-input sort-output
> 14/08/07 08:57:55 INFO terasort.TeraSort: starting
> 2014-08-07 08:57:56.229 java[36572:1903] Unable to load realm info from
> SCDynamicStore
> 14/08/07 08:57:56 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/08/07 08:57:57 INFO input.FileInputFormat: Total input paths to process : 2
> Spent 156ms computing base-splits.
> Spent 2ms computing TeraScheduler splits.
> Computing input splits took 159ms
> Sampling 2 splits of 2
> Making 1 from 10 sampled records
> Computing parititions took 626ms
> Spent 789ms computing partitions.
> 14/08/07 08:57:57 INFO client.RMProxy: Connecting to ResourceManager at
> localhost/127.0.0.1:8032
> 14/08/07 08:57:58 INFO mapreduce.JobSubmitter: Cleaning up the staging area
> /tmp/hadoop-yarn/staging/wypoon/.staging/job_1407426900134_0001
> java.lang.IllegalArgumentException: Can not create a Path from an empty URI
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:140)
> at org.apache.hadoop.fs.Path.(Path.java:192)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.checkPermissionOfOther(ClientDistributedCacheManager.java:275)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.ancestorsHaveExecutePermissions(ClientDistributedCacheManager.java:256)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.isPublic(ClientDistributedCacheManager.java:243)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineCacheVisibilities(ClientDistributedCacheManager.java:162)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:58)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:265)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:301)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:389)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
> at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:316)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:325)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145)
> at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> {noformat}
> If I used absolute paths for the input and out directories, the job runs fine.
> This breakage is due to HAD
[jira] [Commented] (MAPREDUCE-6027) mr jobs with relative paths can fail
[
https://issues.apache.org/jira/browse/MAPREDUCE-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090325#comment-14090325
]
Hadoop QA commented on MAPREDUCE-6027:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660521/MAPREDUCE-6027.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4793//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4793//console
This message is automatically generated.
> mr jobs with relative paths can fail
>
>
> Key: MAPREDUCE-6027
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6027
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission
>Reporter: Wing Yew Poon
>Assignee: Wing Yew Poon
> Attachments: MAPREDUCE-6027.patch
>
>
> I built hadoop from branch-2 and tried to run terasort as follows:
> {noformat}
> wypoon$ bin/hadoop jar
> share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-SNAPSHOT.jar terasort
> sort-input sort-output
> 14/08/07 08:57:55 INFO terasort.TeraSort: starting
> 2014-08-07 08:57:56.229 java[36572:1903] Unable to load realm info from
> SCDynamicStore
> 14/08/07 08:57:56 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/08/07 08:57:57 INFO input.FileInputFormat: Total input paths to process : 2
> Spent 156ms computing base-splits.
> Spent 2ms computing TeraScheduler splits.
> Computing input splits took 159ms
> Sampling 2 splits of 2
> Making 1 from 10 sampled records
> Computing parititions took 626ms
> Spent 789ms computing partitions.
> 14/08/07 08:57:57 INFO client.RMProxy: Connecting to ResourceManager at
> localhost/127.0.0.1:8032
> 14/08/07 08:57:58 INFO mapreduce.JobSubmitter: Cleaning up the staging area
> /tmp/hadoop-yarn/staging/wypoon/.staging/job_1407426900134_0001
> java.lang.IllegalArgumentException: Can not create a Path from an empty URI
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:140)
> at org.apache.hadoop.fs.Path.(Path.java:192)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.checkPermissionOfOther(ClientDistributedCacheManager.java:275)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.ancestorsHaveExecutePermissions(ClientDistributedCacheManager.java:256)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.isPublic(ClientDistributedCacheManager.java:243)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineCacheVisibilities(ClientDistributedCacheManager.java:162)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:58)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:265)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:301)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:389)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303
[jira] [Commented] (MAPREDUCE-6027) mr jobs with relative paths can fail
[
https://issues.apache.org/jira/browse/MAPREDUCE-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090204#comment-14090204
]
Hadoop QA commented on MAPREDUCE-6027:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660521/MAPREDUCE-6027.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4792//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4792//console
This message is automatically generated.
> mr jobs with relative paths can fail
>
>
> Key: MAPREDUCE-6027
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6027
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission
>Reporter: Wing Yew Poon
>Assignee: Wing Yew Poon
> Attachments: MAPREDUCE-6027.patch
>
>
> I built hadoop from branch-2 and tried to run terasort as follows:
> {noformat}
> wypoon$ bin/hadoop jar
> share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-SNAPSHOT.jar terasort
> sort-input sort-output
> 14/08/07 08:57:55 INFO terasort.TeraSort: starting
> 2014-08-07 08:57:56.229 java[36572:1903] Unable to load realm info from
> SCDynamicStore
> 14/08/07 08:57:56 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/08/07 08:57:57 INFO input.FileInputFormat: Total input paths to process : 2
> Spent 156ms computing base-splits.
> Spent 2ms computing TeraScheduler splits.
> Computing input splits took 159ms
> Sampling 2 splits of 2
> Making 1 from 10 sampled records
> Computing parititions took 626ms
> Spent 789ms computing partitions.
> 14/08/07 08:57:57 INFO client.RMProxy: Connecting to ResourceManager at
> localhost/127.0.0.1:8032
> 14/08/07 08:57:58 INFO mapreduce.JobSubmitter: Cleaning up the staging area
> /tmp/hadoop-yarn/staging/wypoon/.staging/job_1407426900134_0001
> java.lang.IllegalArgumentException: Can not create a Path from an empty URI
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:140)
> at org.apache.hadoop.fs.Path.(Path.java:192)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.checkPermissionOfOther(ClientDistributedCacheManager.java:275)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.ancestorsHaveExecutePermissions(ClientDistributedCacheManager.java:256)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.isPublic(ClientDistributedCacheManager.java:243)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineCacheVisibilities(ClientDistributedCacheManager.java:162)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:58)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:265)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:301)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:389)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303
[jira] [Commented] (MAPREDUCE-6027) mr jobs with relative paths can fail
[
https://issues.apache.org/jira/browse/MAPREDUCE-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14090071#comment-14090071
]
Hadoop QA commented on MAPREDUCE-6027:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12660491/MAPREDUCE-6027.diff
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:
org.apache.hadoop.mapreduce.filecache.TestClientDistributedCacheManager
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4791//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4791//console
This message is automatically generated.
> mr jobs with relative paths can fail
>
>
> Key: MAPREDUCE-6027
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6027
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission
>Reporter: Wing Yew Poon
>Assignee: Wing Yew Poon
> Attachments: MAPREDUCE-6027.diff
>
>
> I built hadoop from branch-2 and tried to run terasort as follows:
> {noformat}
> wypoon$ bin/hadoop jar
> share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-SNAPSHOT.jar terasort
> sort-input sort-output
> 14/08/07 08:57:55 INFO terasort.TeraSort: starting
> 2014-08-07 08:57:56.229 java[36572:1903] Unable to load realm info from
> SCDynamicStore
> 14/08/07 08:57:56 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/08/07 08:57:57 INFO input.FileInputFormat: Total input paths to process : 2
> Spent 156ms computing base-splits.
> Spent 2ms computing TeraScheduler splits.
> Computing input splits took 159ms
> Sampling 2 splits of 2
> Making 1 from 10 sampled records
> Computing parititions took 626ms
> Spent 789ms computing partitions.
> 14/08/07 08:57:57 INFO client.RMProxy: Connecting to ResourceManager at
> localhost/127.0.0.1:8032
> 14/08/07 08:57:58 INFO mapreduce.JobSubmitter: Cleaning up the staging area
> /tmp/hadoop-yarn/staging/wypoon/.staging/job_1407426900134_0001
> java.lang.IllegalArgumentException: Can not create a Path from an empty URI
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:140)
> at org.apache.hadoop.fs.Path.(Path.java:192)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.checkPermissionOfOther(ClientDistributedCacheManager.java:275)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.ancestorsHaveExecutePermissions(ClientDistributedCacheManager.java:256)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.isPublic(ClientDistributedCacheManager.java:243)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineCacheVisibilities(ClientDistributedCacheManager.java:162)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:58)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:265)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:301)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:389)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at org.apache.hadoop.mapreduce.Job.subm
[jira] [Commented] (MAPREDUCE-6027) mr jobs with relative paths can fail
[
https://issues.apache.org/jira/browse/MAPREDUCE-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14089947#comment-14089947
]
Wing Yew Poon commented on MAPREDUCE-6027:
--
I also tested that submitting terasort with the relative paths works fine with
my change after rebuilding hadoop.
> mr jobs with relative paths can fail
>
>
> Key: MAPREDUCE-6027
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6027
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission
>Reporter: Wing Yew Poon
>Assignee: Wing Yew Poon
> Attachments: MAPREDUCE-6027.diff
>
>
> I built hadoop from branch-2 and tried to run terasort as follows:
> {noformat}
> wypoon$ bin/hadoop jar
> share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-SNAPSHOT.jar terasort
> sort-input sort-output
> 14/08/07 08:57:55 INFO terasort.TeraSort: starting
> 2014-08-07 08:57:56.229 java[36572:1903] Unable to load realm info from
> SCDynamicStore
> 14/08/07 08:57:56 WARN util.NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/08/07 08:57:57 INFO input.FileInputFormat: Total input paths to process : 2
> Spent 156ms computing base-splits.
> Spent 2ms computing TeraScheduler splits.
> Computing input splits took 159ms
> Sampling 2 splits of 2
> Making 1 from 10 sampled records
> Computing parititions took 626ms
> Spent 789ms computing partitions.
> 14/08/07 08:57:57 INFO client.RMProxy: Connecting to ResourceManager at
> localhost/127.0.0.1:8032
> 14/08/07 08:57:58 INFO mapreduce.JobSubmitter: Cleaning up the staging area
> /tmp/hadoop-yarn/staging/wypoon/.staging/job_1407426900134_0001
> java.lang.IllegalArgumentException: Can not create a Path from an empty URI
> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:140)
> at org.apache.hadoop.fs.Path.(Path.java:192)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.checkPermissionOfOther(ClientDistributedCacheManager.java:275)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.ancestorsHaveExecutePermissions(ClientDistributedCacheManager.java:256)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.isPublic(ClientDistributedCacheManager.java:243)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineCacheVisibilities(ClientDistributedCacheManager.java:162)
> at
> org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:58)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:265)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:301)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:389)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
> at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:316)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:325)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145)
> at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> {noformat}
> If I used absolute paths f
