[jira] [Commented] (HADOOP-9865) FileContext.globStatus() has a regression with respect to relative path
[
https://issues.apache.org/jira/browse/HADOOP-9865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13742092#comment-13742092
]
Hudson commented on HADOOP-9865:
SUCCESS: Integrated in Hadoop-Yarn-trunk #303 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/303/])
HADOOP-9865. FileContext#globStatus has a regression with respect to relative
path. (Contributed by Chaun Lin) (cmccabe:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1514531)
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Globber.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestGlobPaths.java
FileContext.globStatus() has a regression with respect to relative path
---
Key: HADOOP-9865
URL: https://issues.apache.org/jira/browse/HADOOP-9865
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.0.0, 2.3.0
Reporter: Chuan Liu
Assignee: Chuan Liu
Fix For: 2.3.0
Attachments: HADOOP-9865-demo.patch, HADOOP-9865-trunk.2.patch,
HADOOP-9865-trunk.3.patch, HADOOP-9865-trunk.patch
I discovered the problem when running unit test TestMRJobClient on Windows.
The cause is indirect in this case. In the unit test, we try to launch a job
and list its status. The job failed, and caused the list command get a result
of 0, which triggered the unit test assert. From the log and debug, the job
failed because we failed to create the Jar with classpath (see code around
{{FileUtil.createJarWithClassPath}}) in {{ContainerLaunch}}. This is a
Windows specific step right now; so the test still passes on Linux. This step
failed because we passed in a relative path to {{FileContext.globStatus()}}
in {{FileUtil.createJarWithClassPath}}. The relevant log looks like the
following.
{noformat}
2013-08-12 16:12:05,937 WARN [ContainersLauncher #0]
launcher.ContainerLaunch (ContainerLaunch.java:call(270)) - Failed to launch
container.
org.apache.hadoop.HadoopIllegalArgumentException: Path is relative
at org.apache.hadoop.fs.Path.checkNotRelative(Path.java:74)
at org.apache.hadoop.fs.FileContext.getFSofPath(FileContext.java:304)
at org.apache.hadoop.fs.Globber.schemeFromPath(Globber.java:107)
at org.apache.hadoop.fs.Globber.glob(Globber.java:128)
at
org.apache.hadoop.fs.FileContext$Util.globStatus(FileContext.java:1908)
at
org.apache.hadoop.fs.FileUtil.createJarWithClassPath(FileUtil.java:1247)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.sanitizeEnv(ContainerLaunch.java:679)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:232)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:1)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}
I think this is a regression from HADOOP-9817. I modified some code and the
unit test passed. (See the attached patch.) However, I think the impact is
larger. I will add some unit tests to verify the behavior, and work on a more
complete fix.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9865) FileContext.globStatus() has a regression with respect to relative path
[
https://issues.apache.org/jira/browse/HADOOP-9865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13742207#comment-13742207
]
Hudson commented on HADOOP-9865:
SUCCESS: Integrated in Hadoop-Hdfs-trunk #1493 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1493/])
HADOOP-9865. FileContext#globStatus has a regression with respect to relative
path. (Contributed by Chaun Lin) (cmccabe:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1514531)
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Globber.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestGlobPaths.java
FileContext.globStatus() has a regression with respect to relative path
---
Key: HADOOP-9865
URL: https://issues.apache.org/jira/browse/HADOOP-9865
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.0.0, 2.3.0
Reporter: Chuan Liu
Assignee: Chuan Liu
Fix For: 2.3.0
Attachments: HADOOP-9865-demo.patch, HADOOP-9865-trunk.2.patch,
HADOOP-9865-trunk.3.patch, HADOOP-9865-trunk.patch
I discovered the problem when running unit test TestMRJobClient on Windows.
The cause is indirect in this case. In the unit test, we try to launch a job
and list its status. The job failed, and caused the list command get a result
of 0, which triggered the unit test assert. From the log and debug, the job
failed because we failed to create the Jar with classpath (see code around
{{FileUtil.createJarWithClassPath}}) in {{ContainerLaunch}}. This is a
Windows specific step right now; so the test still passes on Linux. This step
failed because we passed in a relative path to {{FileContext.globStatus()}}
in {{FileUtil.createJarWithClassPath}}. The relevant log looks like the
following.
{noformat}
2013-08-12 16:12:05,937 WARN [ContainersLauncher #0]
launcher.ContainerLaunch (ContainerLaunch.java:call(270)) - Failed to launch
container.
org.apache.hadoop.HadoopIllegalArgumentException: Path is relative
at org.apache.hadoop.fs.Path.checkNotRelative(Path.java:74)
at org.apache.hadoop.fs.FileContext.getFSofPath(FileContext.java:304)
at org.apache.hadoop.fs.Globber.schemeFromPath(Globber.java:107)
at org.apache.hadoop.fs.Globber.glob(Globber.java:128)
at
org.apache.hadoop.fs.FileContext$Util.globStatus(FileContext.java:1908)
at
org.apache.hadoop.fs.FileUtil.createJarWithClassPath(FileUtil.java:1247)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.sanitizeEnv(ContainerLaunch.java:679)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:232)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:1)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}
I think this is a regression from HADOOP-9817. I modified some code and the
unit test passed. (See the attached patch.) However, I think the impact is
larger. I will add some unit tests to verify the behavior, and work on a more
complete fix.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9865) FileContext.globStatus() has a regression with respect to relative path
[
https://issues.apache.org/jira/browse/HADOOP-9865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13742279#comment-13742279
]
Hudson commented on HADOOP-9865:
FAILURE: Integrated in Hadoop-Mapreduce-trunk #1520 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1520/])
HADOOP-9865. FileContext#globStatus has a regression with respect to relative
path. (Contributed by Chaun Lin) (cmccabe:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1514531)
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Globber.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestGlobPaths.java
FileContext.globStatus() has a regression with respect to relative path
---
Key: HADOOP-9865
URL: https://issues.apache.org/jira/browse/HADOOP-9865
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.0.0, 2.3.0
Reporter: Chuan Liu
Assignee: Chuan Liu
Fix For: 2.3.0
Attachments: HADOOP-9865-demo.patch, HADOOP-9865-trunk.2.patch,
HADOOP-9865-trunk.3.patch, HADOOP-9865-trunk.patch
I discovered the problem when running unit test TestMRJobClient on Windows.
The cause is indirect in this case. In the unit test, we try to launch a job
and list its status. The job failed, and caused the list command get a result
of 0, which triggered the unit test assert. From the log and debug, the job
failed because we failed to create the Jar with classpath (see code around
{{FileUtil.createJarWithClassPath}}) in {{ContainerLaunch}}. This is a
Windows specific step right now; so the test still passes on Linux. This step
failed because we passed in a relative path to {{FileContext.globStatus()}}
in {{FileUtil.createJarWithClassPath}}. The relevant log looks like the
following.
{noformat}
2013-08-12 16:12:05,937 WARN [ContainersLauncher #0]
launcher.ContainerLaunch (ContainerLaunch.java:call(270)) - Failed to launch
container.
org.apache.hadoop.HadoopIllegalArgumentException: Path is relative
at org.apache.hadoop.fs.Path.checkNotRelative(Path.java:74)
at org.apache.hadoop.fs.FileContext.getFSofPath(FileContext.java:304)
at org.apache.hadoop.fs.Globber.schemeFromPath(Globber.java:107)
at org.apache.hadoop.fs.Globber.glob(Globber.java:128)
at
org.apache.hadoop.fs.FileContext$Util.globStatus(FileContext.java:1908)
at
org.apache.hadoop.fs.FileUtil.createJarWithClassPath(FileUtil.java:1247)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.sanitizeEnv(ContainerLaunch.java:679)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:232)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:1)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}
I think this is a regression from HADOOP-9817. I modified some code and the
unit test passed. (See the attached patch.) However, I think the impact is
larger. I will add some unit tests to verify the behavior, and work on a more
complete fix.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9865) FileContext.globStatus() has a regression with respect to relative path
[
https://issues.apache.org/jira/browse/HADOOP-9865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13740758#comment-13740758
]
Hadoop QA commented on HADOOP-9865:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12598147/HADOOP-9865-trunk.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2984//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2984//console
This message is automatically generated.
FileContext.globStatus() has a regression with respect to relative path
---
Key: HADOOP-9865
URL: https://issues.apache.org/jira/browse/HADOOP-9865
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.0.0, 2.3.0
Reporter: Chuan Liu
Assignee: Chuan Liu
Attachments: HADOOP-9865-demo.patch, HADOOP-9865-trunk.2.patch,
HADOOP-9865-trunk.3.patch, HADOOP-9865-trunk.patch
I discovered the problem when running unit test TestMRJobClient on Windows.
The cause is indirect in this case. In the unit test, we try to launch a job
and list its status. The job failed, and caused the list command get a result
of 0, which triggered the unit test assert. From the log and debug, the job
failed because we failed to create the Jar with classpath (see code around
{{FileUtil.createJarWithClassPath}}) in {{ContainerLaunch}}. This is a
Windows specific step right now; so the test still passes on Linux. This step
failed because we passed in a relative path to {{FileContext.globStatus()}}
in {{FileUtil.createJarWithClassPath}}. The relevant log looks like the
following.
{noformat}
2013-08-12 16:12:05,937 WARN [ContainersLauncher #0]
launcher.ContainerLaunch (ContainerLaunch.java:call(270)) - Failed to launch
container.
org.apache.hadoop.HadoopIllegalArgumentException: Path is relative
at org.apache.hadoop.fs.Path.checkNotRelative(Path.java:74)
at org.apache.hadoop.fs.FileContext.getFSofPath(FileContext.java:304)
at org.apache.hadoop.fs.Globber.schemeFromPath(Globber.java:107)
at org.apache.hadoop.fs.Globber.glob(Globber.java:128)
at
org.apache.hadoop.fs.FileContext$Util.globStatus(FileContext.java:1908)
at
org.apache.hadoop.fs.FileUtil.createJarWithClassPath(FileUtil.java:1247)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.sanitizeEnv(ContainerLaunch.java:679)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:232)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:1)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}
I think this is a regression from HADOOP-9817. I modified some code and the
unit test passed. (See the attached patch.) However, I think the impact is
larger. I will add some unit tests to verify the behavior, and work on a more
complete fix.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9865) FileContext.globStatus() has a regression with respect to relative path
[
https://issues.apache.org/jira/browse/HADOOP-9865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13741615#comment-13741615
]
Colin Patrick McCabe commented on HADOOP-9865:
--
+1, thanks Chuan
FileContext.globStatus() has a regression with respect to relative path
---
Key: HADOOP-9865
URL: https://issues.apache.org/jira/browse/HADOOP-9865
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.0.0, 2.3.0
Reporter: Chuan Liu
Assignee: Chuan Liu
Attachments: HADOOP-9865-demo.patch, HADOOP-9865-trunk.2.patch,
HADOOP-9865-trunk.3.patch, HADOOP-9865-trunk.patch
I discovered the problem when running unit test TestMRJobClient on Windows.
The cause is indirect in this case. In the unit test, we try to launch a job
and list its status. The job failed, and caused the list command get a result
of 0, which triggered the unit test assert. From the log and debug, the job
failed because we failed to create the Jar with classpath (see code around
{{FileUtil.createJarWithClassPath}}) in {{ContainerLaunch}}. This is a
Windows specific step right now; so the test still passes on Linux. This step
failed because we passed in a relative path to {{FileContext.globStatus()}}
in {{FileUtil.createJarWithClassPath}}. The relevant log looks like the
following.
{noformat}
2013-08-12 16:12:05,937 WARN [ContainersLauncher #0]
launcher.ContainerLaunch (ContainerLaunch.java:call(270)) - Failed to launch
container.
org.apache.hadoop.HadoopIllegalArgumentException: Path is relative
at org.apache.hadoop.fs.Path.checkNotRelative(Path.java:74)
at org.apache.hadoop.fs.FileContext.getFSofPath(FileContext.java:304)
at org.apache.hadoop.fs.Globber.schemeFromPath(Globber.java:107)
at org.apache.hadoop.fs.Globber.glob(Globber.java:128)
at
org.apache.hadoop.fs.FileContext$Util.globStatus(FileContext.java:1908)
at
org.apache.hadoop.fs.FileUtil.createJarWithClassPath(FileUtil.java:1247)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.sanitizeEnv(ContainerLaunch.java:679)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:232)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:1)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}
I think this is a regression from HADOOP-9817. I modified some code and the
unit test passed. (See the attached patch.) However, I think the impact is
larger. I will add some unit tests to verify the behavior, and work on a more
complete fix.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9865) FileContext.globStatus() has a regression with respect to relative path
[
https://issues.apache.org/jira/browse/HADOOP-9865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13741682#comment-13741682
]
Hudson commented on HADOOP-9865:
SUCCESS: Integrated in Hadoop-trunk-Commit #4276 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/4276/])
HADOOP-9865. FileContext#globStatus has a regression with respect to relative
path. (Contributed by Chaun Lin) (cmccabe:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1514531)
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Globber.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestGlobPaths.java
FileContext.globStatus() has a regression with respect to relative path
---
Key: HADOOP-9865
URL: https://issues.apache.org/jira/browse/HADOOP-9865
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.0.0, 2.3.0
Reporter: Chuan Liu
Assignee: Chuan Liu
Fix For: 2.3.0
Attachments: HADOOP-9865-demo.patch, HADOOP-9865-trunk.2.patch,
HADOOP-9865-trunk.3.patch, HADOOP-9865-trunk.patch
I discovered the problem when running unit test TestMRJobClient on Windows.
The cause is indirect in this case. In the unit test, we try to launch a job
and list its status. The job failed, and caused the list command get a result
of 0, which triggered the unit test assert. From the log and debug, the job
failed because we failed to create the Jar with classpath (see code around
{{FileUtil.createJarWithClassPath}}) in {{ContainerLaunch}}. This is a
Windows specific step right now; so the test still passes on Linux. This step
failed because we passed in a relative path to {{FileContext.globStatus()}}
in {{FileUtil.createJarWithClassPath}}. The relevant log looks like the
following.
{noformat}
2013-08-12 16:12:05,937 WARN [ContainersLauncher #0]
launcher.ContainerLaunch (ContainerLaunch.java:call(270)) - Failed to launch
container.
org.apache.hadoop.HadoopIllegalArgumentException: Path is relative
at org.apache.hadoop.fs.Path.checkNotRelative(Path.java:74)
at org.apache.hadoop.fs.FileContext.getFSofPath(FileContext.java:304)
at org.apache.hadoop.fs.Globber.schemeFromPath(Globber.java:107)
at org.apache.hadoop.fs.Globber.glob(Globber.java:128)
at
org.apache.hadoop.fs.FileContext$Util.globStatus(FileContext.java:1908)
at
org.apache.hadoop.fs.FileUtil.createJarWithClassPath(FileUtil.java:1247)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.sanitizeEnv(ContainerLaunch.java:679)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:232)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:1)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}
I think this is a regression from HADOOP-9817. I modified some code and the
unit test passed. (See the attached patch.) However, I think the impact is
larger. I will add some unit tests to verify the behavior, and work on a more
complete fix.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9865) FileContext.globStatus() has a regression with respect to relative path
[
https://issues.apache.org/jira/browse/HADOOP-9865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13740345#comment-13740345
]
Colin Patrick McCabe commented on HADOOP-9865:
--
Thanks for working on this, Chuan.
In TestRelativePath, I would suggest using {{TestPath#mergeStatuses}} to get
back a string which you can compare using {{Assert#assertEquals}}. The reason
is because {{globStatus}} doesn't guarantee the order in which it returns
results-- you test could break if someone changes Globber.java slightly and
that changes the order in which results are returned.
{{TestPath#mergeStatuses}} avoids this problem by sorting all the results.
It's also much nicer to see a test error message that includes all results.
FileContext.globStatus() has a regression with respect to relative path
---
Key: HADOOP-9865
URL: https://issues.apache.org/jira/browse/HADOOP-9865
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.0.0, 2.3.0
Reporter: Chuan Liu
Assignee: Chuan Liu
Attachments: HADOOP-9865-demo.patch, HADOOP-9865-trunk.patch
I discovered the problem when running unit test TestMRJobClient on Windows.
The cause is indirect in this case. In the unit test, we try to launch a job
and list its status. The job failed, and caused the list command get a result
of 0, which triggered the unit test assert. From the log and debug, the job
failed because we failed to create the Jar with classpath (see code around
{{FileUtil.createJarWithClassPath}}) in {{ContainerLaunch}}. This is a
Windows specific step right now; so the test still passes on Linux. This step
failed because we passed in a relative path to {{FileContext.globStatus()}}
in {{FileUtil.createJarWithClassPath}}. The relevant log looks like the
following.
{noformat}
2013-08-12 16:12:05,937 WARN [ContainersLauncher #0]
launcher.ContainerLaunch (ContainerLaunch.java:call(270)) - Failed to launch
container.
org.apache.hadoop.HadoopIllegalArgumentException: Path is relative
at org.apache.hadoop.fs.Path.checkNotRelative(Path.java:74)
at org.apache.hadoop.fs.FileContext.getFSofPath(FileContext.java:304)
at org.apache.hadoop.fs.Globber.schemeFromPath(Globber.java:107)
at org.apache.hadoop.fs.Globber.glob(Globber.java:128)
at
org.apache.hadoop.fs.FileContext$Util.globStatus(FileContext.java:1908)
at
org.apache.hadoop.fs.FileUtil.createJarWithClassPath(FileUtil.java:1247)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.sanitizeEnv(ContainerLaunch.java:679)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:232)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:1)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}
I think this is a regression from HADOOP-9817. I modified some code and the
unit test passed. (See the attached patch.) However, I think the impact is
larger. I will add some unit tests to verify the behavior, and work on a more
complete fix.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9865) FileContext.globStatus() has a regression with respect to relative path
[
https://issues.apache.org/jira/browse/HADOOP-9865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738688#comment-13738688
]
Colin Patrick McCabe commented on HADOOP-9865:
--
{code}
+String scheme = schemeFromPath(fixRelativePart(pathPattern));
+String authority = authorityFromPath(fixRelativePart(pathPattern));
{code}
This is a good start, but the problem is that pathPattern is not actually a
path-- it's a pattern. So it may be something like {/,a}/foo, which you can't
really make into an absolute path in a sensible way.
I think the right fix is something like this:
{code}
diff --git
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Globber.java
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Globber.java
index ad28478..378311a 100644
---
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Globber.java
+++
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Globber.java
@@ -99,24 +99,24 @@ private Path fixRelativePart(Path path) {
}
private String schemeFromPath(Path path) throws IOException {
-String scheme = pathPattern.toUri().getScheme();
+String scheme = path.toUri().getScheme();
if (scheme == null) {
if (fs != null) {
scheme = fs.getUri().getScheme();
} else {
-scheme = fc.getFSofPath(path).getUri().getScheme();
+scheme = fc.getDefaultFileSystem().getUri().getScheme();
}
}
return scheme;
}
private String authorityFromPath(Path path) throws IOException {
-String authority = pathPattern.toUri().getAuthority();
+String authority = path.toUri().getAuthority();
if (authority == null) {
if (fs != null) {
authority = fs.getUri().getAuthority();
} else {
-authority = fc.getFSofPath(path).getUri().getAuthority();
+authority = fc.getDefaultFileSystem().getUri().getAuthority();
}
}
return authority ;
{code}
This probably needs more testing, including unit tests...
FileContext.globStatus() has a regression with respect to relative path
---
Key: HADOOP-9865
URL: https://issues.apache.org/jira/browse/HADOOP-9865
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.0.0, 2.3.0
Reporter: Chuan Liu
Assignee: Chuan Liu
Attachments: HADOOP-9865-demo.patch
I discovered the problem when running unit test TestMRJobClient on Windows.
The cause is indirect in this case. In the unit test, we try to launch a job
and list its status. The job failed, and caused the list command get a result
of 0, which triggered the unit test assert. From the log and debug, the job
failed because we failed to create the Jar with classpath (see code around
{{FileUtil.createJarWithClassPath}}) in {{ContainerLaunch}}. This is a
Windows specific step right now; so the test still passes on Linux. This step
failed because we passed in a relative path to {{FileContext.globStatus()}}
in {{FileUtil.createJarWithClassPath}}. The relevant log looks like the
following.
{noformat}
2013-08-12 16:12:05,937 WARN [ContainersLauncher #0]
launcher.ContainerLaunch (ContainerLaunch.java:call(270)) - Failed to launch
container.
org.apache.hadoop.HadoopIllegalArgumentException: Path is relative
at org.apache.hadoop.fs.Path.checkNotRelative(Path.java:74)
at org.apache.hadoop.fs.FileContext.getFSofPath(FileContext.java:304)
at org.apache.hadoop.fs.Globber.schemeFromPath(Globber.java:107)
at org.apache.hadoop.fs.Globber.glob(Globber.java:128)
at
org.apache.hadoop.fs.FileContext$Util.globStatus(FileContext.java:1908)
at
org.apache.hadoop.fs.FileUtil.createJarWithClassPath(FileUtil.java:1247)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.sanitizeEnv(ContainerLaunch.java:679)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:232)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:1)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}
I think this is a regression from HADOOP-9817. I modified some code and the
unit test passed. (See the attached patch.) However, I think the impact is
larger. I will add some unit tests to verify the behavior, and
[jira] [Commented] (HADOOP-9865) FileContext.globStatus() has a regression with respect to relative path
[
https://issues.apache.org/jira/browse/HADOOP-9865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13738807#comment-13738807
]
Chuan Liu commented on HADOOP-9865:
---
Thanks for the suggestion, Colin! I was also thinking something similar. The
patch was just to demo the problem in the description. I will add some unit
tests to cover the scenarios as well!
FileContext.globStatus() has a regression with respect to relative path
---
Key: HADOOP-9865
URL: https://issues.apache.org/jira/browse/HADOOP-9865
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.0.0, 2.3.0
Reporter: Chuan Liu
Assignee: Chuan Liu
Attachments: HADOOP-9865-demo.patch
I discovered the problem when running unit test TestMRJobClient on Windows.
The cause is indirect in this case. In the unit test, we try to launch a job
and list its status. The job failed, and caused the list command get a result
of 0, which triggered the unit test assert. From the log and debug, the job
failed because we failed to create the Jar with classpath (see code around
{{FileUtil.createJarWithClassPath}}) in {{ContainerLaunch}}. This is a
Windows specific step right now; so the test still passes on Linux. This step
failed because we passed in a relative path to {{FileContext.globStatus()}}
in {{FileUtil.createJarWithClassPath}}. The relevant log looks like the
following.
{noformat}
2013-08-12 16:12:05,937 WARN [ContainersLauncher #0]
launcher.ContainerLaunch (ContainerLaunch.java:call(270)) - Failed to launch
container.
org.apache.hadoop.HadoopIllegalArgumentException: Path is relative
at org.apache.hadoop.fs.Path.checkNotRelative(Path.java:74)
at org.apache.hadoop.fs.FileContext.getFSofPath(FileContext.java:304)
at org.apache.hadoop.fs.Globber.schemeFromPath(Globber.java:107)
at org.apache.hadoop.fs.Globber.glob(Globber.java:128)
at
org.apache.hadoop.fs.FileContext$Util.globStatus(FileContext.java:1908)
at
org.apache.hadoop.fs.FileUtil.createJarWithClassPath(FileUtil.java:1247)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.sanitizeEnv(ContainerLaunch.java:679)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:232)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:1)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}
I think this is a regression from HADOOP-9817. I modified some code and the
unit test passed. (See the attached patch.) However, I think the impact is
larger. I will add some unit tests to verify the behavior, and work on a more
complete fix.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
