[jira] [Resolved] (HADOOP-9597) FileSystem open() API is not clear if FileNotFoundException is throw when the path does not exist
[ https://issues.apache.org/jira/browse/HADOOP-9597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HADOOP-9597. - Resolution: Invalid FileSystem open() API is not clear if FileNotFoundException is throw when the path does not exist - Key: HADOOP-9597 URL: https://issues.apache.org/jira/browse/HADOOP-9597 Project: Hadoop Common Issue Type: Improvement Components: documentation, fs Affects Versions: 2.0.4-alpha Reporter: Jerry He Priority: Minor Fix For: 2.0.4.1-alpha The current FileSystem open() method throws a generic IOException in its API specification. Some FileSystem implementations (DFS, RawLocalFileSystem ...) throws more specific FileNotFoundException if the path does not exist. Some throws IOException only (FTPFileSystem, HftpFileSystem ...). If we have a new FileSystem implementation, what should we follow exactly for open()? What should the application expect in this case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9597) FileSystem open() API is not clear if FileNotFoundException is throw when the path does not exist
[ https://issues.apache.org/jira/browse/HADOOP-9597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666025#comment-13666025 ] Suresh Srinivas commented on HADOOP-9597: - Jerry, please do not open jiras for asking questions. Jiras are for reporting problem. FileSystem.java is not well documented and is implemented differently by different file systems. My recommendation would be to follow DistributedFileSystem implementation. You could also use FileContext javadoc to guide in what exception to throw. FileSystem open() API is not clear if FileNotFoundException is throw when the path does not exist - Key: HADOOP-9597 URL: https://issues.apache.org/jira/browse/HADOOP-9597 Project: Hadoop Common Issue Type: Improvement Components: documentation, fs Affects Versions: 2.0.4-alpha Reporter: Jerry He Priority: Minor Fix For: 2.0.4.1-alpha The current FileSystem open() method throws a generic IOException in its API specification. Some FileSystem implementations (DFS, RawLocalFileSystem ...) throws more specific FileNotFoundException if the path does not exist. Some throws IOException only (FTPFileSystem, HftpFileSystem ...). If we have a new FileSystem implementation, what should we follow exactly for open()? What should the application expect in this case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HADOOP-9435) Minimal code change to support IBM jvm build dependency
[ https://issues.apache.org/jira/browse/HADOOP-9435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13664943#comment-13664943 ] Suresh Srinivas edited comment on HADOOP-9435 at 5/24/13 6:23 AM: -- [~tianhonglouis] Want to make sure that this is ready to be committed and no tests are pending. was (Author: sureshms): [~tianhonglouis] Want to make sure that this is ready to be committed and not tests are pending. Minimal code change to support IBM jvm build dependency --- Key: HADOOP-9435 URL: https://issues.apache.org/jira/browse/HADOOP-9435 Project: Hadoop Common Issue Type: Bug Components: build Reporter: Tian Hong Wang Assignee: Tian Hong Wang Labels: patch Attachments: HADOOP-9435.patch, HADOOP-9435-v1.patch When native build hadoop-common-project with IBM java using command like: mvn package -Pnative it will exist the following errors. [exec] -- Configuring incomplete, errors occurred! [exec] JAVA_HOME=, JAVA_JVM_LIBRARY=/home/louis/ibm-java-i386-60/jre/lib/i386/classic/libjvm.so [exec] JAVA_INCLUDE_PATH=/home/louis/ibm-java-i386-60/include, JAVA_INCLUDE_PATH2=JAVA_INCLUDE_PATH2-NOTFOUND [exec] CMake Error at JNIFlags.cmake:113 (MESSAGE): [exec] Failed to find a viable JVM installation under JAVA_HOME. [exec] Call Stack (most recent call first): [exec] CMakeLists.txt:24 (include) The reason is that IBM java uses $JAVA_HOME/include/jniport.h instead of $JAVA_HOME/include/jni_md.h in non-IBM java. [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlsym' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlerror' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dladdr' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlopen' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlclose' [exec] collect2: ld returned 1 exit status [exec] make[2]: *** [test_libhdfs_ops] Error 1 [exec] make[1]: *** [CMakeFiles/test_libhdfs_ops.dir/all] Error 2 [exec] make: *** [all] Error The reason is libjvm.so need libdl when linking. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9598) test coverage for org.apache.hadoop.yarn.server.resourcemanager.tools.TestRMAdmin
Aleksey Gorshkov created HADOOP-9598: Summary: test coverage for org.apache.hadoop.yarn.server.resourcemanager.tools.TestRMAdmin Key: HADOOP-9598 URL: https://issues.apache.org/jira/browse/HADOOP-9598 Project: Hadoop Common Issue Type: Test Affects Versions: 3.0.0, 0.23.8, 2.0.4.1-alpha Reporter: Aleksey Gorshkov -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9598) test coverage for org.apache.hadoop.yarn.server.resourcemanager.tools.TestRMAdmin
[ https://issues.apache.org/jira/browse/HADOOP-9598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Gorshkov updated HADOOP-9598: - Attachment: HADOOP-9598-trunk.patch HADOOP-9598-branch-0.23.patch test coverage for org.apache.hadoop.yarn.server.resourcemanager.tools.TestRMAdmin - Key: HADOOP-9598 URL: https://issues.apache.org/jira/browse/HADOOP-9598 Project: Hadoop Common Issue Type: Test Affects Versions: 3.0.0, 0.23.8, 2.0.4.1-alpha Reporter: Aleksey Gorshkov Attachments: HADOOP-9598-branch-0.23.patch, HADOOP-9598-trunk.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9598) test coverage for org.apache.hadoop.yarn.server.resourcemanager.tools.TestRMAdmin
[ https://issues.apache.org/jira/browse/HADOOP-9598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666104#comment-13666104 ] Aleksey Gorshkov commented on HADOOP-9598: -- patch HADOOP-9598-branch-0.23.patch for branch-0.23 patch HADOOP-9598-trunk.patch for branch-2 and trunk test coverage for org.apache.hadoop.yarn.server.resourcemanager.tools.TestRMAdmin - Key: HADOOP-9598 URL: https://issues.apache.org/jira/browse/HADOOP-9598 Project: Hadoop Common Issue Type: Test Affects Versions: 3.0.0, 0.23.8, 2.0.4.1-alpha Reporter: Aleksey Gorshkov Attachments: HADOOP-9598-branch-0.23.patch, HADOOP-9598-trunk.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9598) test coverage for org.apache.hadoop.yarn.server.resourcemanager.tools.TestRMAdmin
[ https://issues.apache.org/jira/browse/HADOOP-9598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Gorshkov updated HADOOP-9598: - Status: Patch Available (was: Open) test coverage for org.apache.hadoop.yarn.server.resourcemanager.tools.TestRMAdmin - Key: HADOOP-9598 URL: https://issues.apache.org/jira/browse/HADOOP-9598 Project: Hadoop Common Issue Type: Test Affects Versions: 3.0.0, 0.23.8, 2.0.4.1-alpha Reporter: Aleksey Gorshkov Attachments: HADOOP-9598-branch-0.23.patch, HADOOP-9598-trunk.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9598) test coverage for org.apache.hadoop.yarn.server.resourcemanager.tools.TestRMAdmin
[
https://issues.apache.org/jira/browse/HADOOP-9598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666117#comment-13666117
]
Hadoop QA commented on HADOOP-9598:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584659/HADOOP-9598-trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2564//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2564//console
This message is automatically generated.
test coverage for
org.apache.hadoop.yarn.server.resourcemanager.tools.TestRMAdmin
-
Key: HADOOP-9598
URL: https://issues.apache.org/jira/browse/HADOOP-9598
Project: Hadoop Common
Issue Type: Test
Affects Versions: 3.0.0, 0.23.8, 2.0.4.1-alpha
Reporter: Aleksey Gorshkov
Attachments: HADOOP-9598-branch-0.23.patch, HADOOP-9598-trunk.patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9063) enhance unit-test coverage of class org.apache.hadoop.fs.FileUtil
[ https://issues.apache.org/jira/browse/HADOOP-9063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dennis Y updated HADOOP-9063: - Attachment: HADOOP-9063-branch-2--N2.patch resolved merge conflict in patch for branch-2 enhance unit-test coverage of class org.apache.hadoop.fs.FileUtil - Key: HADOOP-9063 URL: https://issues.apache.org/jira/browse/HADOOP-9063 Project: Hadoop Common Issue Type: Test Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6 Reporter: Ivan A. Veselovsky Assignee: Ivan A. Veselovsky Priority: Minor Attachments: HADOOP-9063--b.patch, HADOOP-9063-branch-0.23--b.patch, HADOOP-9063-branch-0.23--c.patch, HADOOP-9063-branch-2--N1.patch, HADOOP-9063-branch-2--N2.patch, HADOOP-9063.patch, HADOOP-9063-trunk--c.patch, HADOOP-9063-trunk--c.patch, HADOOP-9063-trunk--N2.patch, HADOOP-9063-trunk--N3.patch, HADOOP-9063-trunk--N6.patch Some methods of class org.apache.hadoop.fs.FileUtil are covered by unit-tests poorly or not covered at all. Enhance the coverage. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9593) stack trace printed at ERROR for all yarn clients without hadoop.home set
[ https://issues.apache.org/jira/browse/HADOOP-9593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666199#comment-13666199 ] Hudson commented on HADOOP-9593: Integrated in Hadoop-Yarn-trunk #219 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/219/]) HADOOP-9593 stack trace printed at ERROR for all yarn clients without hadoop.home set (Revision 1485842) Result = SUCCESS stevel : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1485842 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java stack trace printed at ERROR for all yarn clients without hadoop.home set - Key: HADOOP-9593 URL: https://issues.apache.org/jira/browse/HADOOP-9593 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 3.0.0 Reporter: Steve Loughran Attachments: HADOOP-9593-001.patch, HADOOP-9593-002.patch This is the problem of HADOOP-9482 now showing up in a different application -one whose log4j settings haven't turned off all Shell logging. Unless you do that, all yarn clients will have a stack trace at error in their logs, which is generating false alarms and is utterly pointless. Why does this merit a stack trace? Why log it at error? It's not an error for a client app to not have these values set as long as they have the relevant JARs on their classpath. And if they don't, they'll get some classpath error instead -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9063) enhance unit-test coverage of class org.apache.hadoop.fs.FileUtil
[
https://issues.apache.org/jira/browse/HADOOP-9063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666214#comment-13666214
]
Hadoop QA commented on HADOOP-9063:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584679/HADOOP-9063-branch-2--N2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2565//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2565//console
This message is automatically generated.
enhance unit-test coverage of class org.apache.hadoop.fs.FileUtil
-
Key: HADOOP-9063
URL: https://issues.apache.org/jira/browse/HADOOP-9063
Project: Hadoop Common
Issue Type: Test
Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6
Reporter: Ivan A. Veselovsky
Assignee: Ivan A. Veselovsky
Priority: Minor
Attachments: HADOOP-9063--b.patch, HADOOP-9063-branch-0.23--b.patch,
HADOOP-9063-branch-0.23--c.patch, HADOOP-9063-branch-2--N1.patch,
HADOOP-9063-branch-2--N2.patch, HADOOP-9063.patch,
HADOOP-9063-trunk--c.patch, HADOOP-9063-trunk--c.patch,
HADOOP-9063-trunk--N2.patch, HADOOP-9063-trunk--N3.patch,
HADOOP-9063-trunk--N6.patch
Some methods of class org.apache.hadoop.fs.FileUtil are covered by unit-tests
poorly or not covered at all. Enhance the coverage.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9593) stack trace printed at ERROR for all yarn clients without hadoop.home set
[ https://issues.apache.org/jira/browse/HADOOP-9593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666287#comment-13666287 ] Hudson commented on HADOOP-9593: Integrated in Hadoop-Hdfs-trunk #1409 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1409/]) HADOOP-9593 stack trace printed at ERROR for all yarn clients without hadoop.home set (Revision 1485842) Result = FAILURE stevel : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1485842 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java stack trace printed at ERROR for all yarn clients without hadoop.home set - Key: HADOOP-9593 URL: https://issues.apache.org/jira/browse/HADOOP-9593 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 3.0.0 Reporter: Steve Loughran Attachments: HADOOP-9593-001.patch, HADOOP-9593-002.patch This is the problem of HADOOP-9482 now showing up in a different application -one whose log4j settings haven't turned off all Shell logging. Unless you do that, all yarn clients will have a stack trace at error in their logs, which is generating false alarms and is utterly pointless. Why does this merit a stack trace? Why log it at error? It's not an error for a client app to not have these values set as long as they have the relevant JARs on their classpath. And if they don't, they'll get some classpath error instead -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9593) stack trace printed at ERROR for all yarn clients without hadoop.home set
[ https://issues.apache.org/jira/browse/HADOOP-9593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666334#comment-13666334 ] Hudson commented on HADOOP-9593: Integrated in Hadoop-Mapreduce-trunk #1435 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1435/]) HADOOP-9593 stack trace printed at ERROR for all yarn clients without hadoop.home set (Revision 1485842) Result = SUCCESS stevel : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1485842 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/Shell.java stack trace printed at ERROR for all yarn clients without hadoop.home set - Key: HADOOP-9593 URL: https://issues.apache.org/jira/browse/HADOOP-9593 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 3.0.0 Reporter: Steve Loughran Attachments: HADOOP-9593-001.patch, HADOOP-9593-002.patch This is the problem of HADOOP-9482 now showing up in a different application -one whose log4j settings haven't turned off all Shell logging. Unless you do that, all yarn clients will have a stack trace at error in their logs, which is generating false alarms and is utterly pointless. Why does this merit a stack trace? Why log it at error? It's not an error for a client app to not have these values set as long as they have the relevant JARs on their classpath. And if they don't, they'll get some classpath error instead -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9573) Fix test-patch script to work with the enhanced PreCommit-Admin script.
[ https://issues.apache.org/jira/browse/HADOOP-9573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666428#comment-13666428 ] Matt Foley commented on HADOOP-9573: +1, Thanks, Giri! Fix test-patch script to work with the enhanced PreCommit-Admin script. --- Key: HADOOP-9573 URL: https://issues.apache.org/jira/browse/HADOOP-9573 Project: Hadoop Common Issue Type: Sub-task Components: build Affects Versions: 1.0.3 Reporter: Giridharan Kesavan Assignee: Giridharan Kesavan Attachments: 0001-Fix-test-patch-scrit-to-work-with-the-enhanced-PreCo.patch, hadoop-9573.patch test-patch script currently take the latest available patch for a given jira and performs the test. This jira is to enhance test-patch script to take attachment-id of a patch as an input and perform the tests using that attachment-id -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8468) Umbrella of enhancements to support different failure and locality topologies
[ https://issues.apache.org/jira/browse/HADOOP-8468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666454#comment-13666454 ] Junping Du commented on HADOOP-8468: I start the work to backport NodeGroup-aware topology to branch-2, there are 7 patches below in Common HDFS: patch 1 - HADOOP-8469: make NetworkTopology pluggable patch 2 - HADOOP-8470: implementation of NetworkTopologyWithNodeGroup patch 3 - HDFS-3498: Make BlockPlacementPolicyWithDefault extensible and Block Placement Removal policy extensible patch 4 - HDFS-3601: implementation of BlockPlacementPolicyWithNodeGroup patch 5 - HDFS-4240: fix a bug in nodegroup-aware case patch 6 - HDFS-3495: Make Balancer support node group layer patch 7 - HDFS-4261: Fix a timeout issue in TestBalancerWithNodeGroup case Umbrella of enhancements to support different failure and locality topologies - Key: HADOOP-8468 URL: https://issues.apache.org/jira/browse/HADOOP-8468 Project: Hadoop Common Issue Type: Improvement Components: ha, io Affects Versions: 1.0.0, 2.0.0-alpha Reporter: Junping Du Assignee: Junping Du Attachments: HADOOP-8468-total.patch, HADOOP-8468-total-v3.patch, HVE_Hadoop World Meetup 2012.pptx, HVE User Guide on branch-1(draft ).pdf, Proposal for enchanced failure and locality topologies.pdf, Proposal for enchanced failure and locality topologies (revised-1.0).pdf The current hadoop network topology (described in some previous issues like: Hadoop-692) works well in classic three-tiers network when it comes out. However, it does not take into account other failure models or changes in the infrastructure that can affect network bandwidth efficiency like: virtualization. Virtualized platform has following genes that shouldn't been ignored by hadoop topology in scheduling tasks, placing replica, do balancing or fetching block for reading: 1. VMs on the same physical host are affected by the same hardware failure. In order to match the reliability of a physical deployment, replication of data across two virtual machines on the same host should be avoided. 2. The network between VMs on the same physical host has higher throughput and lower latency and does not consume any physical switch bandwidth. Thus, we propose to make hadoop network topology extend-able and introduce a new level in the hierarchical topology, a node group level, which maps well onto an infrastructure that is based on a virtualized environment. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9597) FileSystem open() API is not clear if FileNotFoundException is throw when the path does not exist
[ https://issues.apache.org/jira/browse/HADOOP-9597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666493#comment-13666493 ] Jerry He commented on HADOOP-9597: -- Hi, Suresh Thanks for you comment. But isn't this a problem? FileSystem interface is such an important interface. We have a proprietary implementation of FileSystem that does not throws FileNotFoundException. It causes a problem for HBase, which expects FileNotFoundException in one particular case. There could be other implementations out there. Java FileInputStream specifically throws FileNotFoundException, which is commonly used. We need to be clear, of course unless we intend to be ambiguous in this case. FileSystem open() API is not clear if FileNotFoundException is throw when the path does not exist - Key: HADOOP-9597 URL: https://issues.apache.org/jira/browse/HADOOP-9597 Project: Hadoop Common Issue Type: Improvement Components: documentation, fs Affects Versions: 2.0.4-alpha Reporter: Jerry He Priority: Minor Fix For: 2.0.4.1-alpha The current FileSystem open() method throws a generic IOException in its API specification. Some FileSystem implementations (DFS, RawLocalFileSystem ...) throws more specific FileNotFoundException if the path does not exist. Some throws IOException only (FTPFileSystem, HftpFileSystem ...). If we have a new FileSystem implementation, what should we follow exactly for open()? What should the application expect in this case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9599) hadoop-config.cmd doesn't set JAVA_LIBRARY_PATH correctly
Mostafa Elhemali created HADOOP-9599: Summary: hadoop-config.cmd doesn't set JAVA_LIBRARY_PATH correctly Key: HADOOP-9599 URL: https://issues.apache.org/jira/browse/HADOOP-9599 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0 Environment: Windows Reporter: Mostafa Elhemali Assignee: Mostafa Elhemali In Windows, hadoop-config.cmd uses the non-existent-variable HADOOP_CORE_HOME when setting the JAVA_LIBRAR_PATH variable. It should use HADOOP_HOME. The net effect is that running e.g. hdfs namenode would error out with UnsatisfiedLinkError because it can't access hadoop.dll. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9599) hadoop-config.cmd doesn't set JAVA_LIBRARY_PATH correctly
[ https://issues.apache.org/jira/browse/HADOOP-9599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mostafa Elhemali updated HADOOP-9599: - Attachment: HADOOP-9599.patch hadoop-config.cmd doesn't set JAVA_LIBRARY_PATH correctly - Key: HADOOP-9599 URL: https://issues.apache.org/jira/browse/HADOOP-9599 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0 Environment: Windows Reporter: Mostafa Elhemali Assignee: Mostafa Elhemali Attachments: HADOOP-9599.patch In Windows, hadoop-config.cmd uses the non-existent-variable HADOOP_CORE_HOME when setting the JAVA_LIBRAR_PATH variable. It should use HADOOP_HOME. The net effect is that running e.g. hdfs namenode would error out with UnsatisfiedLinkError because it can't access hadoop.dll. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9599) hadoop-config.cmd doesn't set JAVA_LIBRARY_PATH correctly
[ https://issues.apache.org/jira/browse/HADOOP-9599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mostafa Elhemali updated HADOOP-9599: - Status: Patch Available (was: Open) hadoop-config.cmd doesn't set JAVA_LIBRARY_PATH correctly - Key: HADOOP-9599 URL: https://issues.apache.org/jira/browse/HADOOP-9599 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0 Environment: Windows Reporter: Mostafa Elhemali Assignee: Mostafa Elhemali Attachments: HADOOP-9599.patch In Windows, hadoop-config.cmd uses the non-existent-variable HADOOP_CORE_HOME when setting the JAVA_LIBRAR_PATH variable. It should use HADOOP_HOME. The net effect is that running e.g. hdfs namenode would error out with UnsatisfiedLinkError because it can't access hadoop.dll. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9600) In Windows: Hadoop fails to run when JAVA_HOME has spaces in it
Mostafa Elhemali created HADOOP-9600: Summary: In Windows: Hadoop fails to run when JAVA_HOME has spaces in it Key: HADOOP-9600 URL: https://issues.apache.org/jira/browse/HADOOP-9600 Project: Hadoop Common Issue Type: Bug Environment: Windows Reporter: Mostafa Elhemali Assignee: Mostafa Elhemali hadoop-config.cmd misbehaves when JAVA_HOME has spaces in it (e.g. if Java is in c:\Program Files). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9600) In Windows: Hadoop fails to run when JAVA_HOME has spaces in it
[ https://issues.apache.org/jira/browse/HADOOP-9600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mostafa Elhemali updated HADOOP-9600: - Attachment: HADOOP-9600.patch In Windows: Hadoop fails to run when JAVA_HOME has spaces in it --- Key: HADOOP-9600 URL: https://issues.apache.org/jira/browse/HADOOP-9600 Project: Hadoop Common Issue Type: Bug Environment: Windows Reporter: Mostafa Elhemali Assignee: Mostafa Elhemali Attachments: HADOOP-9600.patch hadoop-config.cmd misbehaves when JAVA_HOME has spaces in it (e.g. if Java is in c:\Program Files). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9600) In Windows: Hadoop fails to run when JAVA_HOME has spaces in it
[ https://issues.apache.org/jira/browse/HADOOP-9600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mostafa Elhemali updated HADOOP-9600: - Status: Patch Available (was: Open) In Windows: Hadoop fails to run when JAVA_HOME has spaces in it --- Key: HADOOP-9600 URL: https://issues.apache.org/jira/browse/HADOOP-9600 Project: Hadoop Common Issue Type: Bug Environment: Windows Reporter: Mostafa Elhemali Assignee: Mostafa Elhemali Attachments: HADOOP-9600.patch hadoop-config.cmd misbehaves when JAVA_HOME has spaces in it (e.g. if Java is in c:\Program Files). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9290) Some tests cannot load native library
[
https://issues.apache.org/jira/browse/HADOOP-9290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suresh Srinivas updated HADOOP-9290:
Fix Version/s: 3.0.0
Some tests cannot load native library
-
Key: HADOOP-9290
URL: https://issues.apache.org/jira/browse/HADOOP-9290
Project: Hadoop Common
Issue Type: Bug
Components: build, native
Affects Versions: 3.0.0
Reporter: Arpit Agarwal
Assignee: Chris Nauroth
Fix For: 3.0.0
Attachments: HADOOP-9290.2.patch, HADOOP-9290.3.patch,
HADOOP-9290.4.patch, HADOOP-9290.patch
Some tests are unable to load the native DLL on Windows. The culprit appears
to be an incorrect PATH configuration for the native-win profile in
hadoop-project/pom.xml.
{code:xml}
profile
idnative-win/id
activation
os
familyWindows/family
/os
/activation
build
plugins
plugin
groupIdorg.apache.maven.plugins/groupId
artifactIdmaven-surefire-plugin/artifactId
configuration
environmentVariables
!-- Specify where to look for the native DLL on Windows --
PATH${env.PATH};${basedir}\..\..\hadoop-common-project\hadoop-common\target\bin;/PATH
/environmentVariables
/configuration
/plugin
/plugins
/build
/profile
{code}
This is evaluated independently by each project and the relative path is not
always correct.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9600) In Windows: Hadoop fails to run when JAVA_HOME has spaces in it
[
https://issues.apache.org/jira/browse/HADOOP-9600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666552#comment-13666552
]
Hadoop QA commented on HADOOP-9600:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584733/HADOOP-9600.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2566//console
This message is automatically generated.
In Windows: Hadoop fails to run when JAVA_HOME has spaces in it
---
Key: HADOOP-9600
URL: https://issues.apache.org/jira/browse/HADOOP-9600
Project: Hadoop Common
Issue Type: Bug
Environment: Windows
Reporter: Mostafa Elhemali
Assignee: Mostafa Elhemali
Attachments: HADOOP-9600.patch
hadoop-config.cmd misbehaves when JAVA_HOME has spaces in it (e.g. if Java is
in c:\Program Files).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9599) hadoop-config.cmd doesn't set JAVA_LIBRARY_PATH correctly
[
https://issues.apache.org/jira/browse/HADOOP-9599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666554#comment-13666554
]
Hadoop QA commented on HADOOP-9599:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584729/HADOOP-9599.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2567//console
This message is automatically generated.
hadoop-config.cmd doesn't set JAVA_LIBRARY_PATH correctly
-
Key: HADOOP-9599
URL: https://issues.apache.org/jira/browse/HADOOP-9599
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.0.0
Environment: Windows
Reporter: Mostafa Elhemali
Assignee: Mostafa Elhemali
Attachments: HADOOP-9599.patch
In Windows, hadoop-config.cmd uses the non-existent-variable HADOOP_CORE_HOME
when setting the JAVA_LIBRAR_PATH variable. It should use HADOOP_HOME.
The net effect is that running e.g. hdfs namenode would error out with
UnsatisfiedLinkError because it can't access hadoop.dll.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9600) In Windows: Hadoop fails to run when JAVA_HOME has spaces in it
[ https://issues.apache.org/jira/browse/HADOOP-9600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mostafa Elhemali updated HADOOP-9600: - Attachment: HADOOP-9600.2.patch In Windows: Hadoop fails to run when JAVA_HOME has spaces in it --- Key: HADOOP-9600 URL: https://issues.apache.org/jira/browse/HADOOP-9600 Project: Hadoop Common Issue Type: Bug Environment: Windows Reporter: Mostafa Elhemali Assignee: Mostafa Elhemali Attachments: HADOOP-9600.2.patch, HADOOP-9600.patch hadoop-config.cmd misbehaves when JAVA_HOME has spaces in it (e.g. if Java is in c:\Program Files). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9600) In Windows: Hadoop fails to run when JAVA_HOME has spaces in it
[
https://issues.apache.org/jira/browse/HADOOP-9600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666572#comment-13666572
]
Hadoop QA commented on HADOOP-9600:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584736/HADOOP-9600.2.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2568//console
This message is automatically generated.
In Windows: Hadoop fails to run when JAVA_HOME has spaces in it
---
Key: HADOOP-9600
URL: https://issues.apache.org/jira/browse/HADOOP-9600
Project: Hadoop Common
Issue Type: Bug
Environment: Windows
Reporter: Mostafa Elhemali
Assignee: Mostafa Elhemali
Attachments: HADOOP-9600.2.patch, HADOOP-9600.patch
hadoop-config.cmd misbehaves when JAVA_HOME has spaces in it (e.g. if Java is
in c:\Program Files).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9287) Parallel testing hadoop-common
[ https://issues.apache.org/jira/browse/HADOOP-9287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666592#comment-13666592 ] Andrey Klochkov commented on HADOOP-9287: - Definitely separate the yarn changes, create a new ticket at https://issues.apache.org/jira/browse/YARN. Tx. All changes in YARN code introduced in this patch are required because YARN test classes depend on classes from hadoop-common. If we extract these changes somewhere else, this patch would lead to a non-compilable code. Parallel testing hadoop-common -- Key: HADOOP-9287 URL: https://issues.apache.org/jira/browse/HADOOP-9287 Project: Hadoop Common Issue Type: Test Components: test Affects Versions: 3.0.0 Reporter: Tsuyoshi OZAWA Assignee: Andrey Klochkov Attachments: HADOOP-9287.1.patch, HADOOP-9287--N3.patch, HADOOP-9287--N3.patch, HADOOP-9287--N4.patch, HADOOP-9287--N5.patch, HADOOP-9287.patch, HADOOP-9287.patch The maven surefire plugin supports parallel testing feature. By using it, the tests can be run more faster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9407) commons-daemon 1.0.3 dependency has bad group id causing build issues
[ https://issues.apache.org/jira/browse/HADOOP-9407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated HADOOP-9407: --- Fix Version/s: 2.0.4.1-alpha commons-daemon 1.0.3 dependency has bad group id causing build issues - Key: HADOOP-9407 URL: https://issues.apache.org/jira/browse/HADOOP-9407 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 2.0.3-alpha, 2.0.4-alpha Reporter: Sangjin Lee Assignee: Sangjin Lee Labels: 2.0.4 Fix For: 2.0.5-beta, 2.0.4.1-alpha Attachments: HDFS-4497.patch The commons-daemon dependency of the hadoop-hdfs module has been at version 1.0.3 for a while. However, 1.0.3 has a pretty well-known groupId error in its pom (org.apache.commons as opposed to commons-daemon). This problem has since been corrected on commons-daemon starting 1.0.4. This causes build problems for many who depend on hadoop-hdfs directly and indirectly, however. Maven can skip over this metadata inconsistency. But other less forgiving build systems such as ivy and gradle have much harder time working around this problem. For example, in gradle, pretty much the only obvious way to work around this is to override this dependency version. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9593) stack trace printed at ERROR for all yarn clients without hadoop.home set
[ https://issues.apache.org/jira/browse/HADOOP-9593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-9593: --- Resolution: Fixed Fix Version/s: 2.0.5-beta 3.0.0 Assignee: Steve Loughran Status: Resolved (was: Patch Available) stack trace printed at ERROR for all yarn clients without hadoop.home set - Key: HADOOP-9593 URL: https://issues.apache.org/jira/browse/HADOOP-9593 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 3.0.0 Reporter: Steve Loughran Assignee: Steve Loughran Fix For: 3.0.0, 2.0.5-beta Attachments: HADOOP-9593-001.patch, HADOOP-9593-002.patch This is the problem of HADOOP-9482 now showing up in a different application -one whose log4j settings haven't turned off all Shell logging. Unless you do that, all yarn clients will have a stack trace at error in their logs, which is generating false alarms and is utterly pointless. Why does this merit a stack trace? Why log it at error? It's not an error for a client app to not have these values set as long as they have the relevant JARs on their classpath. And if they don't, they'll get some classpath error instead -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9582) Non-existent file to hadoop fs -conf doesn't throw error
[ https://issues.apache.org/jira/browse/HADOOP-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated HADOOP-9582: --- Attachment: HADOOP-9582.txt I've turned on the 'quiet' flag in FsShell. This would cause a Runtime exception to be raised when the user gives a nonexistent resource. I've also changed INFO logs(which depends on the flag) to debug so that output is not verbose. Non-existent file to hadoop fs -conf doesn't throw error -- Key: HADOOP-9582 URL: https://issues.apache.org/jira/browse/HADOOP-9582 Project: Hadoop Common Issue Type: Bug Components: conf Affects Versions: 3.0.0, 0.23.7, 2.0.4-alpha Reporter: Ashwin Shankar Attachments: HADOOP-9582.txt, HADOOP-9582.txt, HADOOP-9582.txt When we run : hadoop fs -conf BAD_FILE -ls / we expect hadoop to throw an error,but it doesn't. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9407) commons-daemon 1.0.3 dependency has bad group id causing build issues
[ https://issues.apache.org/jira/browse/HADOOP-9407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1320#comment-1320 ] Konstantin Boudnik commented on HADOOP-9407: I have just committed this fix to 2.0.4.1-alpha release (branch-2.0.4-alpha). Thank you Sangjin. commons-daemon 1.0.3 dependency has bad group id causing build issues - Key: HADOOP-9407 URL: https://issues.apache.org/jira/browse/HADOOP-9407 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 2.0.3-alpha, 2.0.4-alpha Reporter: Sangjin Lee Assignee: Sangjin Lee Labels: 2.0.4 Fix For: 2.0.5-beta, 2.0.4.1-alpha Attachments: HDFS-4497.patch The commons-daemon dependency of the hadoop-hdfs module has been at version 1.0.3 for a while. However, 1.0.3 has a pretty well-known groupId error in its pom (org.apache.commons as opposed to commons-daemon). This problem has since been corrected on commons-daemon starting 1.0.4. This causes build problems for many who depend on hadoop-hdfs directly and indirectly, however. Maven can skip over this metadata inconsistency. But other less forgiving build systems such as ivy and gradle have much harder time working around this problem. For example, in gradle, pretty much the only obvious way to work around this is to override this dependency version. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9287) Parallel testing hadoop-common
[ https://issues.apache.org/jira/browse/HADOOP-9287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1323#comment-1323 ] Jason Lowe commented on HADOOP-9287: bq. All changes in YARN code introduced in this patch are required because YARN test classes depend on classes from hadoop-common. If we extract these changes somewhere else, this patch would lead to a non-compilable code. That doesn't seem to be the case. I extracted the changes under hadoop-yarn-project into a separate patch, applied just that patch to trunk, and trunk built just fine. Even if that were the case, it's better to track the changes for YARN under a separate JIRA that is dependent upon this one being integrated. The only reason we aren't doing the same for HDFS is that those changes are intertwined. We can't commit just the common changes without breaking the HDFS build and vice-versa. That is not the case with the YARN changes, as the patch builds just fine without them. Since they are independent changes, they should be tracked in their respective projects. Parallel testing hadoop-common -- Key: HADOOP-9287 URL: https://issues.apache.org/jira/browse/HADOOP-9287 Project: Hadoop Common Issue Type: Test Components: test Affects Versions: 3.0.0 Reporter: Tsuyoshi OZAWA Assignee: Andrey Klochkov Attachments: HADOOP-9287.1.patch, HADOOP-9287--N3.patch, HADOOP-9287--N3.patch, HADOOP-9287--N4.patch, HADOOP-9287--N5.patch, HADOOP-9287.patch, HADOOP-9287.patch The maven surefire plugin supports parallel testing feature. By using it, the tests can be run more faster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HADOOP-9597) FileSystem open() API is not clear if FileNotFoundException is throw when the path does not exist
[ https://issues.apache.org/jira/browse/HADOOP-9597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reopened HADOOP-9597: FileSystem open() API is not clear if FileNotFoundException is throw when the path does not exist - Key: HADOOP-9597 URL: https://issues.apache.org/jira/browse/HADOOP-9597 Project: Hadoop Common Issue Type: Improvement Components: documentation, fs Affects Versions: 2.0.4-alpha Reporter: Jerry He Priority: Minor Fix For: 2.0.4.1-alpha The current FileSystem open() method throws a generic IOException in its API specification. Some FileSystem implementations (DFS, RawLocalFileSystem ...) throws more specific FileNotFoundException if the path does not exist. Some throws IOException only (FTPFileSystem, HftpFileSystem ...). If we have a new FileSystem implementation, what should we follow exactly for open()? What should the application expect in this case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9597) FileSystem open() API is not clear if FileNotFoundException is throw when the path does not exist
[ https://issues.apache.org/jira/browse/HADOOP-9597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-9597: --- Issue Type: Sub-task (was: Improvement) Parent: HADOOP-9361 FileSystem open() API is not clear if FileNotFoundException is throw when the path does not exist - Key: HADOOP-9597 URL: https://issues.apache.org/jira/browse/HADOOP-9597 Project: Hadoop Common Issue Type: Sub-task Components: documentation, fs Affects Versions: 2.0.4-alpha Reporter: Jerry He Priority: Minor Fix For: 2.0.4.1-alpha The current FileSystem open() method throws a generic IOException in its API specification. Some FileSystem implementations (DFS, RawLocalFileSystem ...) throws more specific FileNotFoundException if the path does not exist. Some throws IOException only (FTPFileSystem, HftpFileSystem ...). If we have a new FileSystem implementation, what should we follow exactly for open()? What should the application expect in this case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9407) commons-daemon 1.0.3 dependency has bad group id causing build issues
[ https://issues.apache.org/jira/browse/HADOOP-9407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1334#comment-1334 ] Sangjin Lee commented on HADOOP-9407: - Thanks Konstantin! commons-daemon 1.0.3 dependency has bad group id causing build issues - Key: HADOOP-9407 URL: https://issues.apache.org/jira/browse/HADOOP-9407 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 2.0.3-alpha, 2.0.4-alpha Reporter: Sangjin Lee Assignee: Sangjin Lee Labels: 2.0.4 Fix For: 2.0.5-beta, 2.0.4.1-alpha Attachments: HDFS-4497.patch The commons-daemon dependency of the hadoop-hdfs module has been at version 1.0.3 for a while. However, 1.0.3 has a pretty well-known groupId error in its pom (org.apache.commons as opposed to commons-daemon). This problem has since been corrected on commons-daemon starting 1.0.4. This causes build problems for many who depend on hadoop-hdfs directly and indirectly, however. Maven can skip over this metadata inconsistency. But other less forgiving build systems such as ivy and gradle have much harder time working around this problem. For example, in gradle, pretty much the only obvious way to work around this is to override this dependency version. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9407) commons-daemon 1.0.3 dependency has bad group id causing build issues
[ https://issues.apache.org/jira/browse/HADOOP-9407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1337#comment-1337 ] Konstantin Boudnik commented on HADOOP-9407: I see this already committed to the branch-2, which is in my understanding is effectively tracking 2.0.5 release now. So, I guess the ticket should be closed. commons-daemon 1.0.3 dependency has bad group id causing build issues - Key: HADOOP-9407 URL: https://issues.apache.org/jira/browse/HADOOP-9407 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 2.0.3-alpha, 2.0.4-alpha Reporter: Sangjin Lee Assignee: Sangjin Lee Labels: 2.0.4 Fix For: 2.0.5-beta, 2.0.4.1-alpha Attachments: HDFS-4497.patch The commons-daemon dependency of the hadoop-hdfs module has been at version 1.0.3 for a while. However, 1.0.3 has a pretty well-known groupId error in its pom (org.apache.commons as opposed to commons-daemon). This problem has since been corrected on commons-daemon starting 1.0.4. This causes build problems for many who depend on hadoop-hdfs directly and indirectly, however. Maven can skip over this metadata inconsistency. But other less forgiving build systems such as ivy and gradle have much harder time working around this problem. For example, in gradle, pretty much the only obvious way to work around this is to override this dependency version. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9582) Non-existent file to hadoop fs -conf doesn't throw error
[
https://issues.apache.org/jira/browse/HADOOP-9582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1341#comment-1341
]
Hadoop QA commented on HADOOP-9582:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584745/HADOOP-9582.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2569//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2569//console
This message is automatically generated.
Non-existent file to hadoop fs -conf doesn't throw error
--
Key: HADOOP-9582
URL: https://issues.apache.org/jira/browse/HADOOP-9582
Project: Hadoop Common
Issue Type: Bug
Components: conf
Affects Versions: 3.0.0, 0.23.7, 2.0.4-alpha
Reporter: Ashwin Shankar
Attachments: HADOOP-9582.txt, HADOOP-9582.txt, HADOOP-9582.txt
When we run :
hadoop fs -conf BAD_FILE -ls /
we expect hadoop to throw an error,but it doesn't.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9600) In Windows: Hadoop fails to run when JAVA_HOME has spaces in it
[ https://issues.apache.org/jira/browse/HADOOP-9600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mostafa Elhemali updated HADOOP-9600: - Attachment: HADOOP-9600.3.patch JAVA_HOME is also used within the code in Yarn application definitions - fixed what I could find there to quote it as well. In Windows: Hadoop fails to run when JAVA_HOME has spaces in it --- Key: HADOOP-9600 URL: https://issues.apache.org/jira/browse/HADOOP-9600 Project: Hadoop Common Issue Type: Bug Environment: Windows Reporter: Mostafa Elhemali Assignee: Mostafa Elhemali Attachments: HADOOP-9600.2.patch, HADOOP-9600.3.patch, HADOOP-9600.patch hadoop-config.cmd misbehaves when JAVA_HOME has spaces in it (e.g. if Java is in c:\Program Files). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9287) Parallel testing hadoop-common
[ https://issues.apache.org/jira/browse/HADOOP-9287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1367#comment-1367 ] Andrey Klochkov commented on HADOOP-9287: - Indeed, I was wrong and the changes in YARN are not necessary in this patch. Seems these changes are not needed at all as yarn tests are not supposed to be run in parallel. Eliminating these from the patch. Thanks Jason! Parallel testing hadoop-common -- Key: HADOOP-9287 URL: https://issues.apache.org/jira/browse/HADOOP-9287 Project: Hadoop Common Issue Type: Test Components: test Affects Versions: 3.0.0 Reporter: Tsuyoshi OZAWA Assignee: Andrey Klochkov Attachments: HADOOP-9287.1.patch, HADOOP-9287--N3.patch, HADOOP-9287--N3.patch, HADOOP-9287--N4.patch, HADOOP-9287--N5.patch, HADOOP-9287.patch, HADOOP-9287.patch The maven surefire plugin supports parallel testing feature. By using it, the tests can be run more faster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9287) Parallel testing hadoop-common
[ https://issues.apache.org/jira/browse/HADOOP-9287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov updated HADOOP-9287: Attachment: HADOOP-9287--N6.patch Parallel testing hadoop-common -- Key: HADOOP-9287 URL: https://issues.apache.org/jira/browse/HADOOP-9287 Project: Hadoop Common Issue Type: Test Components: test Affects Versions: 3.0.0 Reporter: Tsuyoshi OZAWA Assignee: Andrey Klochkov Attachments: HADOOP-9287.1.patch, HADOOP-9287--N3.patch, HADOOP-9287--N3.patch, HADOOP-9287--N4.patch, HADOOP-9287--N5.patch, HADOOP-9287--N6.patch, HADOOP-9287.patch, HADOOP-9287.patch The maven surefire plugin supports parallel testing feature. By using it, the tests can be run more faster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9579) Contrib ant test target not setting the java.library.path
[ https://issues.apache.org/jira/browse/HADOOP-9579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666724#comment-13666724 ] Chuan Liu commented on HADOOP-9579: --- +1 Do we know which tests will behave differently with or without native library? Contrib ant test target not setting the java.library.path - Key: HADOOP-9579 URL: https://issues.apache.org/jira/browse/HADOOP-9579 Project: Hadoop Common Issue Type: Bug Affects Versions: 1-win Reporter: Ivan Mitic Assignee: Ivan Mitic Attachments: HADOOP-9579.branch-1-win.patch build-contrib.xml does not set java.library.path causing tests to run without the hadoop native library. This is a bigger problem on Windows as having the native library is required. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9600) In Windows: Hadoop fails to run when JAVA_HOME has spaces in it
[
https://issues.apache.org/jira/browse/HADOOP-9600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666730#comment-13666730
]
Hadoop QA commented on HADOOP-9600:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584754/HADOOP-9600.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2570//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2570//console
This message is automatically generated.
In Windows: Hadoop fails to run when JAVA_HOME has spaces in it
---
Key: HADOOP-9600
URL: https://issues.apache.org/jira/browse/HADOOP-9600
Project: Hadoop Common
Issue Type: Bug
Environment: Windows
Reporter: Mostafa Elhemali
Assignee: Mostafa Elhemali
Attachments: HADOOP-9600.2.patch, HADOOP-9600.3.patch,
HADOOP-9600.patch
hadoop-config.cmd misbehaves when JAVA_HOME has spaces in it (e.g. if Java is
in c:\Program Files).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing
[ https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666735#comment-13666735 ] Daryn Sharp commented on HADOOP-9421: - Out-of-sync issues are inevitable if the client or server goes into a specific mode where it assumes a specific protobuf. Ie. How does a secure client know if it will get a RPC failure or SASL response after it sends the connection header? How can you fix the bad version response to handle that? Simply wrapping the sasl protobufs like other rpc calls allows the client/server to know how to decode the subsequent protbuf message. I don't feel it's a hack so much as good design. If this is our foreseeable last chance to modify RPC, then we need to make rpc decoding as flexible as possible. Convert SASL to use ProtoBuf and add lengths for non-blocking processing Key: HADOOP-9421 URL: https://issues.apache.org/jira/browse/HADOOP-9421 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 2.0.3-alpha Reporter: Sanjay Radia Assignee: Daryn Sharp Attachments: HADOOP-9421.patch, HADOOP-9421-v2-demo.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9373) Merge CHANGES.branch-trunk-win.txt to CHANGES.txt
[ https://issues.apache.org/jira/browse/HADOOP-9373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HADOOP-9373: Fix Version/s: (was: 3.0.0) 2.0.5-beta Merged change to branch-2 Merge CHANGES.branch-trunk-win.txt to CHANGES.txt - Key: HADOOP-9373 URL: https://issues.apache.org/jira/browse/HADOOP-9373 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0 Reporter: Suresh Srinivas Assignee: Suresh Srinivas Priority: Minor Fix For: 2.0.5-beta Attachments: HADOOP-9373.patch, HADOOP-9373.patch This is to merge the changes from CHANGES.branch-trunk-win.txt to appropriate CHANGES.txt files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9376) TestProxyUserFromEnv fails on a Windows domain joined machine
[ https://issues.apache.org/jira/browse/HADOOP-9376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HADOOP-9376: Fix Version/s: (was: 3.0.0) 2.0.5-beta Merged changed to branch-2 TestProxyUserFromEnv fails on a Windows domain joined machine - Key: HADOOP-9376 URL: https://issues.apache.org/jira/browse/HADOOP-9376 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0 Reporter: Ivan Mitic Assignee: Ivan Mitic Fix For: 2.0.5-beta Attachments: HADOOP-9376.trunk.patch TestProxyUserFromEnv#testProxyUserFromEnvironment fails with the following error on my machine: org.junit.ComparisonFailure: expected:[redmond\]ivanmi but was:[]ivanmi at org.junit.Assert.assertEquals(Assert.java:123) at org.junit.Assert.assertEquals(Assert.java:145) at org.apache.hadoop.security.TestProxyUserFromEnv.testProxyUserFromEnvironment(TestProxyUserFromEnv.java:45) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) at org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184) at org.junit.runners.ParentRunner.run(ParentRunner.java:236) at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50) at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9365) TestHAZKUtil fails on Windows
[ https://issues.apache.org/jira/browse/HADOOP-9365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HADOOP-9365: Fix Version/s: (was: 3.0.0) 2.0.5-beta Merged the change to branch-2 TestHAZKUtil fails on Windows - Key: HADOOP-9365 URL: https://issues.apache.org/jira/browse/HADOOP-9365 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0, trunk-win Reporter: Ivan Mitic Assignee: Ivan Mitic Fix For: 2.0.5-beta Attachments: HADOOP-9365.branch-trunk-win.hazkutil.patch TestHAZKUtil#testConfIndirectionfails on the following validation: assertTrue(fnfe.getMessage().startsWith(BOGUS_FILE)); because the path separators do not match: Expected: /-this-does-not-exist Actual: \-this-does-not-exist -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9364) PathData#expandAsGlob does not return correct results for absolute paths on Windows
[
https://issues.apache.org/jira/browse/HADOOP-9364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suresh Srinivas updated HADOOP-9364:
Fix Version/s: (was: 3.0.0)
2.0.5-beta
Merged the change to branch-2
PathData#expandAsGlob does not return correct results for absolute paths on
Windows
---
Key: HADOOP-9364
URL: https://issues.apache.org/jira/browse/HADOOP-9364
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.0.0, trunk-win
Reporter: Ivan Mitic
Assignee: Ivan Mitic
Fix For: 2.0.5-beta
Attachments: HADOOP-9364.2.patch,
HADOOP-9364.branch-trunk-win.pathdata.patch
This causes {{FsShell ls}} not to work properly for absolute paths. For
example:
{code}
-fs hdfs://127.0.0.1:58559 -ls -R /dir0
{code}
returns
{code}
drwxr-xr-x - ivanmi supergroup 0 2013-03-05 11:15 ../../dir0/dir1
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8973) DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs
[ https://issues.apache.org/jira/browse/HADOOP-8973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HADOOP-8973: Target Version/s: (was: 3.0.0, 1-win) Fix Version/s: (was: 3.0.0) 2.0.5-beta Merged the patch to branch-2 DiskChecker cannot reliably detect an inaccessible disk on Windows with NTFS ACLs - Key: HADOOP-8973 URL: https://issues.apache.org/jira/browse/HADOOP-8973 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 3.0.0, 1-win Reporter: Chris Nauroth Assignee: Chris Nauroth Fix For: 1-win, 2.0.5-beta Attachments: DiskChecker.proto.patch, HADOOP-8973.3.patch, HADOOP-8973-branch-1-win.3.patch, HADOOP-8973-branch-trunk-win.2.patch, HADOOP-8973-branch-trunk-win.patch DiskChecker.checkDir uses File.canRead, File.canWrite, and File.canExecute to check if a directory is inaccessible. These APIs are not reliable on Windows with NTFS ACLs due to a known JVM bug. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing
[ https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated HADOOP-9421: Attachment: HADOOP-9421.patch I scaled back my patch to the simplest skeleton implementation. The highlights are: * SASL is very stateful * SASL is wrapped with RPC header like normal requests to facility dynamic decoding * Client can correctly handle RPC exceptions during SASL negotiation * Server advertises SASL mechs to client Deficiencies to be addressed in another jira: * RPC client still pre-instantiates a SASL client and blindly uses it - instead of using the server advertisement * RPC server is also pre-creating a SASL server * Protocol and serverId not included in the SASL mech advertisement or the client initiate Code cleanups: * RPC client loop for SASL and normal operation should be folded together simplify and reduce redundant code * RPC server decoding paths for connect/sasl/rpc can be greatly simplified - the existing impl is already a twisty maze I've started on some of the cleanup, but I wanted to get a functional patch to unblock the start of v9 testing, and that I can use as a base for work of a larger scope. Convert SASL to use ProtoBuf and add lengths for non-blocking processing Key: HADOOP-9421 URL: https://issues.apache.org/jira/browse/HADOOP-9421 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 2.0.3-alpha Reporter: Sanjay Radia Assignee: Daryn Sharp Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421-v2-demo.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing
[ https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated HADOOP-9421: Status: Patch Available (was: Open) Convert SASL to use ProtoBuf and add lengths for non-blocking processing Key: HADOOP-9421 URL: https://issues.apache.org/jira/browse/HADOOP-9421 Project: Hadoop Common Issue Type: Sub-task Affects Versions: 2.0.3-alpha Reporter: Sanjay Radia Assignee: Daryn Sharp Attachments: HADOOP-9421.patch, HADOOP-9421.patch, HADOOP-9421-v2-demo.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9388) TestFsShellCopy fails on Windows
[
https://issues.apache.org/jira/browse/HADOOP-9388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suresh Srinivas updated HADOOP-9388:
Fix Version/s: (was: 3.0.0)
2.0.5-beta
Merged the patch to branch-2
TestFsShellCopy fails on Windows
Key: HADOOP-9388
URL: https://issues.apache.org/jira/browse/HADOOP-9388
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Ivan Mitic
Assignee: Ivan Mitic
Fix For: 2.0.5-beta
Attachments: HADOOP-9388.trunk.patch
Test fails on below test cases:
{code}
Tests run: 11, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 4.343 sec
FAILURE!
testMoveDirFromLocal(org.apache.hadoop.fs.TestFsShellCopy) Time elapsed: 29
sec FAILURE!
java.lang.AssertionError: expected:0 but was:1
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.failNotEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:126)
at org.junit.Assert.assertEquals(Assert.java:470)
at org.junit.Assert.assertEquals(Assert.java:454)
at
org.apache.hadoop.fs.TestFsShellCopy.testMoveDirFromLocal(TestFsShellCopy.java:392)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at
org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
testMoveDirFromLocalDestExists(org.apache.hadoop.fs.TestFsShellCopy) Time
elapsed: 25 sec FAILURE!
java.lang.AssertionError: expected:0 but was:1
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.failNotEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:126)
at org.junit.Assert.assertEquals(Assert.java:470)
at org.junit.Assert.assertEquals(Assert.java:454)
at
org.apache.hadoop.fs.TestFsShellCopy.testMoveDirFromLocalDestExists(TestFsShellCopy.java:410)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at
[jira] [Updated] (HADOOP-9437) TestNativeIO#testRenameTo fails on Windows due to assumption that POSIX errno is embedded in NativeIOException
[
https://issues.apache.org/jira/browse/HADOOP-9437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suresh Srinivas updated HADOOP-9437:
Target Version/s: (was: 3.0.0)
Fix Version/s: (was: 3.0.0)
2.0.5-beta
I merged the patch to branch-2.
TestNativeIO#testRenameTo fails on Windows due to assumption that POSIX errno
is embedded in NativeIOException
--
Key: HADOOP-9437
URL: https://issues.apache.org/jira/browse/HADOOP-9437
Project: Hadoop Common
Issue Type: Bug
Components: test
Affects Versions: 3.0.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Fix For: 2.0.5-beta
Attachments: HADOOP-9437.1.patch, HADOOP-9437.2.patch,
HADOOP-9437.3.patch
HDFS-4428 added a detailed error message for failures to rename files by
embedding the POSIX errno in the {{NativeIOException}}. On Windows, the
mapping of errno is not performed, so the errno enum value will not be
present in the {{NativeIOException}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9353) Activate native-win profile by default on Windows
[ https://issues.apache.org/jira/browse/HADOOP-9353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HADOOP-9353: Fix Version/s: (was: 3.0.0) 2.0.5-beta I merged the patch to branch-2. Activate native-win profile by default on Windows - Key: HADOOP-9353 URL: https://issues.apache.org/jira/browse/HADOOP-9353 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 3.0.0 Reporter: Arpit Agarwal Assignee: Arpit Agarwal Labels: windows Fix For: 2.0.5-beta Attachments: HADOOP-9353.1.patch, HADOOP-9353.patch Original Estimate: 1m Remaining Estimate: 1m Hadoop on Windows requires native components to be available so the native-win profile should be activated by default. The fix should update BUILDING.txt appropriately. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9486) Promote Windows and Shell related utils from YARN to Hadoop Common
[ https://issues.apache.org/jira/browse/HADOOP-9486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HADOOP-9486: Fix Version/s: (was: 3.0.0) 2.0.5-beta I merged the patch to branch-2. Promote Windows and Shell related utils from YARN to Hadoop Common -- Key: HADOOP-9486 URL: https://issues.apache.org/jira/browse/HADOOP-9486 Project: Hadoop Common Issue Type: Bug Reporter: Vinod Kumar Vavilapalli Assignee: Chris Nauroth Fix For: 2.0.5-beta Attachments: HADOOP-9486.patch This is happening as part of YARN-493, this is a tracking ticket for common changes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9488) FileUtil#createJarWithClassPath only substitutes environment variables from current process environment/does not support overriding when launching new process
[
https://issues.apache.org/jira/browse/HADOOP-9488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suresh Srinivas updated HADOOP-9488:
Fix Version/s: (was: 3.0.0)
2.0.5-beta
I merged the patch to branch-2.
FileUtil#createJarWithClassPath only substitutes environment variables from
current process environment/does not support overriding when launching new
process
--
Key: HADOOP-9488
URL: https://issues.apache.org/jira/browse/HADOOP-9488
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 3.0.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Fix For: 2.0.5-beta
Attachments: HADOOP-9488.1.patch, HADOOP-9488.consolidated.1.patch,
HADOOP-9488.consolidated.2.patch
{{FileUtil#createJarWithClassPath}} always uses {{System#getEnv}} for
substitution of environment variables in the classpath bundled into the jar
manifest. YARN launches container processes with a different set of
environment variables, so the method needs to support providing environment
variables different from the current process.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9490) LocalFileSystem#reportChecksumFailure not closing the checksum file handle before rename
[
https://issues.apache.org/jira/browse/HADOOP-9490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suresh Srinivas updated HADOOP-9490:
Fix Version/s: (was: 3.0.0)
2.0.5-beta
I merged the patch to branch-2.
LocalFileSystem#reportChecksumFailure not closing the checksum file handle
before rename
Key: HADOOP-9490
URL: https://issues.apache.org/jira/browse/HADOOP-9490
Project: Hadoop Common
Issue Type: Bug
Components: fs
Affects Versions: 3.0.0
Reporter: Ivan Mitic
Assignee: Ivan Mitic
Fix For: 2.0.5-beta
Attachments: HADOOP-9490.patch
LocalFileSystem#reportChecksumFailure is not closing the open stream on the
checksum file before it moves it to the bad_files folder, what causes the
operation to fail on Windows.
TestLocalFileSystem fail on Windows because of this:
{code}
testReportChecksumFailure(org.apache.hadoop.fs.TestLocalFileSystem) Time
elapsed: 31 sec FAILURE!
java.lang.AssertionError:
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertTrue(Assert.java:54)
at
org.apache.hadoop.fs.TestLocalFileSystem.testReportChecksumFailure(TestLocalFileSystem.java:335)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9500) TestUserGroupInformation#testGetServerSideGroups fails on Windows due to failure to find winutils.exe
[ https://issues.apache.org/jira/browse/HADOOP-9500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HADOOP-9500: Fix Version/s: (was: 3.0.0) 2.0.5-beta I merged the patch to branch-2. TestUserGroupInformation#testGetServerSideGroups fails on Windows due to failure to find winutils.exe - Key: HADOOP-9500 URL: https://issues.apache.org/jira/browse/HADOOP-9500 Project: Hadoop Common Issue Type: Bug Components: test Affects Versions: 3.0.0 Reporter: Chris Nauroth Assignee: Chris Nauroth Fix For: 2.0.5-beta Attachments: HADOOP-9500.1.patch The test attempts to run the winutils groups command, but the initialization logic fails to find the path to winutils.exe. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9290) Some tests cannot load native library
[
https://issues.apache.org/jira/browse/HADOOP-9290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suresh Srinivas updated HADOOP-9290:
Fix Version/s: (was: 3.0.0)
2.0.5-beta
I merged the patch to branch-2.
Some tests cannot load native library
-
Key: HADOOP-9290
URL: https://issues.apache.org/jira/browse/HADOOP-9290
Project: Hadoop Common
Issue Type: Bug
Components: build, native
Affects Versions: 3.0.0
Reporter: Arpit Agarwal
Assignee: Chris Nauroth
Fix For: 2.0.5-beta
Attachments: HADOOP-9290.2.patch, HADOOP-9290.3.patch,
HADOOP-9290.4.patch, HADOOP-9290.patch
Some tests are unable to load the native DLL on Windows. The culprit appears
to be an incorrect PATH configuration for the native-win profile in
hadoop-project/pom.xml.
{code:xml}
profile
idnative-win/id
activation
os
familyWindows/family
/os
/activation
build
plugins
plugin
groupIdorg.apache.maven.plugins/groupId
artifactIdmaven-surefire-plugin/artifactId
configuration
environmentVariables
!-- Specify where to look for the native DLL on Windows --
PATH${env.PATH};${basedir}\..\..\hadoop-common-project\hadoop-common\target\bin;/PATH
/environmentVariables
/configuration
/plugin
/plugins
/build
/profile
{code}
This is evaluated independently by each project and the relative path is not
always correct.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9443) Port winutils static code analysis change to trunk
[ https://issues.apache.org/jira/browse/HADOOP-9443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HADOOP-9443: Fix Version/s: (was: 3.0.0) 2.0.5-beta I merged the patch to branch-2. Port winutils static code analysis change to trunk -- Key: HADOOP-9443 URL: https://issues.apache.org/jira/browse/HADOOP-9443 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0 Reporter: Chuan Liu Assignee: Chuan Liu Fix For: 2.0.5-beta Attachments: HADOOP-9443-trunk.patch We hit a problem in winutils when running tests on Windows. The static code analysis change will fix the problem. More specifically, the old code always assumes the security descriptor get from GetSecurityDescriptorControl() is relative, and will make an absolute security descriptor out of it. The new absolute security descriptor will then pass to SetSecurityDescriptorDacl() to set permissions on the file. If the security descriptor is absolute, the new absolute security descriptor will be NULL, and we will run into the problem. This is what happened exactly in our case. The fix from static code analysis will solve the problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9553) TestAuthenticationToken fails on Windows
[ https://issues.apache.org/jira/browse/HADOOP-9553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HADOOP-9553: Fix Version/s: (was: 3.0.0) 2.0.5-beta I merged the patch to branch-2. TestAuthenticationToken fails on Windows Key: HADOOP-9553 URL: https://issues.apache.org/jira/browse/HADOOP-9553 Project: Hadoop Common Issue Type: Bug Components: test Affects Versions: 3.0.0 Reporter: Arpit Agarwal Assignee: Arpit Agarwal Fix For: 2.0.5-beta Attachments: HADOOP-9553.001.patch, HADOOP-9553.002.patch Likely test issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9556) disable HA tests on Windows that fail due to ZooKeeper client connection management bug
[ https://issues.apache.org/jira/browse/HADOOP-9556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HADOOP-9556: Fix Version/s: (was: 3.0.0) 2.0.5-beta I merged the patch to branch-2. disable HA tests on Windows that fail due to ZooKeeper client connection management bug --- Key: HADOOP-9556 URL: https://issues.apache.org/jira/browse/HADOOP-9556 Project: Hadoop Common Issue Type: Bug Components: ha, test Affects Versions: 3.0.0 Reporter: Chris Nauroth Assignee: Chris Nauroth Fix For: 2.0.5-beta Attachments: HADOOP-9556.1.patch ZOOKEEPER-1702 tracks a client connection management bug that manifests frequently on Windows. Until a patch is available in a ZooKeeper release, we need to disable the failing tests temporarily on Windows. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9483) winutils support for readlink command
[ https://issues.apache.org/jira/browse/HADOOP-9483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HADOOP-9483: Target Version/s: (was: 3.0.0) Fix Version/s: (was: 3.0.0) 2.0.5-beta I merged the patch to branch-2. winutils support for readlink command - Key: HADOOP-9483 URL: https://issues.apache.org/jira/browse/HADOOP-9483 Project: Hadoop Common Issue Type: Improvement Components: util Affects Versions: 3.0.0 Reporter: Chris Nauroth Assignee: Arpit Agarwal Fix For: 2.0.5-beta Attachments: HADOOP-9483.003.patch, HADOOP-9483.004.patch, HADOOP-9483.005.patch, HADOOP-9483.007.patch, HADOOP-9483.patch, HADOOP-9483.patch The current codebase relies on the Unix readlink command to determine the target of a symlink on the local file system. winutils currently does not support this functionality on Windows. Adding the command to winutils will prevent the need to use GnuWin32 or Cygwin for readlink support. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9043) disallow in winutils creating symlinks with forwards slashes
[ https://issues.apache.org/jira/browse/HADOOP-9043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HADOOP-9043: Fix Version/s: (was: 3.0.0) 2.0.5-beta I merged the patch to branch-2. disallow in winutils creating symlinks with forwards slashes Key: HADOOP-9043 URL: https://issues.apache.org/jira/browse/HADOOP-9043 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 3.0.0, 1-win Reporter: Chris Nauroth Assignee: Chris Nauroth Fix For: 1-win, 2.0.5-beta Attachments: HADOOP-9043.branch-1.2.patch, HADOOP-9043.branch-1-win.5.patch, HADOOP-9043.branch-1-win.6.patch, HADOOP-9043.branch-1-win.patch, HADOOP-9043.trunk.2.patch, HADOOP-9043.trunk.3.patch, HADOOP-9043.trunk.4.patch, HADOOP-9043.trunk.5.patch, HADOOP-9043.trunk.6.patch, HADOOP-9043.trunk.patch In general, the winutils symlink command rejects attempts to create symlinks targeting a destination file that does not exist. However, if given a symlink destination with forward slashes pointing at a file that does exist, then it creates the symlink with the forward slashes, and then attempts to open the file through the symlink will fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9413) Introduce common utils for File#setReadable/Writable/Executable and File#canRead/Write/Execute that work cross-platform
[ https://issues.apache.org/jira/browse/HADOOP-9413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HADOOP-9413: Target Version/s: (was: 3.0.0) Fix Version/s: (was: 3.0.0) 2.0.5-beta I merged the patch to branch-2. Introduce common utils for File#setReadable/Writable/Executable and File#canRead/Write/Execute that work cross-platform --- Key: HADOOP-9413 URL: https://issues.apache.org/jira/browse/HADOOP-9413 Project: Hadoop Common Issue Type: Bug Reporter: Ivan Mitic Assignee: Ivan Mitic Fix For: 2.0.5-beta Attachments: HADOOP-9413.commonfileutils.2.patch, HADOOP-9413.commonfileutils.3.patch, HADOOP-9413.commonfileutils.4.patch, HADOOP-9413.commonfileutils.5.patch, HADOOP-9413.commonfileutils.patch So far, we've seen many unittest and product bugs in Hadoop on Windows because Java's APIs that manipulate with permissions do not work as expected. We've addressed many of these problems on one-by-one basis (by either changing code a bit or disabling the test). While debugging the remaining unittest failures we continue to run into the same patterns of problems, and instead of addressing them one-by-one, I propose that we expose a set of equivalent wrapper APIs that will work well for all platforms. Scanning thru the codebase, this will actually be a simple change as there are very few places that use File#setReadable/Writable/Executable and File#canRead/Write/Execute (5 files in Common, 9 files in HDFS). HADOOP-8973 contains additional context on the problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9524) Fix ShellCommandFencer to work on Windows
[ https://issues.apache.org/jira/browse/HADOOP-9524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HADOOP-9524: Fix Version/s: (was: 3.0.0) 2.0.5-beta Fix ShellCommandFencer to work on Windows - Key: HADOOP-9524 URL: https://issues.apache.org/jira/browse/HADOOP-9524 Project: Hadoop Common Issue Type: Bug Components: ha Affects Versions: 3.0.0 Environment: Windows Reporter: Arpit Agarwal Assignee: Arpit Agarwal Labels: windows Fix For: 2.0.5-beta Attachments: HADOOP-9524.001.patch ShellcommandFencer has a hard-coded dependency on bash. Since we no longer require cygwin/bash on Windows we must fix it to use cmd.exe instead. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9287) Parallel testing hadoop-common
[
https://issues.apache.org/jira/browse/HADOOP-9287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666818#comment-13666818
]
Hadoop QA commented on HADOOP-9287:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584759/HADOOP-9287--N6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 49 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs
hadoop-tools/hadoop-distcp.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2571//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2571//console
This message is automatically generated.
Parallel testing hadoop-common
--
Key: HADOOP-9287
URL: https://issues.apache.org/jira/browse/HADOOP-9287
Project: Hadoop Common
Issue Type: Test
Components: test
Affects Versions: 3.0.0
Reporter: Tsuyoshi OZAWA
Assignee: Andrey Klochkov
Attachments: HADOOP-9287.1.patch, HADOOP-9287--N3.patch,
HADOOP-9287--N3.patch, HADOOP-9287--N4.patch, HADOOP-9287--N5.patch,
HADOOP-9287--N6.patch, HADOOP-9287.patch, HADOOP-9287.patch
The maven surefire plugin supports parallel testing feature. By using it, the
tests can be run more faster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9421) Convert SASL to use ProtoBuf and add lengths for non-blocking processing
[
https://issues.apache.org/jira/browse/HADOOP-9421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666823#comment-13666823
]
Hadoop QA commented on HADOOP-9421:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584769/HADOOP-9421.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2572//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2572//console
This message is automatically generated.
Convert SASL to use ProtoBuf and add lengths for non-blocking processing
Key: HADOOP-9421
URL: https://issues.apache.org/jira/browse/HADOOP-9421
Project: Hadoop Common
Issue Type: Sub-task
Affects Versions: 2.0.3-alpha
Reporter: Sanjay Radia
Assignee: Daryn Sharp
Attachments: HADOOP-9421.patch, HADOOP-9421.patch,
HADOOP-9421-v2-demo.patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9371) Define Semantics of FileSystem and FileContext more rigorously
[
https://issues.apache.org/jira/browse/HADOOP-9371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666838#comment-13666838
]
Konstantin Shvachko commented on HADOOP-9371:
-
Steve, you might want to link the document from github to the jira. Add Link
has an option to add a web link.
Not requiring atomicity for mkdirs() and recursive deletes makes sense to me.
For renames I think we should also restrict atomicity to one special case, when
file or directory name changes, that is file is not moving from one directory
to another. I call it in-place rename, which with inode numbers in place is a
trivial operation. Atomic moves are hard if you build a distributed namespace
service (like Giraffa). Moving a file between directories that are located on
different nodes requires distributed coordination, which can be complex.
Define Semantics of FileSystem and FileContext more rigorously
--
Key: HADOOP-9371
URL: https://issues.apache.org/jira/browse/HADOOP-9371
Project: Hadoop Common
Issue Type: Sub-task
Components: fs
Affects Versions: 1.2.0, 3.0.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Attachments: HADOOP-9361.2.patch, HADOOP-9361.patch,
HadoopFilesystemContract.pdf
Original Estimate: 48h
Remaining Estimate: 48h
The semantics of {{FileSystem}} and {{FileContext}} are not completely
defined in terms of
# core expectations of a filesystem
# consistency requirements.
# concurrency requirements.
# minimum scale limits
Furthermore, methods are not defined strictly enough in terms of their
outcomes and failure modes.
The requirements and method semantics should be defined more strictly.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9574) Add new methods in AbstractDelegationTokenSecretManager for restoring RMDelegationTokens on RMRestart
[
https://issues.apache.org/jira/browse/HADOOP-9574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666867#comment-13666867
]
Vinod Kumar Vavilapalli commented on HADOOP-9574:
-
Want to move forward with this for the sake of YARN-638.
Mostly looks good to me. Can you add tests to make sure these new APIs are
correctly called? You can modify TestDelegationTokenSecretManager.
Also, a couple of simple tests for DelegationKey changes will be great.
Add new methods in AbstractDelegationTokenSecretManager for restoring
RMDelegationTokens on RMRestart
-
Key: HADOOP-9574
URL: https://issues.apache.org/jira/browse/HADOOP-9574
Project: Hadoop Common
Issue Type: Bug
Reporter: Jian He
Assignee: Jian He
Attachments: HADOOP-9574.1.patch
we're considering to add the following methods in
AbstractDelegationTokenSecretManager for restoring RMDelegationTokens, these
methods can also possibly be reused by hdfsDelegationTokenSecretManager, see
YARN-638
protected void storeNewMasterKey(DelegationKey key) throws IOException {
return;
}
protected void removeStoredMasterKey(DelegationKey key) {
return;
}
protected void storeNewToken(TokenIdent ident, long renewDate) {
return;
}
protected void removeStoredToken(TokenIdent ident) throws IOException {
}
protected void updateStoredToken(TokenIdent ident, long renewDate) {
return;
}
Also move addPersistedDelegationToken in hdfs.DelegationTokenSecretManager,
to AbstractDelegationTokenSecretManager
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9599) hadoop-config.cmd doesn't set JAVA_LIBRARY_PATH correctly
[ https://issues.apache.org/jira/browse/HADOOP-9599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666884#comment-13666884 ] Chuan Liu commented on HADOOP-9599: --- Judging from the context, I think it should be HADOOP_COMMON_HOME instead of HADOOP_HOME. Because the purpose of the script is mainly to include JNI library, i.e. native library, in Java library path, and native library is a hadoop-common project. I think it is also good to add some comments as well. hadoop-config.cmd doesn't set JAVA_LIBRARY_PATH correctly - Key: HADOOP-9599 URL: https://issues.apache.org/jira/browse/HADOOP-9599 Project: Hadoop Common Issue Type: Bug Affects Versions: 3.0.0 Environment: Windows Reporter: Mostafa Elhemali Assignee: Mostafa Elhemali Attachments: HADOOP-9599.patch In Windows, hadoop-config.cmd uses the non-existent-variable HADOOP_CORE_HOME when setting the JAVA_LIBRAR_PATH variable. It should use HADOOP_HOME. The net effect is that running e.g. hdfs namenode would error out with UnsatisfiedLinkError because it can't access hadoop.dll. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9600) In Windows: Hadoop fails to run when JAVA_HOME has spaces in it
[
https://issues.apache.org/jira/browse/HADOOP-9600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666890#comment-13666890
]
Chuan Liu commented on HADOOP-9600:
---
Thanks for the patch!
Why we need the following change?
{noformat}
-for /f delims= %%A in ('%JAVA% -Xmx32m %HADOOP_JAVA_PLATFORM_OPTS%
-classpath %CLASSPATH% org.apache.hadoop.util.PlatformName') do set
JAVA_PLATFORM=%%A
+%JAVA% -Xmx32m %HADOOP_JAVA_PLATFORM_OPTS% -classpath %CLASSPATH%
org.apache.hadoop.util.PlatformName %TEMP%\java_platforms.txt
+set /p JAVA_PLATFORM=%TEMP%\java_platforms.txt
+del %TEMP%\java_platforms.txt
@rem replace space with underscore
set JAVA_PLATFORM=%JAVA_PLATFORM: =_%
{noformat}
In Windows: Hadoop fails to run when JAVA_HOME has spaces in it
---
Key: HADOOP-9600
URL: https://issues.apache.org/jira/browse/HADOOP-9600
Project: Hadoop Common
Issue Type: Bug
Environment: Windows
Reporter: Mostafa Elhemali
Assignee: Mostafa Elhemali
Attachments: HADOOP-9600.2.patch, HADOOP-9600.3.patch,
HADOOP-9600.patch
hadoop-config.cmd misbehaves when JAVA_HOME has spaces in it (e.g. if Java is
in c:\Program Files).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-8469) Make NetworkTopology class pluggable
[
https://issues.apache.org/jira/browse/HADOOP-8469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated HADOOP-8469:
---
Target Version/s: 2.0.5-beta (was: 2.0.3-alpha)
Status: Patch Available (was: Reopened)
Make NetworkTopology class pluggable
Key: HADOOP-8469
URL: https://issues.apache.org/jira/browse/HADOOP-8469
Project: Hadoop Common
Issue Type: Sub-task
Affects Versions: 2.0.0-alpha, 1.0.0
Reporter: Junping Du
Assignee: Junping Du
Fix For: 3.0.0
Attachments: HADOOP-8469-branch-2.patch,
HADOOP-8469-NetworkTopology-pluggable.patch,
HADOOP-8469-NetworkTopology-pluggable-v2.patch,
HADOOP-8469-NetworkTopology-pluggable-v3.patch,
HADOOP-8469-NetworkTopology-pluggable-v4.patch,
HADOOP-8469-NetworkTopology-pluggable-v5.patch
The class NetworkTopology is where the three-layer hierarchical topology is
modeled in the current code base and is instantiated directly by the
DatanodeManager and Balancer.
To support alternative topologies, changes were make the topology class
pluggable, that is to support using a user specified topology class specified
in the Hadoop configuration file core-defaul.xml. The user specified topology
class is instantiated using reflection in the same manner as other
customizable classes in Hadoop. If no use specified topology class is found,
the fallback is to use the NetworkTopology to preserve current behavior. To
make it possible to reuse code in NetworkTopology several minor changes were
made to make the class more extensible. The NetworkTopology class is
currently annotated with @InterfaceAudience.LimitedPrivate({HDFS,
MapReduce}) and @InterfaceStability.Unstable.
The proposed changes in NetworkTopology listed below
1. Some fields were changes from private to protected
2. Added some protected methods so that sub classes could override behavior
3. Added a new method,isNodeGroupAware,to NetworkTopology
4. The inner class InnerNode was made a package protected class to it would
be easier to subclass
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8469) Make NetworkTopology class pluggable
[
https://issues.apache.org/jira/browse/HADOOP-8469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666901#comment-13666901
]
Hadoop QA commented on HADOOP-8469:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12584625/HADOOP-8469-branch-2.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/2573//console
This message is automatically generated.
Make NetworkTopology class pluggable
Key: HADOOP-8469
URL: https://issues.apache.org/jira/browse/HADOOP-8469
Project: Hadoop Common
Issue Type: Sub-task
Affects Versions: 1.0.0, 2.0.0-alpha
Reporter: Junping Du
Assignee: Junping Du
Fix For: 3.0.0
Attachments: HADOOP-8469-branch-2.patch,
HADOOP-8469-NetworkTopology-pluggable.patch,
HADOOP-8469-NetworkTopology-pluggable-v2.patch,
HADOOP-8469-NetworkTopology-pluggable-v3.patch,
HADOOP-8469-NetworkTopology-pluggable-v4.patch,
HADOOP-8469-NetworkTopology-pluggable-v5.patch
The class NetworkTopology is where the three-layer hierarchical topology is
modeled in the current code base and is instantiated directly by the
DatanodeManager and Balancer.
To support alternative topologies, changes were make the topology class
pluggable, that is to support using a user specified topology class specified
in the Hadoop configuration file core-defaul.xml. The user specified topology
class is instantiated using reflection in the same manner as other
customizable classes in Hadoop. If no use specified topology class is found,
the fallback is to use the NetworkTopology to preserve current behavior. To
make it possible to reuse code in NetworkTopology several minor changes were
made to make the class more extensible. The NetworkTopology class is
currently annotated with @InterfaceAudience.LimitedPrivate({HDFS,
MapReduce}) and @InterfaceStability.Unstable.
The proposed changes in NetworkTopology listed below
1. Some fields were changes from private to protected
2. Added some protected methods so that sub classes could override behavior
3. Added a new method,isNodeGroupAware,to NetworkTopology
4. The inner class InnerNode was made a package protected class to it would
be easier to subclass
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9597) FileSystem open() API is not clear if FileNotFoundException is throw when the path does not exist
[ https://issues.apache.org/jira/browse/HADOOP-9597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated HADOOP-9597: --- Fix Version/s: (was: 2.0.4.1-alpha) FileSystem open() API is not clear if FileNotFoundException is throw when the path does not exist - Key: HADOOP-9597 URL: https://issues.apache.org/jira/browse/HADOOP-9597 Project: Hadoop Common Issue Type: Sub-task Components: documentation, fs Affects Versions: 2.0.4-alpha Reporter: Jerry He Priority: Minor The current FileSystem open() method throws a generic IOException in its API specification. Some FileSystem implementations (DFS, RawLocalFileSystem ...) throws more specific FileNotFoundException if the path does not exist. Some throws IOException only (FTPFileSystem, HftpFileSystem ...). If we have a new FileSystem implementation, what should we follow exactly for open()? What should the application expect in this case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9407) commons-daemon 1.0.3 dependency has bad group id causing build issues
[ https://issues.apache.org/jira/browse/HADOOP-9407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated HADOOP-9407: --- Resolution: Fixed Status: Resolved (was: Patch Available) Fixed in both targetted releases. CLosing. commons-daemon 1.0.3 dependency has bad group id causing build issues - Key: HADOOP-9407 URL: https://issues.apache.org/jira/browse/HADOOP-9407 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 2.0.3-alpha, 2.0.4-alpha Reporter: Sangjin Lee Assignee: Sangjin Lee Labels: 2.0.4 Fix For: 2.0.5-beta, 2.0.4.1-alpha Attachments: HDFS-4497.patch The commons-daemon dependency of the hadoop-hdfs module has been at version 1.0.3 for a while. However, 1.0.3 has a pretty well-known groupId error in its pom (org.apache.commons as opposed to commons-daemon). This problem has since been corrected on commons-daemon starting 1.0.4. This causes build problems for many who depend on hadoop-hdfs directly and indirectly, however. Maven can skip over this metadata inconsistency. But other less forgiving build systems such as ivy and gradle have much harder time working around this problem. For example, in gradle, pretty much the only obvious way to work around this is to override this dependency version. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9579) Contrib ant test target not setting the java.library.path
[
https://issues.apache.org/jira/browse/HADOOP-9579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666949#comment-13666949
]
Chris Nauroth commented on HADOOP-9579:
---
Chuan, my plan was to try running {{TestGridmixSubmission}} and check logs to
see that {{NativeCodeLoader}} found the library. Unfortunately, it failed with:
{quote}
13/05/24 10:41:00 ERROR util.Shell: Failed to detect a valid hadoop home directo
ry
java.io.IOException: HADOOP_HOME or hadoop.home.dir are not set.
{quote}
Specifically, I was running ant -Dwindows=true
-Dtestcase=TestGridmixSubmission test-contrib from the root of the
branch-1-win repo. Ivan, is there something else that we need to do to verify
this, or does this look like a problem?
Contrib ant test target not setting the java.library.path
-
Key: HADOOP-9579
URL: https://issues.apache.org/jira/browse/HADOOP-9579
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 1-win
Reporter: Ivan Mitic
Assignee: Ivan Mitic
Attachments: HADOOP-9579.branch-1-win.patch
build-contrib.xml does not set java.library.path causing tests to run without
the hadoop native library. This is a bigger problem on Windows as having the
native library is required.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9579) Contrib ant test target not setting the java.library.path
[ https://issues.apache.org/jira/browse/HADOOP-9579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Mitic updated HADOOP-9579: --- Attachment: HADOOP-9579.branch-1-win.2.patch Attaching the updated patch. Contrib ant test target not setting the java.library.path - Key: HADOOP-9579 URL: https://issues.apache.org/jira/browse/HADOOP-9579 Project: Hadoop Common Issue Type: Bug Affects Versions: 1-win Reporter: Ivan Mitic Assignee: Ivan Mitic Attachments: HADOOP-9579.branch-1-win.2.patch, HADOOP-9579.branch-1-win.patch build-contrib.xml does not set java.library.path causing tests to run without the hadoop native library. This is a bigger problem on Windows as having the native library is required. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9579) Contrib ant test target not setting the java.library.path
[ https://issues.apache.org/jira/browse/HADOOP-9579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666956#comment-13666956 ] Ivan Mitic commented on HADOOP-9579: Thanks Chuan and Chris for reviewing! Bq. Specifically, I was running ant -Dwindows=true -Dtestcase=TestGridmixSubmission test-contrib from the root of the branch-1-win repo. Ivan, is there something else that we need to do to verify this, or does this look like a problem? I just took a quick look and indeed there is another (slightly orthogonal) problem here. Basically HADOOP_HOME is not set when contrib tests are run resulting in the above error. Looks like we did not catch this problem earlier because we set HADOOP_HOME as part of our build environment initialization. I went ahead and fixed this as part of this Jira since it’s also related to contrib tests on Windows (v2 patch). Good catch Chris! Contrib ant test target not setting the java.library.path - Key: HADOOP-9579 URL: https://issues.apache.org/jira/browse/HADOOP-9579 Project: Hadoop Common Issue Type: Bug Affects Versions: 1-win Reporter: Ivan Mitic Assignee: Ivan Mitic Attachments: HADOOP-9579.branch-1-win.2.patch, HADOOP-9579.branch-1-win.patch build-contrib.xml does not set java.library.path causing tests to run without the hadoop native library. This is a bigger problem on Windows as having the native library is required. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8562) Enhancements to support Hadoop on Windows Server and Windows Azure environments
[ https://issues.apache.org/jira/browse/HADOOP-8562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13666964#comment-13666964 ] Hudson commented on HADOOP-8562: Integrated in Hadoop-trunk-Commit #3790 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3790/]) Update hadoop-common CHANGES.txt after merging HADOOP-8562 (Revision 1486288) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1486288 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt Enhancements to support Hadoop on Windows Server and Windows Azure environments --- Key: HADOOP-8562 URL: https://issues.apache.org/jira/browse/HADOOP-8562 Project: Hadoop Common Issue Type: New Feature Affects Versions: 3.0.0 Reporter: Bikas Saha Assignee: Bikas Saha Fix For: 2.0.5-beta Attachments: branch-2.merge.patch, branch-2.merge.patch, branch-trunk-win.min-notest.patch, branch-trunk-win-min.patch, branch-trunk-win.min.patch, branch-trunk-win.patch, branch-trunk-win.patch, branch-trunk-win.patch, branch-trunk-win.patch, branch-trunk-win.patch, branch-trunk-win.patch, branch-trunk-win.patch, branch-trunk-win.patch, branch-trunk-win.patch, branch-trunk-win.patch, branch-trunk-win.patch, test-untar.tar, test-untar.tgz This JIRA tracks the work that needs to be done on trunk to enable Hadoop to run on Windows Server and Azure environments. This incorporates porting relevant work from the similar effort on branch 1 tracked via HADOOP-8079. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
