[jira] [Updated] (HADOOP-9459) ActiveStandbyElector can join election even before Service HEALTHY, and results in null data at ActiveBreadCrumb
[ https://issues.apache.org/jira/browse/HADOOP-9459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HADOOP-9459: Resolution: Fixed Fix Version/s: 2.0.5-beta 3.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk and branch-2. Thanks for tracking down this tricky bug, Vinay ActiveStandbyElector can join election even before Service HEALTHY, and results in null data at ActiveBreadCrumb Key: HADOOP-9459 URL: https://issues.apache.org/jira/browse/HADOOP-9459 Project: Hadoop Common Issue Type: Bug Components: ha Affects Versions: 2.0.2-alpha Reporter: Vinay Assignee: Vinay Priority: Critical Fix For: 3.0.0, 2.0.5-beta Attachments: HDFS-4463.patch, hdfs-4463.txt ActiveStandbyElector can store null at ActiveBreadCrumb in the below race condition. At further all failovers will fail resulting NPE. 1. ZKFC restarted. 2. due to machine busy, first zk connection is expired even before the health monitoring returned the status. 3. On re-establishment transitionToActive will be called, at this time appData will be null, 4. So now ActiveBreadCrumb will have null. 5. After this any failovers will fail throwing {noformat}java.lang.NullPointerException at org.apache.hadoop.util.StringUtils.byteToHexString(StringUtils.java:171) at org.apache.hadoop.ha.ActiveStandbyElector.fenceOldActive(ActiveStandbyElector.java:892) at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:797) at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:475) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:545) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497){noformat} Should not join the election before service is HEALTHY -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9307) BufferedFSInputStream.read returns wrong results after certain seeks
[ https://issues.apache.org/jira/browse/HADOOP-9307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656861#comment-13656861 ] Todd Lipcon commented on HADOOP-9307: - Hey Steve. I agree that improving the general cross-filesystem testing is a worthy goal. But, this is a simple bug in an existing implementation, and the patch adds a specific unit test. Given that this breaks HBase running on the local filesystem, I don't think it makes sense to block fixing it on a much bigger project like standardizing tests. BufferedFSInputStream.read returns wrong results after certain seeks Key: HADOOP-9307 URL: https://issues.apache.org/jira/browse/HADOOP-9307 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 1.1.1, 2.0.2-alpha Reporter: Todd Lipcon Assignee: Todd Lipcon Attachments: hadoop-9307.txt After certain sequences of seek/read, BufferedFSInputStream can silently return data from the wrong part of the file. Further description in first comment below. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9307) BufferedFSInputStream.read returns wrong results after certain seeks
[ https://issues.apache.org/jira/browse/HADOOP-9307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656872#comment-13656872 ] Harsh J commented on HADOOP-9307: - +1 - The change and the added regression test looks good. I tested it without the fix as well. Nice find Todd! BufferedFSInputStream.read returns wrong results after certain seeks Key: HADOOP-9307 URL: https://issues.apache.org/jira/browse/HADOOP-9307 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 1.1.1, 2.0.2-alpha Reporter: Todd Lipcon Assignee: Todd Lipcon Attachments: hadoop-9307.txt After certain sequences of seek/read, BufferedFSInputStream can silently return data from the wrong part of the file. Further description in first comment below. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9435) Minimal code change to support IBM jvm build dependency
[ https://issues.apache.org/jira/browse/HADOOP-9435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tian Hong Wang updated HADOOP-9435: --- Summary: Minimal code change to support IBM jvm build dependency (was: Native build hadoop-common-project fails on $JAVA_HOME/include/jni_md.h using ibm java) Minimal code change to support IBM jvm build dependency --- Key: HADOOP-9435 URL: https://issues.apache.org/jira/browse/HADOOP-9435 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 2.0.3-alpha Reporter: Tian Hong Wang Labels: patch Fix For: 2.0.3-alpha Attachments: HADOOP-9435.patch When native build hadoop-common-project with IBM java using command like: mvn package -Pnative it will exist the following errors. [exec] -- Configuring incomplete, errors occurred! [exec] JAVA_HOME=, JAVA_JVM_LIBRARY=/home/louis/ibm-java-i386-60/jre/lib/i386/classic/libjvm.so [exec] JAVA_INCLUDE_PATH=/home/louis/ibm-java-i386-60/include, JAVA_INCLUDE_PATH2=JAVA_INCLUDE_PATH2-NOTFOUND [exec] CMake Error at JNIFlags.cmake:113 (MESSAGE): [exec] Failed to find a viable JVM installation under JAVA_HOME. [exec] Call Stack (most recent call first): [exec] CMakeLists.txt:24 (include) The reason is that IBM java uses $JAVA_HOME/include/jniport.h instead of $JAVA_HOME/include/jni_md.h in Oracle java. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9435) Minimal code change to support IBM jvm build dependency
[ https://issues.apache.org/jira/browse/HADOOP-9435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tian Hong Wang updated HADOOP-9435: --- Description: When native build hadoop-common-project with IBM java using command like: mvn package -Pnative it will exist the following errors. [exec] -- Configuring incomplete, errors occurred! [exec] JAVA_HOME=, JAVA_JVM_LIBRARY=/home/louis/ibm-java-i386-60/jre/lib/i386/classic/libjvm.so [exec] JAVA_INCLUDE_PATH=/home/louis/ibm-java-i386-60/include, JAVA_INCLUDE_PATH2=JAVA_INCLUDE_PATH2-NOTFOUND [exec] CMake Error at JNIFlags.cmake:113 (MESSAGE): [exec] Failed to find a viable JVM installation under JAVA_HOME. [exec] Call Stack (most recent call first): [exec] CMakeLists.txt:24 (include) The reason is that IBM java uses $JAVA_HOME/include/jniport.h instead of $JAVA_HOME/include/jni_md.h in non-IBM java. [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlsym' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlerror' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dladdr' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlopen' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlclose' [exec] collect2: ld returned 1 exit status [exec] make[2]: *** [test_libhdfs_ops] Error 1 [exec] make[1]: *** [CMakeFiles/test_libhdfs_ops.dir/all] Error 2 [exec] make: *** [all] Error The reason is libjvm.so need libdl when linking. was: When native build hadoop-common-project with IBM java using command like: mvn package -Pnative it will exist the following errors. [exec] -- Configuring incomplete, errors occurred! [exec] JAVA_HOME=, JAVA_JVM_LIBRARY=/home/louis/ibm-java-i386-60/jre/lib/i386/classic/libjvm.so [exec] JAVA_INCLUDE_PATH=/home/louis/ibm-java-i386-60/include, JAVA_INCLUDE_PATH2=JAVA_INCLUDE_PATH2-NOTFOUND [exec] CMake Error at JNIFlags.cmake:113 (MESSAGE): [exec] Failed to find a viable JVM installation under JAVA_HOME. [exec] Call Stack (most recent call first): [exec] CMakeLists.txt:24 (include) The reason is that IBM java uses $JAVA_HOME/include/jniport.h instead of $JAVA_HOME/include/jni_md.h in Oracle java. Minimal code change to support IBM jvm build dependency --- Key: HADOOP-9435 URL: https://issues.apache.org/jira/browse/HADOOP-9435 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 2.0.3-alpha Reporter: Tian Hong Wang Labels: patch Fix For: 2.0.3-alpha Attachments: HADOOP-9435.patch When native build hadoop-common-project with IBM java using command like: mvn package -Pnative it will exist the following errors. [exec] -- Configuring incomplete, errors occurred! [exec] JAVA_HOME=, JAVA_JVM_LIBRARY=/home/louis/ibm-java-i386-60/jre/lib/i386/classic/libjvm.so [exec] JAVA_INCLUDE_PATH=/home/louis/ibm-java-i386-60/include, JAVA_INCLUDE_PATH2=JAVA_INCLUDE_PATH2-NOTFOUND [exec] CMake Error at JNIFlags.cmake:113 (MESSAGE): [exec] Failed to find a viable JVM installation under JAVA_HOME. [exec] Call Stack (most recent call first): [exec] CMakeLists.txt:24 (include) The reason is that IBM java uses $JAVA_HOME/include/jniport.h instead of $JAVA_HOME/include/jni_md.h in non-IBM java. [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlsym' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlerror' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dladdr' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlopen' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlclose' [exec] collect2: ld returned 1 exit status [exec] make[2]: *** [test_libhdfs_ops] Error 1 [exec] make[1]: *** [CMakeFiles/test_libhdfs_ops.dir/all] Error 2 [exec] make: *** [all] Error The reason is libjvm.so need libdl when linking. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9435) Minimal code change to support IBM jvm build dependency
[ https://issues.apache.org/jira/browse/HADOOP-9435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tian Hong Wang updated HADOOP-9435: --- Status: Open (was: Patch Available) Minimal code change to support IBM jvm build dependency --- Key: HADOOP-9435 URL: https://issues.apache.org/jira/browse/HADOOP-9435 Project: Hadoop Common Issue Type: Bug Components: build Affects Versions: 2.0.3-alpha Reporter: Tian Hong Wang Labels: patch Fix For: 2.0.3-alpha Attachments: HADOOP-9435.patch When native build hadoop-common-project with IBM java using command like: mvn package -Pnative it will exist the following errors. [exec] -- Configuring incomplete, errors occurred! [exec] JAVA_HOME=, JAVA_JVM_LIBRARY=/home/louis/ibm-java-i386-60/jre/lib/i386/classic/libjvm.so [exec] JAVA_INCLUDE_PATH=/home/louis/ibm-java-i386-60/include, JAVA_INCLUDE_PATH2=JAVA_INCLUDE_PATH2-NOTFOUND [exec] CMake Error at JNIFlags.cmake:113 (MESSAGE): [exec] Failed to find a viable JVM installation under JAVA_HOME. [exec] Call Stack (most recent call first): [exec] CMakeLists.txt:24 (include) The reason is that IBM java uses $JAVA_HOME/include/jniport.h instead of $JAVA_HOME/include/jni_md.h in non-IBM java. [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlsym' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlerror' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dladdr' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlopen' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlclose' [exec] collect2: ld returned 1 exit status [exec] make[2]: *** [test_libhdfs_ops] Error 1 [exec] make[1]: *** [CMakeFiles/test_libhdfs_ops.dir/all] Error 2 [exec] make: *** [all] Error The reason is libjvm.so need libdl when linking. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9435) Minimal code change to support IBM jvm build dependency
[ https://issues.apache.org/jira/browse/HADOOP-9435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tian Hong Wang updated HADOOP-9435: --- Affects Version/s: (was: 2.0.3-alpha) Minimal code change to support IBM jvm build dependency --- Key: HADOOP-9435 URL: https://issues.apache.org/jira/browse/HADOOP-9435 Project: Hadoop Common Issue Type: Bug Components: build Reporter: Tian Hong Wang Labels: patch Fix For: 2.0.3-alpha Attachments: HADOOP-9435.patch When native build hadoop-common-project with IBM java using command like: mvn package -Pnative it will exist the following errors. [exec] -- Configuring incomplete, errors occurred! [exec] JAVA_HOME=, JAVA_JVM_LIBRARY=/home/louis/ibm-java-i386-60/jre/lib/i386/classic/libjvm.so [exec] JAVA_INCLUDE_PATH=/home/louis/ibm-java-i386-60/include, JAVA_INCLUDE_PATH2=JAVA_INCLUDE_PATH2-NOTFOUND [exec] CMake Error at JNIFlags.cmake:113 (MESSAGE): [exec] Failed to find a viable JVM installation under JAVA_HOME. [exec] Call Stack (most recent call first): [exec] CMakeLists.txt:24 (include) The reason is that IBM java uses $JAVA_HOME/include/jniport.h instead of $JAVA_HOME/include/jni_md.h in non-IBM java. [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlsym' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlerror' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dladdr' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlopen' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlclose' [exec] collect2: ld returned 1 exit status [exec] make[2]: *** [test_libhdfs_ops] Error 1 [exec] make[1]: *** [CMakeFiles/test_libhdfs_ops.dir/all] Error 2 [exec] make: *** [all] Error The reason is libjvm.so need libdl when linking. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9435) Minimal code change to support IBM jvm build dependency
[ https://issues.apache.org/jira/browse/HADOOP-9435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tian Hong Wang updated HADOOP-9435: --- Fix Version/s: (was: 2.0.3-alpha) Minimal code change to support IBM jvm build dependency --- Key: HADOOP-9435 URL: https://issues.apache.org/jira/browse/HADOOP-9435 Project: Hadoop Common Issue Type: Bug Components: build Reporter: Tian Hong Wang Labels: patch Attachments: HADOOP-9435.patch When native build hadoop-common-project with IBM java using command like: mvn package -Pnative it will exist the following errors. [exec] -- Configuring incomplete, errors occurred! [exec] JAVA_HOME=, JAVA_JVM_LIBRARY=/home/louis/ibm-java-i386-60/jre/lib/i386/classic/libjvm.so [exec] JAVA_INCLUDE_PATH=/home/louis/ibm-java-i386-60/include, JAVA_INCLUDE_PATH2=JAVA_INCLUDE_PATH2-NOTFOUND [exec] CMake Error at JNIFlags.cmake:113 (MESSAGE): [exec] Failed to find a viable JVM installation under JAVA_HOME. [exec] Call Stack (most recent call first): [exec] CMakeLists.txt:24 (include) The reason is that IBM java uses $JAVA_HOME/include/jniport.h instead of $JAVA_HOME/include/jni_md.h in non-IBM java. [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlsym' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlerror' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dladdr' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlopen' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlclose' [exec] collect2: ld returned 1 exit status [exec] make[2]: *** [test_libhdfs_ops] Error 1 [exec] make[1]: *** [CMakeFiles/test_libhdfs_ops.dir/all] Error 2 [exec] make: *** [all] Error The reason is libjvm.so need libdl when linking. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HADOOP-9435) Minimal code change to support IBM jvm build dependency
[ https://issues.apache.org/jira/browse/HADOOP-9435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tian Hong Wang reassigned HADOOP-9435: -- Assignee: Tian Hong Wang Minimal code change to support IBM jvm build dependency --- Key: HADOOP-9435 URL: https://issues.apache.org/jira/browse/HADOOP-9435 Project: Hadoop Common Issue Type: Bug Components: build Reporter: Tian Hong Wang Assignee: Tian Hong Wang Labels: patch Attachments: HADOOP-9435.patch When native build hadoop-common-project with IBM java using command like: mvn package -Pnative it will exist the following errors. [exec] -- Configuring incomplete, errors occurred! [exec] JAVA_HOME=, JAVA_JVM_LIBRARY=/home/louis/ibm-java-i386-60/jre/lib/i386/classic/libjvm.so [exec] JAVA_INCLUDE_PATH=/home/louis/ibm-java-i386-60/include, JAVA_INCLUDE_PATH2=JAVA_INCLUDE_PATH2-NOTFOUND [exec] CMake Error at JNIFlags.cmake:113 (MESSAGE): [exec] Failed to find a viable JVM installation under JAVA_HOME. [exec] Call Stack (most recent call first): [exec] CMakeLists.txt:24 (include) The reason is that IBM java uses $JAVA_HOME/include/jniport.h instead of $JAVA_HOME/include/jni_md.h in non-IBM java. [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlsym' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlerror' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dladdr' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlopen' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlclose' [exec] collect2: ld returned 1 exit status [exec] make[2]: *** [test_libhdfs_ops] Error 1 [exec] make[1]: *** [CMakeFiles/test_libhdfs_ops.dir/all] Error 2 [exec] make: *** [all] Error The reason is libjvm.so need libdl when linking. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9435) Minimal code change to support IBM jvm build dependency
[ https://issues.apache.org/jira/browse/HADOOP-9435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tian Hong Wang updated HADOOP-9435: --- Attachment: HADOOP-9435-v1.patch Resubmit patch to resolve two build dependency issues using IBM jvm. Minimal code change to support IBM jvm build dependency --- Key: HADOOP-9435 URL: https://issues.apache.org/jira/browse/HADOOP-9435 Project: Hadoop Common Issue Type: Bug Components: build Reporter: Tian Hong Wang Assignee: Tian Hong Wang Labels: patch Attachments: HADOOP-9435.patch, HADOOP-9435-v1.patch When native build hadoop-common-project with IBM java using command like: mvn package -Pnative it will exist the following errors. [exec] -- Configuring incomplete, errors occurred! [exec] JAVA_HOME=, JAVA_JVM_LIBRARY=/home/louis/ibm-java-i386-60/jre/lib/i386/classic/libjvm.so [exec] JAVA_INCLUDE_PATH=/home/louis/ibm-java-i386-60/include, JAVA_INCLUDE_PATH2=JAVA_INCLUDE_PATH2-NOTFOUND [exec] CMake Error at JNIFlags.cmake:113 (MESSAGE): [exec] Failed to find a viable JVM installation under JAVA_HOME. [exec] Call Stack (most recent call first): [exec] CMakeLists.txt:24 (include) The reason is that IBM java uses $JAVA_HOME/include/jniport.h instead of $JAVA_HOME/include/jni_md.h in non-IBM java. [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlsym' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlerror' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dladdr' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlopen' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlclose' [exec] collect2: ld returned 1 exit status [exec] make[2]: *** [test_libhdfs_ops] Error 1 [exec] make[1]: *** [CMakeFiles/test_libhdfs_ops.dir/all] Error 2 [exec] make: *** [all] Error The reason is libjvm.so need libdl when linking. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9435) Minimal code change to support IBM jvm build dependency
[ https://issues.apache.org/jira/browse/HADOOP-9435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tian Hong Wang updated HADOOP-9435: --- Status: Patch Available (was: Open) Minimal code change to support IBM jvm build dependency --- Key: HADOOP-9435 URL: https://issues.apache.org/jira/browse/HADOOP-9435 Project: Hadoop Common Issue Type: Bug Components: build Reporter: Tian Hong Wang Assignee: Tian Hong Wang Labels: patch Attachments: HADOOP-9435.patch, HADOOP-9435-v1.patch When native build hadoop-common-project with IBM java using command like: mvn package -Pnative it will exist the following errors. [exec] -- Configuring incomplete, errors occurred! [exec] JAVA_HOME=, JAVA_JVM_LIBRARY=/home/louis/ibm-java-i386-60/jre/lib/i386/classic/libjvm.so [exec] JAVA_INCLUDE_PATH=/home/louis/ibm-java-i386-60/include, JAVA_INCLUDE_PATH2=JAVA_INCLUDE_PATH2-NOTFOUND [exec] CMake Error at JNIFlags.cmake:113 (MESSAGE): [exec] Failed to find a viable JVM installation under JAVA_HOME. [exec] Call Stack (most recent call first): [exec] CMakeLists.txt:24 (include) The reason is that IBM java uses $JAVA_HOME/include/jniport.h instead of $JAVA_HOME/include/jni_md.h in non-IBM java. [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlsym' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlerror' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dladdr' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlopen' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlclose' [exec] collect2: ld returned 1 exit status [exec] make[2]: *** [test_libhdfs_ops] Error 1 [exec] make[1]: *** [CMakeFiles/test_libhdfs_ops.dir/all] Error 2 [exec] make: *** [all] Error The reason is libjvm.so need libdl when linking. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9046) provide unit-test coverage of class org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT
[ https://issues.apache.org/jira/browse/HADOOP-9046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan A. Veselovsky updated HADOOP-9046: --- Affects Version/s: (was: 2.0.3-alpha) (was: 3.0.0) provide unit-test coverage of class org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT -- Key: HADOOP-9046 URL: https://issues.apache.org/jira/browse/HADOOP-9046 Project: Hadoop Common Issue Type: Test Affects Versions: 0.23.6 Reporter: Ivan A. Veselovsky Assignee: Ivan A. Veselovsky Priority: Minor Attachments: HADOOP-9046-branch-0.23--c.patch, HADOOP-9046-branch-0.23--d.patch, HADOOP-9046-branch-0.23-over-9049.patch, HADOOP-9046-branch-0.23--over-HDFS-4567.patch, HADOOP-9046-branch-0.23.patch, HADOOP-9046-branch-2--over-HDFS-4567.patch, HADOOP-9046--c.patch, HADOOP-9046--d.patch, HADOOP-9046--e.patch, HADOOP-9046-over-9049.patch, HADOOP-9046.patch, HADOOP-9046-trunk--over-HDFS-4567.patch The class org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT has zero coverage in entire cumulative test run. Provide test(s) to cover this class. Note: the request submitted to HDFS project because the class likely to be tested by tests in that project. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9046) provide unit-test coverage of class org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT
[ https://issues.apache.org/jira/browse/HADOOP-9046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656918#comment-13656918 ] Ivan A. Veselovsky commented on HADOOP-9046: Since fix https://issues.apache.org/jira/browse/HADOOP-9549 the patch is no longer applicable to branches trunk and branch-2 (needs merge). It looks like there is no big reason to merge over 9549 because after the fix 9549 the coverage of the target classes is enough. So, the suggested patch is now applicable to branch-0.23 only. Modifying the Affected versions field accordingly. provide unit-test coverage of class org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT -- Key: HADOOP-9046 URL: https://issues.apache.org/jira/browse/HADOOP-9046 Project: Hadoop Common Issue Type: Test Affects Versions: 0.23.6 Reporter: Ivan A. Veselovsky Assignee: Ivan A. Veselovsky Priority: Minor Attachments: HADOOP-9046-branch-0.23--c.patch, HADOOP-9046-branch-0.23--d.patch, HADOOP-9046-branch-0.23-over-9049.patch, HADOOP-9046-branch-0.23--over-HDFS-4567.patch, HADOOP-9046-branch-0.23.patch, HADOOP-9046-branch-2--over-HDFS-4567.patch, HADOOP-9046--c.patch, HADOOP-9046--d.patch, HADOOP-9046--e.patch, HADOOP-9046-over-9049.patch, HADOOP-9046.patch, HADOOP-9046-trunk--over-HDFS-4567.patch The class org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT has zero coverage in entire cumulative test run. Provide test(s) to cover this class. Note: the request submitted to HDFS project because the class likely to be tested by tests in that project. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9220) Unnecessary transition to standby in ActiveStandbyElector
[ https://issues.apache.org/jira/browse/HADOOP-9220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656937#comment-13656937 ] Tom White commented on HADOOP-9220: --- +1 for Todd's patch. I tested it manually and the original problem no longer occurs with the patch. Nit: did you mean to introduce {{new Exception}} in the debug line? Unnecessary transition to standby in ActiveStandbyElector - Key: HADOOP-9220 URL: https://issues.apache.org/jira/browse/HADOOP-9220 Project: Hadoop Common Issue Type: Bug Components: ha Reporter: Tom White Assignee: Tom White Priority: Critical Attachments: HADOOP-9220.patch, HADOOP-9220.patch, hadoop-9220.txt When performing a manual failover from one HA node to a second, under some circumstances the second node will transition from standby - active - standby - active. This is with automatic failover enabled, so there is a ZK cluster doing leader election. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9459) ActiveStandbyElector can join election even before Service HEALTHY, and results in null data at ActiveBreadCrumb
[ https://issues.apache.org/jira/browse/HADOOP-9459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656952#comment-13656952 ] Hudson commented on HADOOP-9459: Integrated in Hadoop-Yarn-trunk #209 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/209/]) HADOOP-9459. ActiveStandbyElector can join election even before Service HEALTHY, and results in null data at ActiveBreadCrumb. Contributed by Vinay and Todd Lipcon. (Revision 1482227) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1482227 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ActiveStandbyElector.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestActiveStandbyElector.java ActiveStandbyElector can join election even before Service HEALTHY, and results in null data at ActiveBreadCrumb Key: HADOOP-9459 URL: https://issues.apache.org/jira/browse/HADOOP-9459 Project: Hadoop Common Issue Type: Bug Components: ha Affects Versions: 2.0.2-alpha Reporter: Vinay Assignee: Vinay Priority: Critical Fix For: 3.0.0, 2.0.5-beta Attachments: HDFS-4463.patch, hdfs-4463.txt ActiveStandbyElector can store null at ActiveBreadCrumb in the below race condition. At further all failovers will fail resulting NPE. 1. ZKFC restarted. 2. due to machine busy, first zk connection is expired even before the health monitoring returned the status. 3. On re-establishment transitionToActive will be called, at this time appData will be null, 4. So now ActiveBreadCrumb will have null. 5. After this any failovers will fail throwing {noformat}java.lang.NullPointerException at org.apache.hadoop.util.StringUtils.byteToHexString(StringUtils.java:171) at org.apache.hadoop.ha.ActiveStandbyElector.fenceOldActive(ActiveStandbyElector.java:892) at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:797) at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:475) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:545) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497){noformat} Should not join the election before service is HEALTHY -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9435) Minimal code change to support IBM jvm build dependency
[ https://issues.apache.org/jira/browse/HADOOP-9435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13656965#comment-13656965 ] Hadoop QA commented on HADOOP-9435: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12583101/HADOOP-9435-v1.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2540//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2540//console This message is automatically generated. Minimal code change to support IBM jvm build dependency --- Key: HADOOP-9435 URL: https://issues.apache.org/jira/browse/HADOOP-9435 Project: Hadoop Common Issue Type: Bug Components: build Reporter: Tian Hong Wang Assignee: Tian Hong Wang Labels: patch Attachments: HADOOP-9435.patch, HADOOP-9435-v1.patch When native build hadoop-common-project with IBM java using command like: mvn package -Pnative it will exist the following errors. [exec] -- Configuring incomplete, errors occurred! [exec] JAVA_HOME=, JAVA_JVM_LIBRARY=/home/louis/ibm-java-i386-60/jre/lib/i386/classic/libjvm.so [exec] JAVA_INCLUDE_PATH=/home/louis/ibm-java-i386-60/include, JAVA_INCLUDE_PATH2=JAVA_INCLUDE_PATH2-NOTFOUND [exec] CMake Error at JNIFlags.cmake:113 (MESSAGE): [exec] Failed to find a viable JVM installation under JAVA_HOME. [exec] Call Stack (most recent call first): [exec] CMakeLists.txt:24 (include) The reason is that IBM java uses $JAVA_HOME/include/jniport.h instead of $JAVA_HOME/include/jni_md.h in non-IBM java. [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlsym' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlerror' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dladdr' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlopen' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlclose' [exec] collect2: ld returned 1 exit status [exec] make[2]: *** [test_libhdfs_ops] Error 1 [exec] make[1]: *** [CMakeFiles/test_libhdfs_ops.dir/all] Error 2 [exec] make: *** [all] Error The reason is libjvm.so need libdl when linking. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9459) ActiveStandbyElector can join election even before Service HEALTHY, and results in null data at ActiveBreadCrumb
[ https://issues.apache.org/jira/browse/HADOOP-9459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657035#comment-13657035 ] Hudson commented on HADOOP-9459: Integrated in Hadoop-Hdfs-trunk #1398 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1398/]) HADOOP-9459. ActiveStandbyElector can join election even before Service HEALTHY, and results in null data at ActiveBreadCrumb. Contributed by Vinay and Todd Lipcon. (Revision 1482227) Result = FAILURE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1482227 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ActiveStandbyElector.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestActiveStandbyElector.java ActiveStandbyElector can join election even before Service HEALTHY, and results in null data at ActiveBreadCrumb Key: HADOOP-9459 URL: https://issues.apache.org/jira/browse/HADOOP-9459 Project: Hadoop Common Issue Type: Bug Components: ha Affects Versions: 2.0.2-alpha Reporter: Vinay Assignee: Vinay Priority: Critical Fix For: 3.0.0, 2.0.5-beta Attachments: HDFS-4463.patch, hdfs-4463.txt ActiveStandbyElector can store null at ActiveBreadCrumb in the below race condition. At further all failovers will fail resulting NPE. 1. ZKFC restarted. 2. due to machine busy, first zk connection is expired even before the health monitoring returned the status. 3. On re-establishment transitionToActive will be called, at this time appData will be null, 4. So now ActiveBreadCrumb will have null. 5. After this any failovers will fail throwing {noformat}java.lang.NullPointerException at org.apache.hadoop.util.StringUtils.byteToHexString(StringUtils.java:171) at org.apache.hadoop.ha.ActiveStandbyElector.fenceOldActive(ActiveStandbyElector.java:892) at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:797) at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:475) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:545) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497){noformat} Should not join the election before service is HEALTHY -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9459) ActiveStandbyElector can join election even before Service HEALTHY, and results in null data at ActiveBreadCrumb
[ https://issues.apache.org/jira/browse/HADOOP-9459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657071#comment-13657071 ] Hudson commented on HADOOP-9459: Integrated in Hadoop-Mapreduce-trunk #1425 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1425/]) HADOOP-9459. ActiveStandbyElector can join election even before Service HEALTHY, and results in null data at ActiveBreadCrumb. Contributed by Vinay and Todd Lipcon. (Revision 1482227) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1482227 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ActiveStandbyElector.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestActiveStandbyElector.java ActiveStandbyElector can join election even before Service HEALTHY, and results in null data at ActiveBreadCrumb Key: HADOOP-9459 URL: https://issues.apache.org/jira/browse/HADOOP-9459 Project: Hadoop Common Issue Type: Bug Components: ha Affects Versions: 2.0.2-alpha Reporter: Vinay Assignee: Vinay Priority: Critical Fix For: 3.0.0, 2.0.5-beta Attachments: HDFS-4463.patch, hdfs-4463.txt ActiveStandbyElector can store null at ActiveBreadCrumb in the below race condition. At further all failovers will fail resulting NPE. 1. ZKFC restarted. 2. due to machine busy, first zk connection is expired even before the health monitoring returned the status. 3. On re-establishment transitionToActive will be called, at this time appData will be null, 4. So now ActiveBreadCrumb will have null. 5. After this any failovers will fail throwing {noformat}java.lang.NullPointerException at org.apache.hadoop.util.StringUtils.byteToHexString(StringUtils.java:171) at org.apache.hadoop.ha.ActiveStandbyElector.fenceOldActive(ActiveStandbyElector.java:892) at org.apache.hadoop.ha.ActiveStandbyElector.becomeActive(ActiveStandbyElector.java:797) at org.apache.hadoop.ha.ActiveStandbyElector.processResult(ActiveStandbyElector.java:475) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:545) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:497){noformat} Should not join the election before service is HEALTHY -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9046) provide unit-test coverage of class org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT
[ https://issues.apache.org/jira/browse/HADOOP-9046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657095#comment-13657095 ] Robert Joseph Evans commented on HADOOP-9046: - Ivan, I am sorry I dropped the ball on this, and did not repond sooner. Sadly though the only time we want to put something into just 0.23 without going into trunk/branch-2 first is if there is a bug that only exists in branch-0.23. This is a very rare occurrence. If you think the coverage is good enough on branch-2 and trunk then would it be acceptable if we resolve this JIRA instead. provide unit-test coverage of class org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT -- Key: HADOOP-9046 URL: https://issues.apache.org/jira/browse/HADOOP-9046 Project: Hadoop Common Issue Type: Test Affects Versions: 0.23.6 Reporter: Ivan A. Veselovsky Assignee: Ivan A. Veselovsky Priority: Minor Attachments: HADOOP-9046-branch-0.23--c.patch, HADOOP-9046-branch-0.23--d.patch, HADOOP-9046-branch-0.23-over-9049.patch, HADOOP-9046-branch-0.23--over-HDFS-4567.patch, HADOOP-9046-branch-0.23.patch, HADOOP-9046-branch-2--over-HDFS-4567.patch, HADOOP-9046--c.patch, HADOOP-9046--d.patch, HADOOP-9046--e.patch, HADOOP-9046-over-9049.patch, HADOOP-9046.patch, HADOOP-9046-trunk--over-HDFS-4567.patch The class org.apache.hadoop.fs.DelegationTokenRenewer.RenewActionT has zero coverage in entire cumulative test run. Provide test(s) to cover this class. Note: the request submitted to HDFS project because the class likely to be tested by tests in that project. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9560) metrics2#JvmMetrics should have max memory size of JVM
[ https://issues.apache.org/jira/browse/HADOOP-9560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HADOOP-9560: Resolution: Fixed Fix Version/s: 2.0.5-beta Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I have committed this change to trunk and branch-2. Thank you Tsuyoshi! metrics2#JvmMetrics should have max memory size of JVM -- Key: HADOOP-9560 URL: https://issues.apache.org/jira/browse/HADOOP-9560 Project: Hadoop Common Issue Type: Improvement Components: metrics Affects Versions: 3.0.0, 2.0.5-beta Reporter: Tsuyoshi OZAWA Assignee: Tsuyoshi OZAWA Priority: Minor Fix For: 2.0.5-beta Attachments: HADOOP-9560.1.patch metrics#JvmMetrics have the max memory size of JVM specified by -Xmx option. metrics2#JvmMetrics doesn't have it but it should also have max memory size of JVM, because it's useful for users. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9307) BufferedFSInputStream.read returns wrong results after certain seeks
[ https://issues.apache.org/jira/browse/HADOOP-9307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HADOOP-9307: Attachment: hadoop-9307-branch-1.txt I committed to branch-2 and trunk. Here's a backport to branch-1 as well BufferedFSInputStream.read returns wrong results after certain seeks Key: HADOOP-9307 URL: https://issues.apache.org/jira/browse/HADOOP-9307 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 1.1.1, 2.0.2-alpha Reporter: Todd Lipcon Assignee: Todd Lipcon Attachments: hadoop-9307-branch-1.txt, hadoop-9307.txt After certain sequences of seek/read, BufferedFSInputStream can silently return data from the wrong part of the file. Further description in first comment below. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9307) BufferedFSInputStream.read returns wrong results after certain seeks
[ https://issues.apache.org/jira/browse/HADOOP-9307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HADOOP-9307: Target Version/s: 2.0.5-beta, 1.3.0 (was: 2.0.5-beta) Fix Version/s: 2.0.5-beta 3.0.0 BufferedFSInputStream.read returns wrong results after certain seeks Key: HADOOP-9307 URL: https://issues.apache.org/jira/browse/HADOOP-9307 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 1.1.1, 2.0.2-alpha Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 3.0.0, 2.0.5-beta Attachments: hadoop-9307-branch-1.txt, hadoop-9307.txt After certain sequences of seek/read, BufferedFSInputStream can silently return data from the wrong part of the file. Further description in first comment below. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9560) metrics2#JvmMetrics should have max memory size of JVM
[ https://issues.apache.org/jira/browse/HADOOP-9560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657133#comment-13657133 ] Tsuyoshi OZAWA commented on HADOOP-9560: Thank you too, Suresh! metrics2#JvmMetrics should have max memory size of JVM -- Key: HADOOP-9560 URL: https://issues.apache.org/jira/browse/HADOOP-9560 Project: Hadoop Common Issue Type: Improvement Components: metrics Affects Versions: 3.0.0, 2.0.5-beta Reporter: Tsuyoshi OZAWA Assignee: Tsuyoshi OZAWA Priority: Minor Fix For: 2.0.5-beta Attachments: HADOOP-9560.1.patch metrics#JvmMetrics have the max memory size of JVM specified by -Xmx option. metrics2#JvmMetrics doesn't have it but it should also have max memory size of JVM, because it's useful for users. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9220) Unnecessary transition to standby in ActiveStandbyElector
[ https://issues.apache.org/jira/browse/HADOOP-9220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657135#comment-13657135 ] Todd Lipcon commented on HADOOP-9220: - Woops, nope, I'll take out that 'new Exception' on commit. I was just using it to help debug. Good catch. Unnecessary transition to standby in ActiveStandbyElector - Key: HADOOP-9220 URL: https://issues.apache.org/jira/browse/HADOOP-9220 Project: Hadoop Common Issue Type: Bug Components: ha Reporter: Tom White Assignee: Tom White Priority: Critical Attachments: HADOOP-9220.patch, HADOOP-9220.patch, hadoop-9220.txt When performing a manual failover from one HA node to a second, under some circumstances the second node will transition from standby - active - standby - active. This is with automatic failover enabled, so there is a ZK cluster doing leader election. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9220) Unnecessary transition to standby in ActiveStandbyElector
[ https://issues.apache.org/jira/browse/HADOOP-9220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HADOOP-9220: Resolution: Fixed Fix Version/s: 2.0.5-beta 3.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to branch-2 and trunk. thanks Tom for tracking this down Unnecessary transition to standby in ActiveStandbyElector - Key: HADOOP-9220 URL: https://issues.apache.org/jira/browse/HADOOP-9220 Project: Hadoop Common Issue Type: Bug Components: ha Reporter: Tom White Assignee: Tom White Priority: Critical Fix For: 3.0.0, 2.0.5-beta Attachments: HADOOP-9220.patch, HADOOP-9220.patch, hadoop-9220.txt When performing a manual failover from one HA node to a second, under some circumstances the second node will transition from standby - active - standby - active. This is with automatic failover enabled, so there is a ZK cluster doing leader election. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9220) Unnecessary transition to standby in ActiveStandbyElector
[ https://issues.apache.org/jira/browse/HADOOP-9220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657144#comment-13657144 ] Hudson commented on HADOOP-9220: Integrated in Hadoop-trunk-Commit #3750 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3750/]) HADOOP-9220. Unnecessary transition to standby in ActiveStandbyElector. Contributed by Tom White and Todd Lipcon. (Revision 1482401) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1482401 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ActiveStandbyElector.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/DummyHAService.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestZKFailoverController.java Unnecessary transition to standby in ActiveStandbyElector - Key: HADOOP-9220 URL: https://issues.apache.org/jira/browse/HADOOP-9220 Project: Hadoop Common Issue Type: Bug Components: ha Reporter: Tom White Assignee: Tom White Priority: Critical Fix For: 3.0.0, 2.0.5-beta Attachments: HADOOP-9220.patch, HADOOP-9220.patch, hadoop-9220.txt When performing a manual failover from one HA node to a second, under some circumstances the second node will transition from standby - active - standby - active. This is with automatic failover enabled, so there is a ZK cluster doing leader election. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9079) LocalDirAllocator throws ArithmeticException
[ https://issues.apache.org/jira/browse/HADOOP-9079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657160#comment-13657160 ] Todd Lipcon commented on HADOOP-9079: - Could you make a unit test by making a directory, calling setReadable(false) and setExecutable(false), and then trying to set up a LocalDirAllocator inside it? LocalDirAllocator throws ArithmeticException Key: HADOOP-9079 URL: https://issues.apache.org/jira/browse/HADOOP-9079 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Attachments: hadoop-9079-v2.txt, trunk-9079.patch 2012-11-19 22:07:41,709 WARN [IPC Server handler 0 on 38671] nodemanager.NMAuditLogger(150): USER=UnknownUserIP= OPERATION=Stop Container RequestTARGET=ContainerManagerImpl RESULT=FAILURE DESCRIPTION=Trying to stop unknown container! APPID=application_1353391620476_0001 CONTAINERID=container_1353391620476_0001_01_10 java.lang.ArithmeticException: / by zero at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:368) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:263) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:849) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-6344) rm and rmr fail to correctly move the user's files to the trash prior to deleting when they are over quota.
[ https://issues.apache.org/jira/browse/HADOOP-6344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657166#comment-13657166 ] Albert XU commented on HADOOP-6344: --- It could be reproduced in the Hadoop-1.0.4 # bin/hadoop dfs -rmr input 13/05/15 00:14:15 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 13/05/15 00:14:15 WARN fs.Trash: Can't create trash directory: file:/C:/Documents and Settings/XU Zheng/.Trash/Current/C:/cygwin/usr/hadoop-1.0.4 Problem with Trash.Failed to set permissions of path: C:\Documents and Settings\XU Zheng\.Trash\Current\C:\cygwin\usr\hadoop-1.0.4 to 0700. Consider using -skipTrash option rmr: Failed to move to trash: file:/C:/cygwin/usr/hadoop-1.0.4/input rm and rmr fail to correctly move the user's files to the trash prior to deleting when they are over quota. - Key: HADOOP-6344 URL: https://issues.apache.org/jira/browse/HADOOP-6344 Project: Hadoop Common Issue Type: Bug Components: fs Affects Versions: 0.20.0, 0.20.1, 0.21.0, 0.22.0 Reporter: gary murry Assignee: Jakob Homan Fix For: 0.21.0, 0.22.0 Attachments: HDFS-740-for-Y20.patch, HDFS-740.patch With trash turned on, if a user is over his quota and does a rm (or rmr), the file is deleted without a copy being placed in the trash. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9079) LocalDirAllocator throws ArithmeticException
[ https://issues.apache.org/jira/browse/HADOOP-9079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HADOOP-9079: Assignee: (was: Jimmy Xiang) LocalDirAllocator throws ArithmeticException Key: HADOOP-9079 URL: https://issues.apache.org/jira/browse/HADOOP-9079 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Jimmy Xiang Priority: Minor Attachments: hadoop-9079-v2.txt, trunk-9079.patch 2012-11-19 22:07:41,709 WARN [IPC Server handler 0 on 38671] nodemanager.NMAuditLogger(150): USER=UnknownUserIP= OPERATION=Stop Container RequestTARGET=ContainerManagerImpl RESULT=FAILURE DESCRIPTION=Trying to stop unknown container! APPID=application_1353391620476_0001 CONTAINERID=container_1353391620476_0001_01_10 java.lang.ArithmeticException: / by zero at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:368) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:263) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:849) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9079) LocalDirAllocator throws ArithmeticException
[ https://issues.apache.org/jira/browse/HADOOP-9079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HADOOP-9079: Status: Open (was: Patch Available) LocalDirAllocator throws ArithmeticException Key: HADOOP-9079 URL: https://issues.apache.org/jira/browse/HADOOP-9079 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Jimmy Xiang Priority: Minor Attachments: hadoop-9079-v2.txt, trunk-9079.patch 2012-11-19 22:07:41,709 WARN [IPC Server handler 0 on 38671] nodemanager.NMAuditLogger(150): USER=UnknownUserIP= OPERATION=Stop Container RequestTARGET=ContainerManagerImpl RESULT=FAILURE DESCRIPTION=Trying to stop unknown container! APPID=application_1353391620476_0001 CONTAINERID=container_1353391620476_0001_01_10 java.lang.ArithmeticException: / by zero at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:368) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:263) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:849) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9079) LocalDirAllocator throws ArithmeticException
[ https://issues.apache.org/jira/browse/HADOOP-9079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657189#comment-13657189 ] Jimmy Xiang commented on HADOOP-9079: - I can work on it later when I get some time. Please feel free to take it over if anybody wants to help. LocalDirAllocator throws ArithmeticException Key: HADOOP-9079 URL: https://issues.apache.org/jira/browse/HADOOP-9079 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Jimmy Xiang Priority: Minor Attachments: hadoop-9079-v2.txt, trunk-9079.patch 2012-11-19 22:07:41,709 WARN [IPC Server handler 0 on 38671] nodemanager.NMAuditLogger(150): USER=UnknownUserIP= OPERATION=Stop Container RequestTARGET=ContainerManagerImpl RESULT=FAILURE DESCRIPTION=Trying to stop unknown container! APPID=application_1353391620476_0001 CONTAINERID=container_1353391620476_0001_01_10 java.lang.ArithmeticException: / by zero at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:368) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:263) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:849) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9435) Minimal code change to support IBM jvm build dependency
[ https://issues.apache.org/jira/browse/HADOOP-9435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657235#comment-13657235 ] Colin Patrick McCabe commented on HADOOP-9435: -- I built this against Sun JDK 6 and it worked fine. It looks good to me, although I didn't test against an IBM JVM. Minimal code change to support IBM jvm build dependency --- Key: HADOOP-9435 URL: https://issues.apache.org/jira/browse/HADOOP-9435 Project: Hadoop Common Issue Type: Bug Components: build Reporter: Tian Hong Wang Assignee: Tian Hong Wang Labels: patch Attachments: HADOOP-9435.patch, HADOOP-9435-v1.patch When native build hadoop-common-project with IBM java using command like: mvn package -Pnative it will exist the following errors. [exec] -- Configuring incomplete, errors occurred! [exec] JAVA_HOME=, JAVA_JVM_LIBRARY=/home/louis/ibm-java-i386-60/jre/lib/i386/classic/libjvm.so [exec] JAVA_INCLUDE_PATH=/home/louis/ibm-java-i386-60/include, JAVA_INCLUDE_PATH2=JAVA_INCLUDE_PATH2-NOTFOUND [exec] CMake Error at JNIFlags.cmake:113 (MESSAGE): [exec] Failed to find a viable JVM installation under JAVA_HOME. [exec] Call Stack (most recent call first): [exec] CMakeLists.txt:24 (include) The reason is that IBM java uses $JAVA_HOME/include/jniport.h instead of $JAVA_HOME/include/jni_md.h in non-IBM java. [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlsym' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlerror' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dladdr' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlopen' [exec] /usr/lib/jvm/java-1.6.0-ibm-1.6.0.12.0.x86_64/jre/lib/amd64/default/libjvm.so: undefined reference to `dlclose' [exec] collect2: ld returned 1 exit status [exec] make[2]: *** [test_libhdfs_ops] Error 1 [exec] make[1]: *** [CMakeFiles/test_libhdfs_ops.dir/all] Error 2 [exec] make: *** [all] Error The reason is libjvm.so need libdl when linking. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9079) LocalDirAllocator throws ArithmeticException
[ https://issues.apache.org/jira/browse/HADOOP-9079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657241#comment-13657241 ] Ted Yu commented on HADOOP-9079: I made modification in TestLocalDirAllocator#testRWBufferDirBecomesRO: {code} Index: hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalDirAllocator.java === --- hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalDirAllocator.java (revision 1482467) +++ hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestLocalDirAllocator.java (working copy) @@ -215,7 +215,9 @@ validateTempDirCreation(buildBufferDir(ROOT, nextDirIdx)); // change buffer directory 2 to be read only - new File(new Path(dir4).toUri().getPath()).setReadOnly(); + File f0 = new File(new Path(dir4).toUri().getPath()); + f0.setReadable(false); + f0.setExecutable(false); validateTempDirCreation(dir3); validateTempDirCreation(dir3); } finally { {code} When I ran the test, I got: {code} initializationError(org.apache.hadoop.fs.TestLocalDirAllocator) Time elapsed: 5 sec FAILURE! java.lang.AssertionError: at org.junit.Assert.fail(Assert.java:91) at org.junit.Assert.assertTrue(Assert.java:43) at org.junit.Assert.assertTrue(Assert.java:54) at org.apache.hadoop.fs.TestLocalDirAllocator.rmBufferDirs(TestLocalDirAllocator.java:104) at org.apache.hadoop.fs.TestLocalDirAllocator.clinit(TestLocalDirAllocator.java:71) {code} LocalDirAllocator throws ArithmeticException Key: HADOOP-9079 URL: https://issues.apache.org/jira/browse/HADOOP-9079 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Jimmy Xiang Priority: Minor Attachments: hadoop-9079-v2.txt, trunk-9079.patch 2012-11-19 22:07:41,709 WARN [IPC Server handler 0 on 38671] nodemanager.NMAuditLogger(150): USER=UnknownUserIP= OPERATION=Stop Container RequestTARGET=ContainerManagerImpl RESULT=FAILURE DESCRIPTION=Trying to stop unknown container! APPID=application_1353391620476_0001 CONTAINERID=container_1353391620476_0001_01_10 java.lang.ArithmeticException: / by zero at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:368) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:150) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:131) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:115) at org.apache.hadoop.yarn.server.nodemanager.LocalDirsHandlerService.getLocalPathForWrite(LocalDirsHandlerService.java:263) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:849) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9533) Centralized Hadoop SSO/Token Server
[ https://issues.apache.org/jira/browse/HADOOP-9533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657251#comment-13657251 ] Brian Swan commented on HADOOP-9533: Thanks for the heavy lifting on this, Larry. This issue is headed in the right direction. As you point out, it addresses the use cases of cloud, on-premises, hybrid enterprise deployments, and 3rd party integration scenarios - all of which are broadly important and are also our (Microsoft) high priority scenarios. To Daryn's comment about getting together at Hadoop Summit - I think that's a great idea. I plan to be there and look forward to discussions on how to rationalize related work and collaborate on the delivery of it. Centralized Hadoop SSO/Token Server --- Key: HADOOP-9533 URL: https://issues.apache.org/jira/browse/HADOOP-9533 Project: Hadoop Common Issue Type: New Feature Components: security Reporter: Larry McCay Attachments: HSSO-Interaction-Overview-rev-1.docx, HSSO-Interaction-Overview-rev-1.pdf This is an umbrella Jira filing to oversee a set of proposals for introducing a new master service for Hadoop Single Sign On (HSSO). There is an increasing need for pluggable authentication providers that authenticate both users and services as well as validate tokens in order to federate identities authenticated by trusted IDPs. These IDPs may be deployed within the enterprise or third-party IDPs that are external to the enterprise. These needs speak to a specific pain point: which is a narrow integration path into the enterprise identity infrastructure. Kerberos is a fine solution for those that already have it in place or are willing to adopt its use but there remains a class of user that finds this unacceptable and needs to integrate with a wider variety of identity management solutions. Another specific pain point is that of rolling and distributing keys. A related and integral part of the HSSO server is library called the Credential Management Framework (CMF), which will be a common library for easing the management of secrets, keys and credentials. Initially, the existing delegation, block access and job tokens will continue to be utilized. There may be some changes required to leverage a PKI based signature facility rather than shared secrets. This is a means to simplify the solution for the pain point of distributing shared secrets. This project will primarily centralize the responsibility of authentication and federation into a single service that is trusted across the Hadoop cluster and optionally across multiple clusters. This greatly simplifies a number of things in the Hadoop ecosystem: 1.a single token format that is used across all of Hadoop regardless of authentication method 2.a single service to have pluggable providers instead of all services 3.a single token authority that would be trusted across the cluster/s and through PKI encryption be able to easily issue cryptographically verifiable tokens 4.automatic rolling of the token authority’s keys and publishing of the public key for easy access by those parties that need to verify incoming tokens 5.use of PKI for signatures eliminates the need for securely sharing and distributing shared secrets In addition to serving as the internal Hadoop SSO service this service will be leveraged by the Knox Gateway from the cluster perimeter in order to acquire the Hadoop cluster tokens. The same token mechanism that is used for internal services will be used to represent user identities. Providing for interesting scenarios such as SSO across Hadoop clusters within an enterprise and/or into the cloud. The HSSO service will be comprised of three major components and capabilities: 1.Federating IDP – authenticates users/services and issues the common Hadoop token 2.Federating SP – validates the token of trusted external IDPs and issues the common Hadoop token 3.Token Authority – management of the common Hadoop tokens – including: a.Issuance b.Renewal c.Revocation As this is a meta Jira for tracking this overall effort, the details of the individual efforts will be submitted along with the child Jira filings. Hadoop-Common would seem to be the most appropriate home for such a service and its related common facilities. We will also leverage and extend existing common mechanisms as appropriate. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9523) Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs
[ https://issues.apache.org/jira/browse/HADOOP-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657258#comment-13657258 ] Kihwal Lee commented on HADOOP-9523: This is an incompatible change. For example, hbase startup script can have an issue. {noformat} JAVA_PLATFORM=`CLASSPATH=${CLASSPATH} ${JAVA} org.apache.hadoop.util.PlatformName | sed -e s/ /_/g` {noformat} Instead of changing the output format, we could have added something like -a for extended output. Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs --- Key: HADOOP-9523 URL: https://issues.apache.org/jira/browse/HADOOP-9523 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.4-alpha Reporter: Tian Hong Wang Assignee: Tian Hong Wang Labels: patch Fix For: 2.0.5-beta Attachments: HADOOP-9523.patch, HADOOP-9523-v1.patch, HADOOP-9523-v2.patch There are several different points between Sun jdk IBM jdk, so there is a need to provide a generic IBM java vendor flag. So enhance PlatformName.java to add Java vendor information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9563) Fix incompatibility introduced by HADOOP-9523
Kihwal Lee created HADOOP-9563: -- Summary: Fix incompatibility introduced by HADOOP-9523 Key: HADOOP-9563 URL: https://issues.apache.org/jira/browse/HADOOP-9563 Project: Hadoop Common Issue Type: Bug Components: util Reporter: Kihwal Lee HADOOP-9523 changed the output format of the platform name util, so hbase won't start unless platform name is specifically set. We could add a command line option to output in the new extended format and keep the old format when no option is given. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9563) Fix incompatibility introduced by HADOOP-9523
[ https://issues.apache.org/jira/browse/HADOOP-9563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HADOOP-9563: --- Affects Version/s: 2.0.5-beta 3.0.0 Fix incompatibility introduced by HADOOP-9523 - Key: HADOOP-9563 URL: https://issues.apache.org/jira/browse/HADOOP-9563 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 3.0.0, 2.0.5-beta Reporter: Kihwal Lee HADOOP-9523 changed the output format of the platform name util, so hbase won't start unless platform name is specifically set. We could add a command line option to output in the new extended format and keep the old format when no option is given. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9562) Create REST interface for HDFS health data
[ https://issues.apache.org/jira/browse/HADOOP-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657276#comment-13657276 ] Steve Loughran commented on HADOOP-9562: This would be good. # must include DFS safe mode status # assuming federation is supported, what are the implications? # what about HFDS HA? Allow all NNs to do an enum of all NNs status? # are there any security implications? If so, access should be restricted. # I'd like to have error codes in the pages, so I can do a {{curl -m 60 http://namenode:50020/service/dfslive.jsp}} and expect an error code if dfs is not up and out out of safemode. The use case would be : let bash scripts (inc Linux HA scripts) verify FS is live just by looking at the return code of curl or {{GET}}. Create REST interface for HDFS health data -- Key: HADOOP-9562 URL: https://issues.apache.org/jira/browse/HADOOP-9562 Project: Hadoop Common Issue Type: New Feature Components: fs Affects Versions: 2.0.4-alpha Reporter: Trevor Lorimer Priority: Minor The HDFS health screen (dfshealth.jsp) displays basic Version, Security and Health information concerning the NameNode, currently this information is accessible from classes in the org.apache.hadoop,hdfs.server.namenode package and cannot be accessed outside the NameNode. This becomes prevalent if the data is required to be displayed using a new user interface. The proposal is to create a REST interface to expose all the information displayed on dfshealth.jsp using GET methods. Wrapper classes will be created to serve the data to the REST root resource within the hadoop-hdfs project. This will enable the HDFS health screen information to be accessed remotely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9523) Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs
[ https://issues.apache.org/jira/browse/HADOOP-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657282#comment-13657282 ] Suresh Srinivas commented on HADOOP-9523: - [~kihwal] The class does not have any InterfaceAudience annotation. That means this it is private and there should not be any expectation of compatibility. If classes get used in the downstream projects, it would be good to annotate classes. Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs --- Key: HADOOP-9523 URL: https://issues.apache.org/jira/browse/HADOOP-9523 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.4-alpha Reporter: Tian Hong Wang Assignee: Tian Hong Wang Labels: patch Fix For: 2.0.5-beta Attachments: HADOOP-9523.patch, HADOOP-9523-v1.patch, HADOOP-9523-v2.patch There are several different points between Sun jdk IBM jdk, so there is a need to provide a generic IBM java vendor flag. So enhance PlatformName.java to add Java vendor information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9523) Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs
[ https://issues.apache.org/jira/browse/HADOOP-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657309#comment-13657309 ] Kihwal Lee commented on HADOOP-9523: That's clear for cases where classes are used programmatically. In this case, however, it was a command line interface that caused the issue. I am not sure how we can effectively mark and inform users of the stability of command line interfaces. I think users expect command line interfaces to be more stable or backward compatible than API. Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs --- Key: HADOOP-9523 URL: https://issues.apache.org/jira/browse/HADOOP-9523 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.4-alpha Reporter: Tian Hong Wang Assignee: Tian Hong Wang Labels: patch Fix For: 2.0.5-beta Attachments: HADOOP-9523.patch, HADOOP-9523-v1.patch, HADOOP-9523-v2.patch There are several different points between Sun jdk IBM jdk, so there is a need to provide a generic IBM java vendor flag. So enhance PlatformName.java to add Java vendor information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9439) JniBasedUnixGroupsMapping: fix some crash bugs
[ https://issues.apache.org/jira/browse/HADOOP-9439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657348#comment-13657348 ] Todd Lipcon commented on HADOOP-9439: - {code} + static String empty[] = new String[0]; {code} Should probably be static final, right? And rename to {{NO_GROUPS}} or {{EMPTY_STIRNG_ARRAY}} or something a little more descriptive {code} + native static String[] getGroupsForUser(String username, String empty[], +boolean reentrant); {code} I don't get why {{empty}} is a parameter here... aside from it being a premature optimization (is allocating an empty array that bad?), it seems like the JNI code could just grab this field itself and return a reference, rather than taking it as a parameter. {code} + groups = getGroupsForUser(user, empty, useReentrant ); {code} Style nit: extra space here before ')' {code} + snprintf(buf, sizeof(buf), getgrouplist error %d, ret); + THROW(env, java/lang/RuntimeException, buf); {code} Can we use strerror here on the return, assuming it's probably a standard errnum? {code} +ret = hadoop_group_info_fetch(ginfo, uinfo-gids[i]); +if (!ret) { {code} Here if the group info lookup fails for a particular gid, you end up swallowing the error without any warnings, etc. Maybe instead we should just return the numeric gid as a group? This is what the 'groups' shell command does: {code} todd@todd-w510:~$ groups todd todd : groups: cannot find name for group ID 1050 1050 adm dialout cdrom audio video plugdev lpadmin admin sambashare mrtest wireshark {code} - I noticed you use different locks for getgrnam and getpwnam. But when I was working on HADOOP-7156, I found that the buggy implementations were racey in other parts of their code -- ie a concurrent getgrnam and a concurrent getpwnam might race with eachother and cause either to fail. I know that you added this function to deal with a crash we saw on a production cluster - did you verify that with the same buggy underlying implementation that the workaround prevents the issue? We may have to actually share this lock and configuration with the lock used in NativeIO.c. Additionally, when I did that patch, I found it simpler to continue to use the reentrant functions wrapped with a lock -- less repeated code, etc. JniBasedUnixGroupsMapping: fix some crash bugs -- Key: HADOOP-9439 URL: https://issues.apache.org/jira/browse/HADOOP-9439 Project: Hadoop Common Issue Type: Bug Components: native Affects Versions: 2.0.4-alpha Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Attachments: HADOOP-9439.001.patch, HDFS-4640.002.patch JniBasedUnixGroupsMapping has some issues. * sometimes on error paths variables are freed prior to being initialized * re-allocate buffers less frequently (can reuse the same buffer for multiple calls to getgrnam) * allow non-reentrant functions to be used, to work around client bugs * don't throw IOException from JNI functions if the JNI functions do not declare this checked exception. * don't bail out if only one group name among all the ones associated with a user can't be looked up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9563) Fix incompatibility introduced by HADOOP-9523
[ https://issues.apache.org/jira/browse/HADOOP-9563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657397#comment-13657397 ] Kihwal Lee commented on HADOOP-9563: I am not sure about the effectiveness of annotation since it was command line interface invoked from the start-up script. Fix incompatibility introduced by HADOOP-9523 - Key: HADOOP-9563 URL: https://issues.apache.org/jira/browse/HADOOP-9563 Project: Hadoop Common Issue Type: Bug Components: util Affects Versions: 3.0.0, 2.0.5-beta Reporter: Kihwal Lee HADOOP-9523 changed the output format of the platform name util, so hbase won't start unless platform name is specifically set. We could add a command line option to output in the new extended format and keep the old format when no option is given. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9523) Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs
[ https://issues.apache.org/jira/browse/HADOOP-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HADOOP-9523: Hadoop Flags: Incompatible change,Reviewed (was: Reviewed) Adding Incompatible Change for now, so folks are aware of it. Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs --- Key: HADOOP-9523 URL: https://issues.apache.org/jira/browse/HADOOP-9523 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.4-alpha Reporter: Tian Hong Wang Assignee: Tian Hong Wang Labels: patch Fix For: 2.0.5-beta Attachments: HADOOP-9523.patch, HADOOP-9523-v1.patch, HADOOP-9523-v2.patch There are several different points between Sun jdk IBM jdk, so there is a need to provide a generic IBM java vendor flag. So enhance PlatformName.java to add Java vendor information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9346) Upgrading to protoc 2.5.0 fails the build
[ https://issues.apache.org/jira/browse/HADOOP-9346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657411#comment-13657411 ] Ravi Prakash commented on HADOOP-9346: -- Hi Harsh. Thanks. This is a thorn in my side as well. The code builds and I tried wordcount on Fedora 19 + protobuf-2.5. However on Fedora 19 + protobuf 2.4 even though the build succeeds with the patch, the DN doesn't come up: 2013-05-14 14:26:30,916 [DataNode: [file:/tmp/hadoop-raviprak/hdfs/data1,file:/tmp/hadoop-raviprak/hdfs/data2] heartbeating to NAMENODE-HOSTNAME/NAMENODE-IP:9000] WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected exception in block pool Block pool registering (storage id unknown) service to NAMENODE-HOSTNAME/NAMENODE-IP:9000 java.lang.UnsupportedOperationException: This is supposed to be overridden by subclasses. at com.google.protobuf.GeneratedMessage.getUnknownFields(GeneratedMessage.java:180) at org.apache.hadoop.hdfs.protocol.proto.HdfsProtos$VersionRequestProto.getSerializedSize(HdfsProtos.java:20296) at com.google.protobuf.AbstractMessageLite.toByteString(AbstractMessageLite.java:49) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.constructRpcRequest(ProtobufRpcEngine.java:149) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:193) at com.sun.proxy.$Proxy9.versionRequest(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83) at com.sun.proxy.$Proxy9.versionRequest(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.versionRequest(DatanodeProtocolClientSideTranslatorPB.java:245) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.retrieveNamespaceInfo(BPServiceActor.java:163) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:217) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:664) at java.lang.Thread.run(Thread.java:722) Were you able to get the daemon up? Is this something wrong with my environment? Without the patch the daemon came up and I was able to run a wordcount Upgrading to protoc 2.5.0 fails the build - Key: HADOOP-9346 URL: https://issues.apache.org/jira/browse/HADOOP-9346 Project: Hadoop Common Issue Type: Task Components: build Affects Versions: 3.0.0 Reporter: Harsh J Assignee: Harsh J Priority: Minor Labels: protobuf Attachments: HADOOP-9346.patch Reported over the impala lists, one of the errors received is: {code} src/hadoop-common-project/hadoop-common/target/generated-sources/java/org/apache/hadoop/ha/proto/ZKFCProtocolProtos.java:[104,37] can not find symbol. symbol: class Parser location: package com.google.protobuf {code} Worth looking into as we'll eventually someday bump our protobuf deps. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9562) Create REST interface for HDFS health data
[ https://issues.apache.org/jira/browse/HADOOP-9562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657413#comment-13657413 ] Chris Nauroth commented on HADOOP-9562: --- {quote} I'd like to have error codes in the pages, so I can do a curl -m 60 http://namenode:50020/service/dfslive.jsp and expect an error code if dfs is not up and out out of safemode. The use case would be : let bash scripts (inc Linux HA scripts) verify FS is live just by looking at the return code of curl or GET. {quote} One difficulty is that we don't start the HTTP server until relatively late in initialization. If you have a ton of edits, then you'll suffer through a long checkpoint before status is visible, either in the web page or a REST interface. I have patches queued up in HDFS-4249 and its sub-tasks to start the HTTP server sooner and display more details about the early phases of namenode startup. Create REST interface for HDFS health data -- Key: HADOOP-9562 URL: https://issues.apache.org/jira/browse/HADOOP-9562 Project: Hadoop Common Issue Type: New Feature Components: fs Affects Versions: 2.0.4-alpha Reporter: Trevor Lorimer Priority: Minor The HDFS health screen (dfshealth.jsp) displays basic Version, Security and Health information concerning the NameNode, currently this information is accessible from classes in the org.apache.hadoop,hdfs.server.namenode package and cannot be accessed outside the NameNode. This becomes prevalent if the data is required to be displayed using a new user interface. The proposal is to create a REST interface to expose all the information displayed on dfshealth.jsp using GET methods. Wrapper classes will be created to serve the data to the REST root resource within the hadoop-hdfs project. This will enable the HDFS health screen information to be accessed remotely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9523) Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs
[ https://issues.apache.org/jira/browse/HADOOP-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657424#comment-13657424 ] Suresh Srinivas commented on HADOOP-9523: - bq. In this case, however, it was a command line interface... It is the output of the main right? Irrespective of programmatically or command line, we need to make sure class is backward compatible. That means method return values or output cannot change. bq. Adding Incompatible Change for now, so folks are aware of it. I actually think this is not necessary unless it has appropriate InterfaceAudience. The whole idea of such classification was to say, if some thing changes in such classes and if you use it, be ready to change. Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs --- Key: HADOOP-9523 URL: https://issues.apache.org/jira/browse/HADOOP-9523 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.4-alpha Reporter: Tian Hong Wang Assignee: Tian Hong Wang Labels: patch Fix For: 2.0.5-beta Attachments: HADOOP-9523.patch, HADOOP-9523-v1.patch, HADOOP-9523-v2.patch There are several different points between Sun jdk IBM jdk, so there is a need to provide a generic IBM java vendor flag. So enhance PlatformName.java to add Java vendor information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9564) DFSClient$DFSOutputStream.closeInternal locks up waiting for namenode.complete
Jin Feng created HADOOP-9564: Summary: DFSClient$DFSOutputStream.closeInternal locks up waiting for namenode.complete Key: HADOOP-9564 URL: https://issues.apache.org/jira/browse/HADOOP-9564 Project: Hadoop Common Issue Type: Bug Components: fs Reporter: Jin Feng Priority: Minor Hi, Our component uses FileSystem.copyFromLocalFile to copy a local file to HDFS cluster. It's working fine in production environment. Its integration tests used to run fine on our dev's local Mac laptop until recently (exact point of time unknown) our tests started to freeze up very frequently with this stack: {code} com.twitter.ads.billing.spendaggregator.jobs.maintenance.billinghourlyarchive.BillingHourlyAggArchiveJobIT-runner prio=5 tid=0x7f9ce52cb800 nid=0x4a503 waiting on condition [0x000165815000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x000152f41378 (a java.util.concurrent.FutureTask$Sync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303) at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:248) at java.util.concurrent.FutureTask.get(FutureTask.java:111) at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:790) - locked 0x00014f568720 (a java.lang.Object) at org.apache.hadoop.ipc.Client.call(Client.java:1080) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226) at $Proxy37.complete(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) at $Proxy37.complete(Unknown Source) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:3566) - locked 0x000152f3f658 (a org.apache.hadoop.hdfs.DFSClient$DFSOutputStream) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3481) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61) at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86) at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59) at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:89) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:224) at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1295) {code} our version is 0.20.2.cdh3u2-t1. In the test suite, we use org.apache.hadoop.hdfs.MiniDFSCluster. I've searched around couldn't find anything resembles this symptom, any helps are really appreciated! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-8562) Enhancements to support Hadoop on Windows Server and Windows Azure environments
[ https://issues.apache.org/jira/browse/HADOOP-8562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657695#comment-13657695 ] Suresh Srinivas commented on HADOOP-8562: - The changes from this jira and related jiras from HDFS and YARN has been in trunk for some time now. If there are no objections, I plan on merging these changes to branch-2 early next week. Enhancements to support Hadoop on Windows Server and Windows Azure environments --- Key: HADOOP-8562 URL: https://issues.apache.org/jira/browse/HADOOP-8562 Project: Hadoop Common Issue Type: New Feature Affects Versions: 3.0.0 Reporter: Bikas Saha Assignee: Bikas Saha Fix For: 3.0.0 Attachments: branch-trunk-win.min-notest.patch, branch-trunk-win-min.patch, branch-trunk-win.min.patch, branch-trunk-win.patch, branch-trunk-win.patch, branch-trunk-win.patch, branch-trunk-win.patch, branch-trunk-win.patch, branch-trunk-win.patch, branch-trunk-win.patch, branch-trunk-win.patch, branch-trunk-win.patch, branch-trunk-win.patch, branch-trunk-win.patch, test-untar.tar, test-untar.tgz This JIRA tracks the work that needs to be done on trunk to enable Hadoop to run on Windows Server and Azure environments. This incorporates porting relevant work from the similar effort on branch 1 tracked via HADOOP-8079. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9564) DFSClient$DFSOutputStream.closeInternal locks up waiting for namenode.complete
[ https://issues.apache.org/jira/browse/HADOOP-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jin Feng updated HADOOP-9564: - Description: Hi, Our component uses FileSystem.copyFromLocalFile to copy a local file to HDFS cluster. It's working fine in production environment. Its integration tests used to run fine on our dev's local Mac laptop until recently (exact point of time unknown) our tests started to freeze up very frequently with this stack: {code} java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x000152f41378 (a java.util.concurrent.FutureTask$Sync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303) at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:248) at java.util.concurrent.FutureTask.get(FutureTask.java:111) at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:790) - locked 0x00014f568720 (a java.lang.Object) at org.apache.hadoop.ipc.Client.call(Client.java:1080) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226) at $Proxy37.complete(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) at $Proxy37.complete(Unknown Source) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:3566) - locked 0x000152f3f658 (a org.apache.hadoop.hdfs.DFSClient$DFSOutputStream) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3481) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61) at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86) at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59) at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:89) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:224) at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1295) {code} our version is 0.20.2.cdh3u2-t1. In the test suite, we use org.apache.hadoop.hdfs.MiniDFSCluster. I've searched around couldn't find anything resembles this symptom, any helps are really appreciated! was: Hi, Our component uses FileSystem.copyFromLocalFile to copy a local file to HDFS cluster. It's working fine in production environment. Its integration tests used to run fine on our dev's local Mac laptop until recently (exact point of time unknown) our tests started to freeze up very frequently with this stack: {code} com.twitter.ads.billing.spendaggregator.jobs.maintenance.billinghourlyarchive.BillingHourlyAggArchiveJobIT-runner prio=5 tid=0x7f9ce52cb800 nid=0x4a503 waiting on condition [0x000165815000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x000152f41378 (a java.util.concurrent.FutureTask$Sync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303) at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:248) at java.util.concurrent.FutureTask.get(FutureTask.java:111) at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:790) - locked 0x00014f568720 (a java.lang.Object) at org.apache.hadoop.ipc.Client.call(Client.java:1080) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226) at $Proxy37.complete(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at
[jira] [Commented] (HADOOP-9564) DFSClient$DFSOutputStream.closeInternal locks up waiting for namenode.complete
[ https://issues.apache.org/jira/browse/HADOOP-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657707#comment-13657707 ] Jin Feng commented on HADOOP-9564: -- This is captured from our test output, seems like the DataBlockScanner is really slow coming up with verification results even though our data/file generated in the tests are minimal. {noformat} 13/05/04 06:50:55 INFO hdfs.StateChange: BLOCK* NameSystem.allocateBlock: our_test_file_name.lzo-f773a37f-3dac-4337-a6cb-004fb94c1d31. blk_-8485988660073681466_1002 13/05/04 06:50:55 INFO datanode.DataNode: Receiving block blk_-8485988660073681466_1002 src: /127.0.0.1:42563 dest: /127.0.0.1:35830 13/05/04 06:50:55 INFO DataNode.clienttrace: src: /127.0.0.1:42563, dest: /127.0.0.1:35830, bytes: 303, op: HDFS_WRITE, cliID: DFSClient_-854844208, offset: 0, srvID: DS-1070312150-10.35.8.106-35830-1367650255272, blockid: blk_-8485988660073681466_1002, duration: 778000 13/05/04 06:50:55 INFO datanode.DataNode: PacketResponder 0 for block blk_-8485988660073681466_1002 terminating 13/05/04 06:50:55 INFO hdfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 127.0.0.1:35830 is added to blk_-8485988660073681466_1002 size 303 13/05/04 06:52:48 INFO datanode.DataBlockScanner: Verification succeeded for blk_-8485988660073681466_1002 13/05/04 07:00:40 INFO datanode.DataBlockScanner: Verification succeeded for blk_586310994067086116_1001 {noformat} Could this be related to this bug: HADOOP-4584? DFSClient$DFSOutputStream.closeInternal locks up waiting for namenode.complete -- Key: HADOOP-9564 URL: https://issues.apache.org/jira/browse/HADOOP-9564 Project: Hadoop Common Issue Type: Bug Components: fs Reporter: Jin Feng Priority: Minor Hi, Our component uses FileSystem.copyFromLocalFile to copy a local file to HDFS cluster. It's working fine in production environment. Its integration tests used to run fine on our dev's local Mac laptop until recently (exact point of time unknown) our tests started to freeze up very frequently with this stack: {code} java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x000152f41378 (a java.util.concurrent.FutureTask$Sync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303) at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:248) at java.util.concurrent.FutureTask.get(FutureTask.java:111) at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:790) - locked 0x00014f568720 (a java.lang.Object) at org.apache.hadoop.ipc.Client.call(Client.java:1080) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226) at $Proxy37.complete(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) at $Proxy37.complete(Unknown Source) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:3566) - locked 0x000152f3f658 (a org.apache.hadoop.hdfs.DFSClient$DFSOutputStream) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3481) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61) at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86) at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59) at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:89) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:224) at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1295) {code} our version is 0.20.2.cdh3u2-t1. In the test suite, we use org.apache.hadoop.hdfs.MiniDFSCluster. I've searched around couldn't find anything resembles this symptom, any helps are really appreciated! -- This message is
[jira] [Commented] (HADOOP-9523) Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs
[ https://issues.apache.org/jira/browse/HADOOP-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657815#comment-13657815 ] Tian Hong Wang commented on HADOOP-9523: Suresh, could you please undo this patch for it has caused some incompatible change? Provide a generic IBM java vendor flag in PlatformName.java to support non-Sun JREs --- Key: HADOOP-9523 URL: https://issues.apache.org/jira/browse/HADOOP-9523 Project: Hadoop Common Issue Type: Improvement Affects Versions: 2.0.4-alpha Reporter: Tian Hong Wang Assignee: Tian Hong Wang Labels: patch Fix For: 2.0.5-beta Attachments: HADOOP-9523.patch, HADOOP-9523-v1.patch, HADOOP-9523-v2.patch There are several different points between Sun jdk IBM jdk, so there is a need to provide a generic IBM java vendor flag. So enhance PlatformName.java to add Java vendor information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9564) DFSClient$DFSOutputStream.closeInternal locks up waiting for namenode.complete
[ https://issues.apache.org/jira/browse/HADOOP-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657832#comment-13657832 ] Todd Lipcon commented on HADOOP-9564: - What's the thread doing on the NN side? Keep in mind you're on a very old version (and a twitter-local build that no one else has), so you may not have a lot of luck getting people to help you diagnose farther. DFSClient$DFSOutputStream.closeInternal locks up waiting for namenode.complete -- Key: HADOOP-9564 URL: https://issues.apache.org/jira/browse/HADOOP-9564 Project: Hadoop Common Issue Type: Bug Components: fs Reporter: Jin Feng Priority: Minor Hi, Our component uses FileSystem.copyFromLocalFile to copy a local file to HDFS cluster. It's working fine in production environment. Its integration tests used to run fine on our dev's local Mac laptop until recently (exact point of time unknown) our tests started to freeze up very frequently with this stack: {code} java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x000152f41378 (a java.util.concurrent.FutureTask$Sync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303) at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:248) at java.util.concurrent.FutureTask.get(FutureTask.java:111) at org.apache.hadoop.ipc.Client$Connection.sendParam(Client.java:790) - locked 0x00014f568720 (a java.lang.Object) at org.apache.hadoop.ipc.Client.call(Client.java:1080) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226) at $Proxy37.complete(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) at $Proxy37.complete(Unknown Source) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:3566) - locked 0x000152f3f658 (a org.apache.hadoop.hdfs.DFSClient$DFSOutputStream) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3481) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(FSDataOutputStream.java:61) at org.apache.hadoop.fs.FSDataOutputStream.close(FSDataOutputStream.java:86) at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59) at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:89) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:224) at org.apache.hadoop.fs.FileSystem.copyFromLocalFile(FileSystem.java:1295) {code} our version is 0.20.2.cdh3u2-t1. In the test suite, we use org.apache.hadoop.hdfs.MiniDFSCluster. I've searched around couldn't find anything resembles this symptom, any helps are really appreciated! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9346) Upgrading to protoc 2.5.0 fails the build
[ https://issues.apache.org/jira/browse/HADOOP-9346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13657834#comment-13657834 ] Harsh J commented on HADOOP-9346: - Thanks Ravi! No I did not actually build a package and run the daemons. The error does look protobuf related. I'll investigate! Upgrading to protoc 2.5.0 fails the build - Key: HADOOP-9346 URL: https://issues.apache.org/jira/browse/HADOOP-9346 Project: Hadoop Common Issue Type: Task Components: build Affects Versions: 3.0.0 Reporter: Harsh J Assignee: Harsh J Priority: Minor Labels: protobuf Attachments: HADOOP-9346.patch Reported over the impala lists, one of the errors received is: {code} src/hadoop-common-project/hadoop-common/target/generated-sources/java/org/apache/hadoop/ha/proto/ZKFCProtocolProtos.java:[104,37] can not find symbol. symbol: class Parser location: package com.google.protobuf {code} Worth looking into as we'll eventually someday bump our protobuf deps. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9346) Upgrading to protoc 2.5.0 fails the build
[ https://issues.apache.org/jira/browse/HADOOP-9346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated HADOOP-9346: Status: Open (was: Patch Available) Cancelling patch while we address Ravi's find (and any other such subtle issues that tests are seemingly not capturing) Upgrading to protoc 2.5.0 fails the build - Key: HADOOP-9346 URL: https://issues.apache.org/jira/browse/HADOOP-9346 Project: Hadoop Common Issue Type: Task Components: build Affects Versions: 3.0.0 Reporter: Harsh J Assignee: Harsh J Priority: Minor Labels: protobuf Attachments: HADOOP-9346.patch Reported over the impala lists, one of the errors received is: {code} src/hadoop-common-project/hadoop-common/target/generated-sources/java/org/apache/hadoop/ha/proto/ZKFCProtocolProtos.java:[104,37] can not find symbol. symbol: class Parser location: package com.google.protobuf {code} Worth looking into as we'll eventually someday bump our protobuf deps. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9392) Token based authentication and Single Sign On
[ https://issues.apache.org/jira/browse/HADOOP-9392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658067#comment-13658067 ] Thomas NGUY commented on HADOOP-9392: - From my point of view, we have two different designs to SSO in Hadoop but not necessary incompatible. Concerning the « Token » design, in HADOOP-9533, the Service Access Token targets a specific ressource (defined by the service URL) and have a low lifetime while in HADOOP-9392, the Identity Token can be used for any services? (belonging to at trusted tokenrealm) and doesnt have an expiration time. Both of them carry extended attributed for fined-grained access control decisions or for the service itself. I'm just curious to learn more about how the Unified Authorization Framework (Hadoop-9466) would used to common token to make decisions. Token based authentication and Single Sign On - Key: HADOOP-9392 URL: https://issues.apache.org/jira/browse/HADOOP-9392 Project: Hadoop Common Issue Type: New Feature Components: security Reporter: Kai Zheng Assignee: Kai Zheng Fix For: 3.0.0 Attachments: token-based-authn-plus-sso.pdf This is an umbrella entry for one of project Rhino’s topic, for details of project Rhino, please refer to https://github.com/intel-hadoop/project-rhino/. The major goal for this entry as described in project Rhino was “Core, HDFS, ZooKeeper, and HBase currently support Kerberos authentication at the RPC layer, via SASL. However this does not provide valuable attributes such as group membership, classification level, organizational identity, or support for user defined attributes. Hadoop components must interrogate external resources for discovering these attributes and at scale this is problematic. There is also no consistent delegation model. HDFS has a simple delegation capability, and only Oozie can take limited advantage of it. We will implement a common token based authentication framework to decouple internal user and service authentication from external mechanisms used to support it (like Kerberos)” We’d like to start our work from Hadoop-Common and try to provide common facilities by extending existing authentication framework which support: 1.Pluggable token provider interface 2.Pluggable token verification protocol and interface 3.Security mechanism to distribute secrets in cluster nodes 4.Delegation model of user authentication -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9346) Upgrading to protoc 2.5.0 fails the build
[ https://issues.apache.org/jira/browse/HADOOP-9346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13658086#comment-13658086 ] Tian Hong Wang commented on HADOOP-9346: hi Harsh, When using protoc2.5.0, I met the problem of HADOOP-9440. It seems that protoc2.5.0 add new code. Upgrading to protoc 2.5.0 fails the build - Key: HADOOP-9346 URL: https://issues.apache.org/jira/browse/HADOOP-9346 Project: Hadoop Common Issue Type: Task Components: build Affects Versions: 3.0.0 Reporter: Harsh J Assignee: Harsh J Priority: Minor Labels: protobuf Attachments: HADOOP-9346.patch Reported over the impala lists, one of the errors received is: {code} src/hadoop-common-project/hadoop-common/target/generated-sources/java/org/apache/hadoop/ha/proto/ZKFCProtocolProtos.java:[104,37] can not find symbol. symbol: class Parser location: package com.google.protobuf {code} Worth looking into as we'll eventually someday bump our protobuf deps. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira