[jira] [Updated] (HADOOP-10110) hadoop-auth has a build break due to missing dependency
[
https://issues.apache.org/jira/browse/HADOOP-10110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Boudnik updated HADOOP-10110:
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to trunk, branch-2, and branch-2.3. Thank you, Chuan!
> hadoop-auth has a build break due to missing dependency
> ---
>
> Key: HADOOP-10110
> URL: https://issues.apache.org/jira/browse/HADOOP-10110
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 2.0.6-alpha, 2.2.0
>Reporter: Chuan Liu
>Assignee: Chuan Liu
>Priority: Blocker
> Fix For: 3.0.0, 2.3.0
>
> Attachments: HADOOP-10110.patch
>
>
> We have a build break in hadoop-auth if build with maven cache cleaned. The
> error looks like the follows. The problem exists on both Windows and Linux.
> If you have old jetty jars in your maven cache, you won't see the error.
> {noformat}
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 1:29.469s
> [INFO] Finished at: Mon Nov 18 12:30:36 PST 2013
> [INFO] Final Memory: 37M/120M
> [INFO]
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile
> (default-testCompile) on project hadoop-auth: Compilation failure:
> Compilation failure:
> [ERROR]
> /home/chuan/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[84,13]
> cannot access org.mortbay.component.AbstractLifeCycle
> [ERROR] class file for org.mortbay.component.AbstractLifeCycle not found
> [ERROR] server = new Server(0);
> [ERROR]
> /home/chuan/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[94,29]
> cannot access org.mortbay.component.LifeCycle
> [ERROR] class file for org.mortbay.component.LifeCycle not found
> [ERROR] server.getConnectors()[0].setHost(host);
> [ERROR]
> /home/chuan/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[96,10]
> cannot find symbol
> [ERROR] symbol : method start()
> [ERROR] location: class org.mortbay.jetty.Server
> [ERROR]
> /home/chuan/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[102,12]
> cannot find symbol
> [ERROR] symbol : method stop()
> [ERROR] location: class org.mortbay.jetty.Server
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :hadoop-auth
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HADOOP-10110) hadoop-auth has a build break due to missing dependency
[
https://issues.apache.org/jira/browse/HADOOP-10110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13877211#comment-13877211
]
Konstantin Boudnik commented on HADOOP-10110:
-
I will commit this as soon as my SVN password issue is resolved (hopefully by
tomorrow). If someone want to commit it before - please do!
> hadoop-auth has a build break due to missing dependency
> ---
>
> Key: HADOOP-10110
> URL: https://issues.apache.org/jira/browse/HADOOP-10110
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 2.0.6-alpha, 2.2.0
>Reporter: Chuan Liu
>Assignee: Chuan Liu
>Priority: Blocker
> Fix For: 3.0.0, 2.3.0
>
> Attachments: HADOOP-10110.patch
>
>
> We have a build break in hadoop-auth if build with maven cache cleaned. The
> error looks like the follows. The problem exists on both Windows and Linux.
> If you have old jetty jars in your maven cache, you won't see the error.
> {noformat}
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 1:29.469s
> [INFO] Finished at: Mon Nov 18 12:30:36 PST 2013
> [INFO] Final Memory: 37M/120M
> [INFO]
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile
> (default-testCompile) on project hadoop-auth: Compilation failure:
> Compilation failure:
> [ERROR]
> /home/chuan/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[84,13]
> cannot access org.mortbay.component.AbstractLifeCycle
> [ERROR] class file for org.mortbay.component.AbstractLifeCycle not found
> [ERROR] server = new Server(0);
> [ERROR]
> /home/chuan/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[94,29]
> cannot access org.mortbay.component.LifeCycle
> [ERROR] class file for org.mortbay.component.LifeCycle not found
> [ERROR] server.getConnectors()[0].setHost(host);
> [ERROR]
> /home/chuan/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[96,10]
> cannot find symbol
> [ERROR] symbol : method start()
> [ERROR] location: class org.mortbay.jetty.Server
> [ERROR]
> /home/chuan/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[102,12]
> cannot find symbol
> [ERROR] symbol : method stop()
> [ERROR] location: class org.mortbay.jetty.Server
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :hadoop-auth
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HADOOP-10143) replace WritableFactories's hashmap with ConcurrentHashMap
[ https://issues.apache.org/jira/browse/HADOOP-10143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HADOOP-10143: --- Resolution: Fixed Fix Version/s: 2.3.0 3.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to branch 2 and to trunk. Thank you for the patch [~xieliang007] > replace WritableFactories's hashmap with ConcurrentHashMap > -- > > Key: HADOOP-10143 > URL: https://issues.apache.org/jira/browse/HADOOP-10143 > Project: Hadoop Common > Issue Type: Improvement > Components: io >Affects Versions: 2.0.0-alpha, 2.2.0 >Reporter: Liang Xie >Assignee: Liang Xie > Fix For: 3.0.0, 2.3.0 > > Attachments: HADOOP-10143.txt > > > We observed a lock contend hotspot from a HBase cluster: > "IPC Reader 9 on port 12600" daemon prio=10 tid=0x7f85b8aceed0 nid=0x4be8 > waiting for monitor entry [0x7f8501c57000] >java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.hadoop.io.WritableFactories.getFactory(WritableFactories.java:44) > - locked <0x0007fd1328a8> (a java.lang.Class for > org.apache.hadoop.io.WritableFactories) > at > org.apache.hadoop.io.WritableFactories.newInstance(WritableFactories.java:49) > at > org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:680) > at > org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:586) > at > org.apache.hadoop.hbase.client.MultiAction.readFields(MultiAction.java:116) > at > org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:682) > at > org.apache.hadoop.hbase.ipc.Invocation.readFields(Invocation.java:126) > at > org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.processData(SecureServer.java:618) > at > org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.processOneRpc(SecureServer.java:596) > at > org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.saslReadAndProcess(SecureServer.java:362) > at > org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.readAndProcess(SecureServer.java:492) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:770) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:561) > - locked <0x00043da3fea0> (a > org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:536) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > "IPC Reader 7 on port 12600" daemon prio=10 tid=0x7f85b8a99df0 nid=0x4be6 > waiting for monitor entry [0x7f8501cd9000] >java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.hadoop.io.WritableFactories.getFactory(WritableFactories.java:44) > - locked <0x0007fd1328a8> (a java.lang.Class for > org.apache.hadoop.io.WritableFactories) > at > org.apache.hadoop.io.WritableFactories.newInstance(WritableFactories.java:49) > at > org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:680) > at > org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:586) > at > org.apache.hadoop.hbase.client.MultiAction.readFields(MultiAction.java:116) > at > org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:682) > at > org.apache.hadoop.hbase.ipc.Invocation.readFields(Invocation.java:126) > at > org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.processData(SecureServer.java:618) > at > org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.processOneRpc(SecureServer.java:596) > at > org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.saslReadAndProcess(SecureServer.java:362) > at > org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.readAndProcess(SecureServer.java:492) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:770) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:561) > - locked <0x00043da232e8> (a > org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:536) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at >
[jira] [Commented] (HADOOP-10213) Fix bugs parsing ACL spec in FsShell setfacl.
[ https://issues.apache.org/jira/browse/HADOOP-10213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13877168#comment-13877168 ] Vinay commented on HADOOP-10213: Thanks Chris for the detailed reviews and commit. > Fix bugs parsing ACL spec in FsShell setfacl. > - > > Key: HADOOP-10213 > URL: https://issues.apache.org/jira/browse/HADOOP-10213 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: HDFS ACLs (HDFS-4685) >Reporter: Chris Nauroth >Assignee: Vinay > Fix For: HDFS ACLs (HDFS-4685) > > Attachments: HADOOP-10213.patch, HADOOP-10213.patch, > HADOOP-10213.patch, HADOOP-10213.patch, HADOOP-10213.patch > > > When calling setfacl -x to remove ACL entries, it does not make sense for the > entries in the ACL spec to contain permissions. The permissions should be > unspecified, and the CLI should return an error if the user attempts to > provide permissions. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10110) hadoop-auth has a build break due to missing dependency
[
https://issues.apache.org/jira/browse/HADOOP-10110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13877149#comment-13877149
]
Sangjin Lee commented on HADOOP-10110:
--
+1 on this patch as well. We have also encountered this issue.
> hadoop-auth has a build break due to missing dependency
> ---
>
> Key: HADOOP-10110
> URL: https://issues.apache.org/jira/browse/HADOOP-10110
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Affects Versions: 2.0.6-alpha, 2.2.0
>Reporter: Chuan Liu
>Assignee: Chuan Liu
>Priority: Blocker
> Fix For: 3.0.0, 2.3.0
>
> Attachments: HADOOP-10110.patch
>
>
> We have a build break in hadoop-auth if build with maven cache cleaned. The
> error looks like the follows. The problem exists on both Windows and Linux.
> If you have old jetty jars in your maven cache, you won't see the error.
> {noformat}
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 1:29.469s
> [INFO] Finished at: Mon Nov 18 12:30:36 PST 2013
> [INFO] Final Memory: 37M/120M
> [INFO]
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile
> (default-testCompile) on project hadoop-auth: Compilation failure:
> Compilation failure:
> [ERROR]
> /home/chuan/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[84,13]
> cannot access org.mortbay.component.AbstractLifeCycle
> [ERROR] class file for org.mortbay.component.AbstractLifeCycle not found
> [ERROR] server = new Server(0);
> [ERROR]
> /home/chuan/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[94,29]
> cannot access org.mortbay.component.LifeCycle
> [ERROR] class file for org.mortbay.component.LifeCycle not found
> [ERROR] server.getConnectors()[0].setHost(host);
> [ERROR]
> /home/chuan/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[96,10]
> cannot find symbol
> [ERROR] symbol : method start()
> [ERROR] location: class org.mortbay.jetty.Server
> [ERROR]
> /home/chuan/trunk/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[102,12]
> cannot find symbol
> [ERROR] symbol : method stop()
> [ERROR] location: class org.mortbay.jetty.Server
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :hadoop-auth
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HADOOP-10245) Hadoop command line always appends "-Xmx" option twice
[ https://issues.apache.org/jira/browse/HADOOP-10245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13877068#comment-13877068 ] shanyu zhao commented on HADOOP-10245: -- [~ywskycn] yes, it is the same issue. Sorry I didn't see HADOOP-9870 before I submit this one. I also found similar JIRAs HADOOP-9211 and HDFS-5087. I went through these JIRAs and here are my thoughts: We should only rely on $HADOOP_HEAPSIZE to control Java heap size, instead of $HADOOP_CLIENT_OPTS. Otherwise it would be very confusing and hard to debug issues. And I've seen many real world issues caused by this confusion. There are arguments that $HADOOP_HEAPSIZE is only for service, and client should have its own settings. Well, we could create HADOOP_CLIENT_HEAPSIZE which is initialized to 512m and used in hadoop.sh. But personally I think it does not worth it to add this new env variable. The client can just simply use $HADOOP_HEAPSIZE which defaults to 1000m. Also, there are scenarios that a java class executed by "hadoop jar" command has a large memory requirements. A real world example: Hive's MapredLocalTask calls "hadoop jar" to build a local hash table. Also, if there's a need to change the heapsize, one can always set env variable $HADOOP_HEAPSIZE. > Hadoop command line always appends "-Xmx" option twice > -- > > Key: HADOOP-10245 > URL: https://issues.apache.org/jira/browse/HADOOP-10245 > Project: Hadoop Common > Issue Type: Bug > Components: bin >Affects Versions: 2.2.0 >Reporter: shanyu zhao >Assignee: shanyu zhao > Attachments: HADOOP-10245.patch > > > The Hadoop command line scripts (hadoop.sh or hadoop.cmd) will call java with > "-Xmx" options twice. The impact is that any user defined HADOOP_HEAP_SIZE > env variable will take no effect because it is overwritten by the second > "-Xmx" option. > For example, here is the java cmd generated for command "hadoop fs -ls /", > Notice that there are two "-Xmx" options: "-Xmx1000m" and "-Xmx512m" in the > command line: > java -Xmx1000m -Dhadoop.log.dir=C:\tmp\logs -Dhadoop.log.file=hadoop.log > -Dhadoop.root.logger=INFO,c > onsole,DRFA -Xmx512m -Dhadoop.security.logger=INFO,RFAS -classpath XXX > org.apache.hadoop.fs.FsShell -ls / > Here is the root cause: > The call flow is: hadoop.sh calls hadoop_config.sh, which in turn calls > hadoop-env.sh. > In hadoop.sh, the command line is generated by the following pseudo code: > java $JAVA_HEAP_MAX $HADOOP_CLIENT_OPTS -classpath ... > In hadoop-config.sh, $JAVA_HEAP_MAX is initialized as "-Xmx1000m" if user > didn't set $HADOOP_HEAP_SIZE env variable. > In hadoop-env.sh, $HADOOP_CLIENT_OPTS is set as this: > export HADOOP_CLIENT_OPTS="-Xmx512m $HADOOP_CLIENT_OPTS" > To fix this problem, we should remove the "-Xmx512m" from HADOOP_CLIENT_OPTS. > If we really want to change the memory settings we need to use > $HADOOP_HEAP_SIZE env variable. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10217) Unable to run 'hadoop' commands, after installing on Cygwin
[ https://issues.apache.org/jira/browse/HADOOP-10217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13877050#comment-13877050 ] Chuan Liu commented on HADOOP-10217: I suggest just set HADOOP_HOME and JAVA_HOME using Windows path instead of cygwin path and run the commands from a Windows cmd shell instead of Windows shell. However, I have never tested the Hadoop on Windows XP; so I cannot guarantee it will work. > Unable to run 'hadoop' commands, after installing on Cygwin > --- > > Key: HADOOP-10217 > URL: https://issues.apache.org/jira/browse/HADOOP-10217 > Project: Hadoop Common > Issue Type: Bug > Components: conf >Affects Versions: 2.2.0 > Environment: Installed on Cygwin (latest version) running Window XP. > Set Java 1.7.0_45 path (JDK) to /cygdrive/e/JDK. Installed ssh (on > /cygdrive/e/Openssh-6.4p1), created all keys and stored on > /home/admin.Installed hadoop-2.2.0 on /cygdrive/e/hadoop-2.2.0 >Reporter: Anand Murali > Labels: test > > Did following > 1. export JAVA_HOME=/cygdrive/e/JDK > 2. export HADOOP_INSTALL=/cygdrive/e/hadoop-2.2.0 > 3. export > PATH=:$PATH:$HADOOP_INSTALL/bin:$HADOOP_INSTALL/sbin:$HADOOP_INSTALL/etc:$HADOOP_INSTALL/share:$HADOOP_INSTALL/lib:$HADOOP_INSTALL/libexec > $hadoop version > Error: Could not find or load main class org.apache.hadoop.util.VersionInfo. > Cannot run anymore commands. I am unable to detect what path problems is > causing this error -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10140) Specification of HADOOP_CONF_DIR via the environment in hadoop_config.cmd
[ https://issues.apache.org/jira/browse/HADOOP-10140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13877041#comment-13877041 ] Chuan Liu commented on HADOOP-10140: Hey [~ian_jack...@trilliumsoftware.com], I think HADOOP_CONF_DIR should be set as YARN_CONF_DIR from my understanding of the code. In most cmd files including yarn-conf.cmd, we will call hadoop-config.cmd. In hadoop-conf.cmd, we will set HADOOP_CONF_DIR to the command line argument of "--config". Please let me know if you see it differently. > Specification of HADOOP_CONF_DIR via the environment in hadoop_config.cmd > - > > Key: HADOOP-10140 > URL: https://issues.apache.org/jira/browse/HADOOP-10140 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.2.0 > Environment: This Windows when using the "Command" processing as > opposed to using Bash from Cygwin. >Reporter: Ian Jackson >Priority: Minor > > It would be nice if HADOOP_CONF_DIR could be set in the environment like > YARN_CONF_DIR. This could be done in lib-exec/hadoop.cmd by setting > HADOOP_CONF_DIR conditionally. > Using the Windows command shell, the modification would be as follows: > if not defined HADOOP_CONF_DIR ( > set HADOOP_CONF_DIR=%HADOOP_HOME%\etc\hadoop > ) > This would allow the Hadoop configuration to be placed in ProgramData more > easily. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10133) winutils detection on windows-cygwin fails
[ https://issues.apache.org/jira/browse/HADOOP-10133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13877037#comment-13877037 ] Chuan Liu commented on HADOOP-10133: Is HADOOP_HOME environment variable or Java property "hadoop.home.dir" been set? We are relying on these two methods to discover "winutils.exe". Also "winutils.exe" have no dependency on cygwin, so there should be no need to start the Hadoop processes from cygwin. > winutils detection on windows-cygwin fails > -- > > Key: HADOOP-10133 > URL: https://issues.apache.org/jira/browse/HADOOP-10133 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.2.0 > Environment: windows 7, cygwin >Reporter: Franjo Markovic > Original Estimate: 1h > Remaining Estimate: 1h > > java.io.IOException: Could not locate executable null\bin\winutils.exe in the > Hadoop binaries. > at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:278) > at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:300) > at org.apache.hadoop.util.Shell.(Shell.java:293) -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10245) Hadoop command line always appends "-Xmx" option twice
[
https://issues.apache.org/jira/browse/HADOOP-10245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13877017#comment-13877017
]
Hadoop QA commented on HADOOP-10245:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12624030/HADOOP-10245.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3449//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3449//console
This message is automatically generated.
> Hadoop command line always appends "-Xmx" option twice
> --
>
> Key: HADOOP-10245
> URL: https://issues.apache.org/jira/browse/HADOOP-10245
> Project: Hadoop Common
> Issue Type: Bug
> Components: bin
>Affects Versions: 2.2.0
>Reporter: shanyu zhao
>Assignee: shanyu zhao
> Attachments: HADOOP-10245.patch
>
>
> The Hadoop command line scripts (hadoop.sh or hadoop.cmd) will call java with
> "-Xmx" options twice. The impact is that any user defined HADOOP_HEAP_SIZE
> env variable will take no effect because it is overwritten by the second
> "-Xmx" option.
> For example, here is the java cmd generated for command "hadoop fs -ls /",
> Notice that there are two "-Xmx" options: "-Xmx1000m" and "-Xmx512m" in the
> command line:
> java -Xmx1000m -Dhadoop.log.dir=C:\tmp\logs -Dhadoop.log.file=hadoop.log
> -Dhadoop.root.logger=INFO,c
> onsole,DRFA -Xmx512m -Dhadoop.security.logger=INFO,RFAS -classpath XXX
> org.apache.hadoop.fs.FsShell -ls /
> Here is the root cause:
> The call flow is: hadoop.sh calls hadoop_config.sh, which in turn calls
> hadoop-env.sh.
> In hadoop.sh, the command line is generated by the following pseudo code:
> java $JAVA_HEAP_MAX $HADOOP_CLIENT_OPTS -classpath ...
> In hadoop-config.sh, $JAVA_HEAP_MAX is initialized as "-Xmx1000m" if user
> didn't set $HADOOP_HEAP_SIZE env variable.
> In hadoop-env.sh, $HADOOP_CLIENT_OPTS is set as this:
> export HADOOP_CLIENT_OPTS="-Xmx512m $HADOOP_CLIENT_OPTS"
> To fix this problem, we should remove the "-Xmx512m" from HADOOP_CLIENT_OPTS.
> If we really want to change the memory settings we need to use
> $HADOOP_HEAP_SIZE env variable.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HADOOP-10245) Hadoop command line always appends "-Xmx" option twice
[ https://issues.apache.org/jira/browse/HADOOP-10245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13877002#comment-13877002 ] Wei Yan commented on HADOOP-10245: -- Hey, shanyu. Is this one related to HADOOP-9870? > Hadoop command line always appends "-Xmx" option twice > -- > > Key: HADOOP-10245 > URL: https://issues.apache.org/jira/browse/HADOOP-10245 > Project: Hadoop Common > Issue Type: Bug > Components: bin >Affects Versions: 2.2.0 >Reporter: shanyu zhao >Assignee: shanyu zhao > Attachments: HADOOP-10245.patch > > > The Hadoop command line scripts (hadoop.sh or hadoop.cmd) will call java with > "-Xmx" options twice. The impact is that any user defined HADOOP_HEAP_SIZE > env variable will take no effect because it is overwritten by the second > "-Xmx" option. > For example, here is the java cmd generated for command "hadoop fs -ls /", > Notice that there are two "-Xmx" options: "-Xmx1000m" and "-Xmx512m" in the > command line: > java -Xmx1000m -Dhadoop.log.dir=C:\tmp\logs -Dhadoop.log.file=hadoop.log > -Dhadoop.root.logger=INFO,c > onsole,DRFA -Xmx512m -Dhadoop.security.logger=INFO,RFAS -classpath XXX > org.apache.hadoop.fs.FsShell -ls / > Here is the root cause: > The call flow is: hadoop.sh calls hadoop_config.sh, which in turn calls > hadoop-env.sh. > In hadoop.sh, the command line is generated by the following pseudo code: > java $JAVA_HEAP_MAX $HADOOP_CLIENT_OPTS -classpath ... > In hadoop-config.sh, $JAVA_HEAP_MAX is initialized as "-Xmx1000m" if user > didn't set $HADOOP_HEAP_SIZE env variable. > In hadoop-env.sh, $HADOOP_CLIENT_OPTS is set as this: > export HADOOP_CLIENT_OPTS="-Xmx512m $HADOOP_CLIENT_OPTS" > To fix this problem, we should remove the "-Xmx512m" from HADOOP_CLIENT_OPTS. > If we really want to change the memory settings we need to use > $HADOOP_HEAP_SIZE env variable. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HADOOP-10245) Hadoop command line always appends "-Xmx" option twice
[ https://issues.apache.org/jira/browse/HADOOP-10245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shanyu zhao updated HADOOP-10245: - Attachment: HADOOP-10245.patch > Hadoop command line always appends "-Xmx" option twice > -- > > Key: HADOOP-10245 > URL: https://issues.apache.org/jira/browse/HADOOP-10245 > Project: Hadoop Common > Issue Type: Bug > Components: bin >Affects Versions: 2.2.0 >Reporter: shanyu zhao >Assignee: shanyu zhao > Attachments: HADOOP-10245.patch > > > The Hadoop command line scripts (hadoop.sh or hadoop.cmd) will call java with > "-Xmx" options twice. The impact is that any user defined HADOOP_HEAP_SIZE > env variable will take no effect because it is overwritten by the second > "-Xmx" option. > For example, here is the java cmd generated for command "hadoop fs -ls /", > Notice that there are two "-Xmx" options: "-Xmx1000m" and "-Xmx512m" in the > command line: > java -Xmx1000m -Dhadoop.log.dir=C:\tmp\logs -Dhadoop.log.file=hadoop.log > -Dhadoop.root.logger=INFO,c > onsole,DRFA -Xmx512m -Dhadoop.security.logger=INFO,RFAS -classpath XXX > org.apache.hadoop.fs.FsShell -ls / > Here is the root cause: > The call flow is: hadoop.sh calls hadoop_config.sh, which in turn calls > hadoop-env.sh. > In hadoop.sh, the command line is generated by the following pseudo code: > java $JAVA_HEAP_MAX $HADOOP_CLIENT_OPTS -classpath ... > In hadoop-config.sh, $JAVA_HEAP_MAX is initialized as "-Xmx1000m" if user > didn't set $HADOOP_HEAP_SIZE env variable. > In hadoop-env.sh, $HADOOP_CLIENT_OPTS is set as this: > export HADOOP_CLIENT_OPTS="-Xmx512m $HADOOP_CLIENT_OPTS" > To fix this problem, we should remove the "-Xmx512m" from HADOOP_CLIENT_OPTS. > If we really want to change the memory settings we need to use > $HADOOP_HEAP_SIZE env variable. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HADOOP-10245) Hadoop command line always appends "-Xmx" option twice
[ https://issues.apache.org/jira/browse/HADOOP-10245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shanyu zhao updated HADOOP-10245: - Status: Patch Available (was: Open) > Hadoop command line always appends "-Xmx" option twice > -- > > Key: HADOOP-10245 > URL: https://issues.apache.org/jira/browse/HADOOP-10245 > Project: Hadoop Common > Issue Type: Bug > Components: bin >Affects Versions: 2.2.0 >Reporter: shanyu zhao >Assignee: shanyu zhao > Attachments: HADOOP-10245.patch > > > The Hadoop command line scripts (hadoop.sh or hadoop.cmd) will call java with > "-Xmx" options twice. The impact is that any user defined HADOOP_HEAP_SIZE > env variable will take no effect because it is overwritten by the second > "-Xmx" option. > For example, here is the java cmd generated for command "hadoop fs -ls /", > Notice that there are two "-Xmx" options: "-Xmx1000m" and "-Xmx512m" in the > command line: > java -Xmx1000m -Dhadoop.log.dir=C:\tmp\logs -Dhadoop.log.file=hadoop.log > -Dhadoop.root.logger=INFO,c > onsole,DRFA -Xmx512m -Dhadoop.security.logger=INFO,RFAS -classpath XXX > org.apache.hadoop.fs.FsShell -ls / > Here is the root cause: > The call flow is: hadoop.sh calls hadoop_config.sh, which in turn calls > hadoop-env.sh. > In hadoop.sh, the command line is generated by the following pseudo code: > java $JAVA_HEAP_MAX $HADOOP_CLIENT_OPTS -classpath ... > In hadoop-config.sh, $JAVA_HEAP_MAX is initialized as "-Xmx1000m" if user > didn't set $HADOOP_HEAP_SIZE env variable. > In hadoop-env.sh, $HADOOP_CLIENT_OPTS is set as this: > export HADOOP_CLIENT_OPTS="-Xmx512m $HADOOP_CLIENT_OPTS" > To fix this problem, we should remove the "-Xmx512m" from HADOOP_CLIENT_OPTS. > If we really want to change the memory settings we need to use > $HADOOP_HEAP_SIZE env variable. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HADOOP-10245) Hadoop command line always appends "-Xmx" option twice
shanyu zhao created HADOOP-10245: Summary: Hadoop command line always appends "-Xmx" option twice Key: HADOOP-10245 URL: https://issues.apache.org/jira/browse/HADOOP-10245 Project: Hadoop Common Issue Type: Bug Components: bin Affects Versions: 2.2.0 Reporter: shanyu zhao Assignee: shanyu zhao The Hadoop command line scripts (hadoop.sh or hadoop.cmd) will call java with "-Xmx" options twice. The impact is that any user defined HADOOP_HEAP_SIZE env variable will take no effect because it is overwritten by the second "-Xmx" option. For example, here is the java cmd generated for command "hadoop fs -ls /", Notice that there are two "-Xmx" options: "-Xmx1000m" and "-Xmx512m" in the command line: java -Xmx1000m -Dhadoop.log.dir=C:\tmp\logs -Dhadoop.log.file=hadoop.log -Dhadoop.root.logger=INFO,c onsole,DRFA -Xmx512m -Dhadoop.security.logger=INFO,RFAS -classpath XXX org.apache.hadoop.fs.FsShell -ls / Here is the root cause: The call flow is: hadoop.sh calls hadoop_config.sh, which in turn calls hadoop-env.sh. In hadoop.sh, the command line is generated by the following pseudo code: java $JAVA_HEAP_MAX $HADOOP_CLIENT_OPTS -classpath ... In hadoop-config.sh, $JAVA_HEAP_MAX is initialized as "-Xmx1000m" if user didn't set $HADOOP_HEAP_SIZE env variable. In hadoop-env.sh, $HADOOP_CLIENT_OPTS is set as this: export HADOOP_CLIENT_OPTS="-Xmx512m $HADOOP_CLIENT_OPTS" To fix this problem, we should remove the "-Xmx512m" from HADOOP_CLIENT_OPTS. If we really want to change the memory settings we need to use $HADOOP_HEAP_SIZE env variable. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-8989) hadoop dfs -find feature
[ https://issues.apache.org/jira/browse/HADOOP-8989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13876872#comment-13876872 ] Jonathan Allen commented on HADOOP-8989: It was waiting for a review, I know Daryn was part way through but needed to put it aside for a while and I don't know if he had a chance to get back to it. I'll sort out the problems it seems to have against the latest trunk and submit a new patch (hopefully won't require major changes). > hadoop dfs -find feature > > > Key: HADOOP-8989 > URL: https://issues.apache.org/jira/browse/HADOOP-8989 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Marco Nicosia >Assignee: Jonathan Allen > Attachments: HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, > HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, > HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch > > > Both sysadmins and users make frequent use of the unix 'find' command, but > Hadoop has no correlate. Without this, users are writing scripts which make > heavy use of hadoop dfs -lsr, and implementing find one-offs. I think hdfs > -lsr is somewhat taxing on the NameNode, and a really slow experience on the > client side. Possibly an in-NameNode find operation would be only a bit more > taxing on the NameNode, but significantly faster from the client's point of > view? > The minimum set of options I can think of which would make a Hadoop find > command generally useful is (in priority order): > * -type (file or directory, for now) > * -atime/-ctime-mtime (... and -creationtime?) (both + and - arguments) > * -print0 (for piping to xargs -0) > * -depth > * -owner/-group (and -nouser/-nogroup) > * -name (allowing for shell pattern, or even regex?) > * -perm > * -size > One possible special case, but could possibly be really cool if it ran from > within the NameNode: > * -delete > The "hadoop dfs -lsr | hadoop dfs -rm" cycle is really, really slow. > Lower priority, some people do use operators, mostly to execute -or searches > such as: > * find / \(-nouser -or -nogroup\) > Finally, I thought I'd include a link to the [Posix spec for > find|http://www.opengroup.org/onlinepubs/009695399/utilities/find.html] -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10244) TestKeyShell improperly tests the results of a Delete
[
https://issues.apache.org/jira/browse/HADOOP-10244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13876857#comment-13876857
]
Hadoop QA commented on HADOOP-10244:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12624011/10244.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3448//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3448//console
This message is automatically generated.
> TestKeyShell improperly tests the results of a Delete
> -
>
> Key: HADOOP-10244
> URL: https://issues.apache.org/jira/browse/HADOOP-10244
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Reporter: Larry McCay
>Assignee: Larry McCay
> Attachments: 10244.patch
>
>
> The TestKeyShell.testKeySuccessfulKeyLifecycle test is supposed to ensure
> that the deleted key is no longer in the results of a subsequent delete
> command. Mistakenly, it is testing that it is STILL there.
> The delete command is actually working but the stdout capture should be reset
> instead of flushed. Therefore, the test is picking up the existence of the
> key name from the deletion message in the previous command.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (HADOOP-10244) TestKeyShell improperly tests the results of a Delete
[ https://issues.apache.org/jira/browse/HADOOP-10244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Larry McCay updated HADOOP-10244: - Release Note: Fix of inappropriate test of delete functionality. Status: Patch Available (was: Open) Fix of inappropriate test of delete functionality. Changed the use of flush to reset and test that the deleted keyName is expected to NOT be in the list output after a delete. > TestKeyShell improperly tests the results of a Delete > - > > Key: HADOOP-10244 > URL: https://issues.apache.org/jira/browse/HADOOP-10244 > Project: Hadoop Common > Issue Type: Bug > Components: security >Reporter: Larry McCay >Assignee: Larry McCay > Attachments: 10244.patch > > > The TestKeyShell.testKeySuccessfulKeyLifecycle test is supposed to ensure > that the deleted key is no longer in the results of a subsequent delete > command. Mistakenly, it is testing that it is STILL there. > The delete command is actually working but the stdout capture should be reset > instead of flushed. Therefore, the test is picking up the existence of the > key name from the deletion message in the previous command. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (HADOOP-10244) TestKeyShell improperly tests the results of a Delete
[ https://issues.apache.org/jira/browse/HADOOP-10244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Larry McCay updated HADOOP-10244: - Attachment: 10244.patch Fix TestKeyShell test. > TestKeyShell improperly tests the results of a Delete > - > > Key: HADOOP-10244 > URL: https://issues.apache.org/jira/browse/HADOOP-10244 > Project: Hadoop Common > Issue Type: Bug > Components: security >Reporter: Larry McCay >Assignee: Larry McCay > Attachments: 10244.patch > > > The TestKeyShell.testKeySuccessfulKeyLifecycle test is supposed to ensure > that the deleted key is no longer in the results of a subsequent delete > command. Mistakenly, it is testing that it is STILL there. > The delete command is actually working but the stdout capture should be reset > instead of flushed. Therefore, the test is picking up the existence of the > key name from the deletion message in the previous command. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HADOOP-10244) TestKeyShell improperly tests the results of a Delete
Larry McCay created HADOOP-10244: Summary: TestKeyShell improperly tests the results of a Delete Key: HADOOP-10244 URL: https://issues.apache.org/jira/browse/HADOOP-10244 Project: Hadoop Common Issue Type: Bug Components: security Reporter: Larry McCay Assignee: Larry McCay The TestKeyShell.testKeySuccessfulKeyLifecycle test is supposed to ensure that the deleted key is no longer in the results of a subsequent delete command. Mistakenly, it is testing that it is STILL there. The delete command is actually working but the stdout capture should be reset instead of flushed. Therefore, the test is picking up the existence of the key name from the deletion message in the previous command. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-8989) hadoop dfs -find feature
[
https://issues.apache.org/jira/browse/HADOOP-8989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13876719#comment-13876719
]
Hadoop QA commented on HADOOP-8989:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12573783/HADOOP-8989.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3447//console
This message is automatically generated.
> hadoop dfs -find feature
>
>
> Key: HADOOP-8989
> URL: https://issues.apache.org/jira/browse/HADOOP-8989
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Marco Nicosia
>Assignee: Jonathan Allen
> Attachments: HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch,
> HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch,
> HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch
>
>
> Both sysadmins and users make frequent use of the unix 'find' command, but
> Hadoop has no correlate. Without this, users are writing scripts which make
> heavy use of hadoop dfs -lsr, and implementing find one-offs. I think hdfs
> -lsr is somewhat taxing on the NameNode, and a really slow experience on the
> client side. Possibly an in-NameNode find operation would be only a bit more
> taxing on the NameNode, but significantly faster from the client's point of
> view?
> The minimum set of options I can think of which would make a Hadoop find
> command generally useful is (in priority order):
> * -type (file or directory, for now)
> * -atime/-ctime-mtime (... and -creationtime?) (both + and - arguments)
> * -print0 (for piping to xargs -0)
> * -depth
> * -owner/-group (and -nouser/-nogroup)
> * -name (allowing for shell pattern, or even regex?)
> * -perm
> * -size
> One possible special case, but could possibly be really cool if it ran from
> within the NameNode:
> * -delete
> The "hadoop dfs -lsr | hadoop dfs -rm" cycle is really, really slow.
> Lower priority, some people do use operators, mostly to execute -or searches
> such as:
> * find / \(-nouser -or -nogroup\)
> Finally, I thought I'd include a link to the [Posix spec for
> find|http://www.opengroup.org/onlinepubs/009695399/utilities/find.html]
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HADOOP-8989) hadoop dfs -find feature
[ https://issues.apache.org/jira/browse/HADOOP-8989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13876716#comment-13876716 ] Mark Miller commented on HADOOP-8989: - Lot's of watchers, Hadoop QA was happy, anyone willing to tip this in? > hadoop dfs -find feature > > > Key: HADOOP-8989 > URL: https://issues.apache.org/jira/browse/HADOOP-8989 > Project: Hadoop Common > Issue Type: New Feature >Reporter: Marco Nicosia >Assignee: Jonathan Allen > Attachments: HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, > HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch, > HADOOP-8989.patch, HADOOP-8989.patch, HADOOP-8989.patch > > > Both sysadmins and users make frequent use of the unix 'find' command, but > Hadoop has no correlate. Without this, users are writing scripts which make > heavy use of hadoop dfs -lsr, and implementing find one-offs. I think hdfs > -lsr is somewhat taxing on the NameNode, and a really slow experience on the > client side. Possibly an in-NameNode find operation would be only a bit more > taxing on the NameNode, but significantly faster from the client's point of > view? > The minimum set of options I can think of which would make a Hadoop find > command generally useful is (in priority order): > * -type (file or directory, for now) > * -atime/-ctime-mtime (... and -creationtime?) (both + and - arguments) > * -print0 (for piping to xargs -0) > * -depth > * -owner/-group (and -nouser/-nogroup) > * -name (allowing for shell pattern, or even regex?) > * -perm > * -size > One possible special case, but could possibly be really cool if it ran from > within the NameNode: > * -delete > The "hadoop dfs -lsr | hadoop dfs -rm" cycle is really, really slow. > Lower priority, some people do use operators, mostly to execute -or searches > such as: > * find / \(-nouser -or -nogroup\) > Finally, I thought I'd include a link to the [Posix spec for > find|http://www.opengroup.org/onlinepubs/009695399/utilities/find.html] -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Work started] (HADOOP-9371) Define Semantics of FileSystem and FileContext more rigorously
[
https://issues.apache.org/jira/browse/HADOOP-9371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HADOOP-9371 started by Steve Loughran.
> Define Semantics of FileSystem and FileContext more rigorously
> --
>
> Key: HADOOP-9371
> URL: https://issues.apache.org/jira/browse/HADOOP-9371
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs
>Affects Versions: 1.2.0, 3.0.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: HADOOP-9361.2.patch, HADOOP-9361.patch,
> HADOOP-9371-003.patch, HadoopFilesystemContract.pdf
>
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> The semantics of {{FileSystem}} and {{FileContext}} are not completely
> defined in terms of
> # core expectations of a filesystem
> # consistency requirements.
> # concurrency requirements.
> # minimum scale limits
> Furthermore, methods are not defined strictly enough in terms of their
> outcomes and failure modes.
> The requirements and method semantics should be defined more strictly.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Resolved] (HADOOP-5348) Create a ThrowableWritable for serializing exceptions robustly
[ https://issues.apache.org/jira/browse/HADOOP-5348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-5348. Resolution: Won't Fix > Create a ThrowableWritable for serializing exceptions robustly > -- > > Key: HADOOP-5348 > URL: https://issues.apache.org/jira/browse/HADOOP-5348 > Project: Hadoop Common > Issue Type: New Feature > Components: ipc >Affects Versions: 0.21.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: HADOOP-5348-2.patch, ThrowableWritable.java, > hadoop-5348.patch > > > HADOOP-5201 and other issues would benefit from a stable representation of > exceptions, one that can be sent over the network, maybe pushed out to web > UIs and which we can be 100% sure that the far end will be able to handle if > they have the hadoop-core JAR on their classpath. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Resolved] (HADOOP-5621) MapReducer to run junit tests under Hadoop
[ https://issues.apache.org/jira/browse/HADOOP-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-5621. Resolution: Won't Fix YARN lets us run tests across a cluster differently > MapReducer to run junit tests under Hadoop > -- > > Key: HADOOP-5621 > URL: https://issues.apache.org/jira/browse/HADOOP-5621 > Project: Hadoop Common > Issue Type: New Feature >Affects Versions: 0.21.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > > This is something I mentioned to some people last week, thought I would start > a discussion on it. > We could run junit tests as a MapReduce job with > # a mapper that takes a list of classes, one per line > # extracts the test suite from each class, and then invokes each test method. > This would be a new junit test runner. > # saves the result (and any exceptions) as the output. Also saves any machine > specific details. > # It also needs to grab the System.out and System.err channels, to map them > to specific tests. > # Measure how long the tests took (incuding setup/teardown time) > # Add an ant task to take filesets and other patterns, and > generate text files from the contents (with stripping of prefixes and > suffices, directory separator substition, file begin/end values, etc, etc). I > have this with tests already. > The result would be that you could point listresources at a directory tree > and create a text file listing all tests to run. These could be executed > across multiple hosts and the results correlated. It would be, initially, a > MapExpand, as the output would be bigger than the input > Feature creep then becomes the analysis > # Add another MR class which runs through all failing tests and creates a new > list of test classes that failed. This could be rescheduled on different > runs, and makes for a faster cycle (only run failing tests until they work) > # Add something to only get failing tests, summarise them (somehow) in a user > readable form > # Something to get partially failing tests and highlight machine differences. > # Add something to compare tests over time, detect those which are getting > slower? > # an MR to regenerate the classic Ant junit XML reports, for presentation in > other tools (like hudson) -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10143) replace WritableFactories's hashmap with ConcurrentHashMap
[ https://issues.apache.org/jira/browse/HADOOP-10143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13876672#comment-13876672 ] Andrew Wang commented on HADOOP-10143: -- I'm +1 too, thanks guys. I think it's safe for branch-2 as well. > replace WritableFactories's hashmap with ConcurrentHashMap > -- > > Key: HADOOP-10143 > URL: https://issues.apache.org/jira/browse/HADOOP-10143 > Project: Hadoop Common > Issue Type: Improvement > Components: io >Affects Versions: 2.0.0-alpha, 2.2.0 >Reporter: Liang Xie >Assignee: Liang Xie > Attachments: HADOOP-10143.txt > > > We observed a lock contend hotspot from a HBase cluster: > "IPC Reader 9 on port 12600" daemon prio=10 tid=0x7f85b8aceed0 nid=0x4be8 > waiting for monitor entry [0x7f8501c57000] >java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.hadoop.io.WritableFactories.getFactory(WritableFactories.java:44) > - locked <0x0007fd1328a8> (a java.lang.Class for > org.apache.hadoop.io.WritableFactories) > at > org.apache.hadoop.io.WritableFactories.newInstance(WritableFactories.java:49) > at > org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:680) > at > org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:586) > at > org.apache.hadoop.hbase.client.MultiAction.readFields(MultiAction.java:116) > at > org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:682) > at > org.apache.hadoop.hbase.ipc.Invocation.readFields(Invocation.java:126) > at > org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.processData(SecureServer.java:618) > at > org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.processOneRpc(SecureServer.java:596) > at > org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.saslReadAndProcess(SecureServer.java:362) > at > org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.readAndProcess(SecureServer.java:492) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:770) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:561) > - locked <0x00043da3fea0> (a > org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:536) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > "IPC Reader 7 on port 12600" daemon prio=10 tid=0x7f85b8a99df0 nid=0x4be6 > waiting for monitor entry [0x7f8501cd9000] >java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.hadoop.io.WritableFactories.getFactory(WritableFactories.java:44) > - locked <0x0007fd1328a8> (a java.lang.Class for > org.apache.hadoop.io.WritableFactories) > at > org.apache.hadoop.io.WritableFactories.newInstance(WritableFactories.java:49) > at > org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:680) > at > org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:586) > at > org.apache.hadoop.hbase.client.MultiAction.readFields(MultiAction.java:116) > at > org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:682) > at > org.apache.hadoop.hbase.ipc.Invocation.readFields(Invocation.java:126) > at > org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.processData(SecureServer.java:618) > at > org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.processOneRpc(SecureServer.java:596) > at > org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.saslReadAndProcess(SecureServer.java:362) > at > org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.readAndProcess(SecureServer.java:492) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:770) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:561) > - locked <0x00043da232e8> (a > org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:536) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > "I
[jira] [Commented] (HADOOP-10143) replace WritableFactories's hashmap with ConcurrentHashMap
[ https://issues.apache.org/jira/browse/HADOOP-10143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13876640#comment-13876640 ] stack commented on HADOOP-10143: I will commit later today unless objection. Thanks. > replace WritableFactories's hashmap with ConcurrentHashMap > -- > > Key: HADOOP-10143 > URL: https://issues.apache.org/jira/browse/HADOOP-10143 > Project: Hadoop Common > Issue Type: Improvement > Components: io >Affects Versions: 2.0.0-alpha, 2.2.0 >Reporter: Liang Xie >Assignee: Liang Xie > Attachments: HADOOP-10143.txt > > > We observed a lock contend hotspot from a HBase cluster: > "IPC Reader 9 on port 12600" daemon prio=10 tid=0x7f85b8aceed0 nid=0x4be8 > waiting for monitor entry [0x7f8501c57000] >java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.hadoop.io.WritableFactories.getFactory(WritableFactories.java:44) > - locked <0x0007fd1328a8> (a java.lang.Class for > org.apache.hadoop.io.WritableFactories) > at > org.apache.hadoop.io.WritableFactories.newInstance(WritableFactories.java:49) > at > org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:680) > at > org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:586) > at > org.apache.hadoop.hbase.client.MultiAction.readFields(MultiAction.java:116) > at > org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:682) > at > org.apache.hadoop.hbase.ipc.Invocation.readFields(Invocation.java:126) > at > org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.processData(SecureServer.java:618) > at > org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.processOneRpc(SecureServer.java:596) > at > org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.saslReadAndProcess(SecureServer.java:362) > at > org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.readAndProcess(SecureServer.java:492) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:770) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:561) > - locked <0x00043da3fea0> (a > org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:536) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > "IPC Reader 7 on port 12600" daemon prio=10 tid=0x7f85b8a99df0 nid=0x4be6 > waiting for monitor entry [0x7f8501cd9000] >java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.hadoop.io.WritableFactories.getFactory(WritableFactories.java:44) > - locked <0x0007fd1328a8> (a java.lang.Class for > org.apache.hadoop.io.WritableFactories) > at > org.apache.hadoop.io.WritableFactories.newInstance(WritableFactories.java:49) > at > org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:680) > at > org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:586) > at > org.apache.hadoop.hbase.client.MultiAction.readFields(MultiAction.java:116) > at > org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:682) > at > org.apache.hadoop.hbase.ipc.Invocation.readFields(Invocation.java:126) > at > org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.processData(SecureServer.java:618) > at > org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.processOneRpc(SecureServer.java:596) > at > org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.saslReadAndProcess(SecureServer.java:362) > at > org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.readAndProcess(SecureServer.java:492) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:770) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:561) > - locked <0x00043da232e8> (a > org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:536) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > "IPC Reader 5 on port 1260
[jira] [Resolved] (HADOOP-10213) Fix bugs parsing ACL spec in FsShell setfacl.
[ https://issues.apache.org/jira/browse/HADOOP-10213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth resolved HADOOP-10213. Resolution: Fixed Fix Version/s: HDFS ACLs (HDFS-4685) Hadoop Flags: Reviewed +1 for the patch. Thanks for fixing all of this, Vinay. I've committed it to the feature branch. > Fix bugs parsing ACL spec in FsShell setfacl. > - > > Key: HADOOP-10213 > URL: https://issues.apache.org/jira/browse/HADOOP-10213 > Project: Hadoop Common > Issue Type: Bug > Components: tools >Affects Versions: HDFS ACLs (HDFS-4685) >Reporter: Chris Nauroth >Assignee: Vinay > Fix For: HDFS ACLs (HDFS-4685) > > Attachments: HADOOP-10213.patch, HADOOP-10213.patch, > HADOOP-10213.patch, HADOOP-10213.patch, HADOOP-10213.patch > > > When calling setfacl -x to remove ACL entries, it does not make sense for the > entries in the ACL spec to contain permissions. The permissions should be > unspecified, and the CLI should return an error if the user attempts to > provide permissions. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-9296) Authenticating users from different realm without a trust relationship
[ https://issues.apache.org/jira/browse/HADOOP-9296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13876615#comment-13876615 ] Alejandro Abdelnur commented on HADOOP-9296: [~daryn], you stated "The root cause is the client "guesses" the principal for a remote NN based on the principal of its default NN.", wouldn't the approach like the one you doing for HTTP SPNEGO, in HADOOP-10158, work for RPC as well? Then we would not need the servers to advertise their principal at all, no? > Authenticating users from different realm without a trust relationship > -- > > Key: HADOOP-9296 > URL: https://issues.apache.org/jira/browse/HADOOP-9296 > Project: Hadoop Common > Issue Type: Improvement > Components: security >Reporter: Benoy Antony >Assignee: Benoy Antony > Attachments: HADOOP-9296-1.1.patch, HADOOP-9296.patch, > HADOOP-9296.patch, multirealm.pdf > > > Hadoop Masters (JobTracker and NameNode) and slaves (Data Node and > TaskTracker) are part of the Hadoop domain, controlled by Hadoop Active > Directory. > The users belong to the CORP domain, controlled by the CORP Active Directory. > In the absence of a one way trust from HADOOP DOMAIN to CORP DOMAIN, how will > Hadoop Servers (JobTracker, NameNode) authenticate CORP users ? > The solution and implementation details are in the attachement -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10158) SPNEGO should work with multiple interfaces/SPNs.
[ https://issues.apache.org/jira/browse/HADOOP-10158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13876575#comment-13876575 ] Alejandro Abdelnur commented on HADOOP-10158: - We should mandate HTTP. Regarding different realms, then we verify, in the provided principals, there are no 2 principals with same host and different realm, as the only way to determine the realm is indirectly via the host extracted from the request URL. > SPNEGO should work with multiple interfaces/SPNs. > - > > Key: HADOOP-10158 > URL: https://issues.apache.org/jira/browse/HADOOP-10158 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.2.0 >Reporter: Kihwal Lee >Assignee: Daryn Sharp > Attachments: HADOOP-10158.patch, HADOOP-10158_multiplerealms.patch, > HADOOP-10158_multiplerealms.patch > > > This is the list of internal servlets added by namenode. > | Name | Auth | Need to be accessible by end users | > | StartupProgressServlet | none | no | > | GetDelegationTokenServlet | internal SPNEGO | yes | > | RenewDelegationTokenServlet | internal SPNEGO | yes | > | CancelDelegationTokenServlet | internal SPNEGO | yes | > | FsckServlet | internal SPNEGO | yes | > | GetImageServlet | internal SPNEGO | no | > | ListPathsServlet | token in query | yes | > | FileDataServlet | token in query | yes | > | FileChecksumServlets | token in query | yes | > | ContentSummaryServlet | token in query | yes | > GetDelegationTokenServlet, RenewDelegationTokenServlet, > CancelDelegationTokenServlet and FsckServlet are accessed by end users, but > hard-coded to use the internal SPNEGO filter. > If a name node HTTP server binds to multiple external IP addresses, the > internal SPNEGO service principal name may not work with an address to which > end users are connecting. The current SPNEGO implementation in Hadoop is > limited to use a single service principal per filter. > If the underlying hadoop kerberos authentication handler cannot easily be > modified, we can at least create a separate auth filter for the end-user > facing servlets so that their service principals can be independently > configured. If not defined, it should fall back to the current behavior. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10195) swiftfs object list stops at 10000 objects
[ https://issues.apache.org/jira/browse/HADOOP-10195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13876493#comment-13876493 ] David Dobbins commented on HADOOP-10195: Is there anything else I need to do here? > swiftfs object list stops at 1 objects > -- > > Key: HADOOP-10195 > URL: https://issues.apache.org/jira/browse/HADOOP-10195 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.3.0 >Reporter: David Dobbins >Assignee: David Dobbins > Attachments: hadoop-10195.patch, hadoop-10195.patch > > > listing objects in a container in swift is limited to 1 objects per > request. swiftfs only makes one request and is therefore limited to the first > 1 objects in the container, ignoring any remaining objects -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HADOOP-10143) replace WritableFactories's hashmap with ConcurrentHashMap
[ https://issues.apache.org/jira/browse/HADOOP-10143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13876306#comment-13876306 ] Liang Xie commented on HADOOP-10143: [~saint@gmail.com], we had deployed this modification into some clusters, so far looks good. To me, it'd better let it into trunk now. > replace WritableFactories's hashmap with ConcurrentHashMap > -- > > Key: HADOOP-10143 > URL: https://issues.apache.org/jira/browse/HADOOP-10143 > Project: Hadoop Common > Issue Type: Improvement > Components: io >Affects Versions: 2.0.0-alpha, 2.2.0 >Reporter: Liang Xie >Assignee: Liang Xie > Attachments: HADOOP-10143.txt > > > We observed a lock contend hotspot from a HBase cluster: > "IPC Reader 9 on port 12600" daemon prio=10 tid=0x7f85b8aceed0 nid=0x4be8 > waiting for monitor entry [0x7f8501c57000] >java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.hadoop.io.WritableFactories.getFactory(WritableFactories.java:44) > - locked <0x0007fd1328a8> (a java.lang.Class for > org.apache.hadoop.io.WritableFactories) > at > org.apache.hadoop.io.WritableFactories.newInstance(WritableFactories.java:49) > at > org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:680) > at > org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:586) > at > org.apache.hadoop.hbase.client.MultiAction.readFields(MultiAction.java:116) > at > org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:682) > at > org.apache.hadoop.hbase.ipc.Invocation.readFields(Invocation.java:126) > at > org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.processData(SecureServer.java:618) > at > org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.processOneRpc(SecureServer.java:596) > at > org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.saslReadAndProcess(SecureServer.java:362) > at > org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.readAndProcess(SecureServer.java:492) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:770) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:561) > - locked <0x00043da3fea0> (a > org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:536) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > "IPC Reader 7 on port 12600" daemon prio=10 tid=0x7f85b8a99df0 nid=0x4be6 > waiting for monitor entry [0x7f8501cd9000] >java.lang.Thread.State: BLOCKED (on object monitor) > at > org.apache.hadoop.io.WritableFactories.getFactory(WritableFactories.java:44) > - locked <0x0007fd1328a8> (a java.lang.Class for > org.apache.hadoop.io.WritableFactories) > at > org.apache.hadoop.io.WritableFactories.newInstance(WritableFactories.java:49) > at > org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:680) > at > org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:586) > at > org.apache.hadoop.hbase.client.MultiAction.readFields(MultiAction.java:116) > at > org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:682) > at > org.apache.hadoop.hbase.ipc.Invocation.readFields(Invocation.java:126) > at > org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.processData(SecureServer.java:618) > at > org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.processOneRpc(SecureServer.java:596) > at > org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.saslReadAndProcess(SecureServer.java:362) > at > org.apache.hadoop.hbase.ipc.SecureServer$SecureConnection.readAndProcess(SecureServer.java:492) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:770) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:561) > - locked <0x00043da232e8> (a > org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:536) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolEx
[jira] [Commented] (HADOOP-8061) Back port trunk metrics2 changes to 1.x branch
[ https://issues.apache.org/jira/browse/HADOOP-8061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13876270#comment-13876270 ] Harsh J commented on HADOOP-8061: - Is this still valid? May we close out? > Back port trunk metrics2 changes to 1.x branch > -- > > Key: HADOOP-8061 > URL: https://issues.apache.org/jira/browse/HADOOP-8061 > Project: Hadoop Common > Issue Type: Improvement > Components: metrics >Affects Versions: 1.0.0 >Reporter: Luke Lu > > For hysterical raisins, metrics2 code in hadoop 1.x branch and trunk > diverged. It looks like HBase needs to support both 1.x and 0.23+ for a > while, hence the more urgent need to clean up the situation. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
