[jira] [Created] (HDFS-3084) FenceMethod.tryFence() and ShellCommandFencer should pass namenodeId as well as host:port
FenceMethod.tryFence() and ShellCommandFencer should pass namenodeId as well as host:port - Key: HDFS-3084 URL: https://issues.apache.org/jira/browse/HDFS-3084 Project: Hadoop HDFS Issue Type: Improvement Components: ha Affects Versions: 0.24.0, 0.23.3 Reporter: Philip Zeyliger The FenceMethod interface passes along the host:port of the NN that needs to be fenced. That's great for the common case. However, it's likely necessary to have extra configuration parameters for fencing, and these are typically keyed off the nameserviceId.namenodeId (if, for nothing else, consistency with all the other parameters that are keyed off of namespaceId.namenodeId). Obviously this can be backed out from the host:port, but it's inconvenient, and requires iterating through all the configs. The shell interface exhibits the same issue: host:port is great for most fencers, but if you need extra configs (like the host:port of the power supply unit), those are harder to pipe through without the namenodeId. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3083) HA+security: failed to run a mapred job from yarn after a manual failover
HA+security: failed to run a mapred job from yarn after a manual failover
-
Key: HDFS-3083
URL: https://issues.apache.org/jira/browse/HDFS-3083
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, security
Affects Versions: 0.24.0, 0.23.3
Reporter: Mingjie Lai
Priority: Critical
Fix For: 0.24.0, 0.23.3
Steps to reproduce:
- turned on ha and security
- run a mapred job, and wait to finish
- failover to another namenode
- run the mapred job again, it fails.
Checking the job delegation token, it's still indicate the original active
namenode. It causes nm failed to obtain a dt for the new nn. (?)
{code}
$ hdfs dfs -cat
hdfs://ns1:8020/tmp/hadoop-yarn/staging/yarn/.staging/job_1331619043691_0001/appTokens
HDTS
ha-hdfs:ns1@(yarn/nn1.hadoop.local@HADOOP.LOCALDOMAINyarn�6
�L��6.�ЛFs��r�%�B�'��{pR�HDFS_DELEGATION_TOKEN
ha-hdfs:ns
{code}
Exceptions:
{code}
12/03/13 06:19:44 INFO mapred.ResourceMgrDelegate: Submitted application
application_1331619043691_0002 to ResourceManager at
nn1.hadoop.local/10.177.23.38:7090
12/03/13 06:19:45 INFO mapreduce.Job: The url to track the job:
http://nn1.hadoop.local:7050/proxy/application_1331619043691_0002/
12/03/13 06:19:45 INFO mapreduce.Job: Running job: job_1331619043691_0002
12/03/13 06:19:47 INFO mapreduce.Job: Job job_1331619043691_0002 running in
uber mode : false
12/03/13 06:19:47 INFO mapreduce.Job: map 0% reduce 0%
12/03/13 06:19:47 INFO mapreduce.Job: Job job_1331619043691_0002 failed with
state FAILED due to: Application application_1331619043691_0002 failed 1 times
due to AM Container for appattempt_1331619043691_0002_01 exited with
exitCode: -1000 due to: RemoteTrace:
org.apache.hadoop.security.token.SecretManager$InvalidToken: token
(HDFS_DELEGATION_TOKEN token 40 for yarn) can't be found in cache
at org.apache.hadoop.ipc.Client.call(Client.java:1159)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:188)
at $Proxy28.getFileInfo(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:622)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83)
at $Proxy29.getFileInfo(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1260)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:718)
at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:88)
at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:49)
at org.apache.hadoop.yarn.util.FSDownload$1.run(FSDownload.java:157)
at org.apache.hadoop.yarn.util.FSDownload$1.run(FSDownload.java:155)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:153)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:49)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
at LocalTrace:
org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl:
token (HDFS_DELEGATION_TOKEN token 40 for yarn) can't be found in cache
at
org.apache.hadoop.yarn.server.nodemanager.api.protocolrecords.impl.pb.LocalResourceStatusPBImpl.convertFromProtoFormat(LocalResourceStatusPBImpl.java:217)
at
org.apache.hadoop.yarn.server.nodemanager.api.protocolrecords.impl.pb.LocalResourceStatusPBImpl.getException(LocalResourceStatusPBImpl.java:147)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.lo
[jira] [Commented] (HDFS-2303) Unbundle jsvc
[ https://issues.apache.org/jira/browse/HDFS-2303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13228243#comment-13228243 ] Hudson commented on HDFS-2303: -- Integrated in Hadoop-Common-0.23-Commit #675 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/675/]) HDFS-2303. svn merge -c 1299963 from trunk (Revision 1299964) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1299964 Files : * /hadoop/common/branches/branch-0.23 * /hadoop/common/branches/branch-0.23/hadoop-common-project * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/pom.xml * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-examples * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++ * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job * /hadoop/common/branches/branch-0.23/hadoop-project * /hadoop/common/branches/branch-0.23/hadoop-project/src/site > Unbundle jsvc > - > > Key: HDFS-2303 > URL: https://issues.apache.org/jira/browse/HDFS-2303 > Project: Hadoop HDFS > Issue Type: Improvement > Components: build, scripts >Affects Versions: 0.23.0, 0.24.0 >Reporter: Roman Shaposhnik >Assignee: Mingjie Lai > Fix For: 0.23.3 > > Attachments: HDFS-2303-2.patch.txt, HDFS-2303-3-trunk.patch, > HDFS-2303-4-trunk.patch, HDFS-2303-5-modcommon-trunk.patch, > HDFS-2303-5-modcommon-trunk.patch, HDFS-2303-5-modcommon-trunk.patch, > HDFS-2303-5-trunk.patch, HDFS-2303.patch.txt > > > It would be n
[jira] [Commented] (HDFS-2303) Unbundle jsvc
[ https://issues.apache.org/jira/browse/HDFS-2303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13228239#comment-13228239 ] Hudson commented on HDFS-2303: -- Integrated in Hadoop-Hdfs-0.23-Commit #666 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/666/]) HDFS-2303. svn merge -c 1299963 from trunk (Revision 1299964) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1299964 Files : * /hadoop/common/branches/branch-0.23 * /hadoop/common/branches/branch-0.23/hadoop-common-project * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/pom.xml * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-examples * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++ * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job * /hadoop/common/branches/branch-0.23/hadoop-project * /hadoop/common/branches/branch-0.23/hadoop-project/src/site > Unbundle jsvc > - > > Key: HDFS-2303 > URL: https://issues.apache.org/jira/browse/HDFS-2303 > Project: Hadoop HDFS > Issue Type: Improvement > Components: build, scripts >Affects Versions: 0.23.0, 0.24.0 >Reporter: Roman Shaposhnik >Assignee: Mingjie Lai > Fix For: 0.23.3 > > Attachments: HDFS-2303-2.patch.txt, HDFS-2303-3-trunk.patch, > HDFS-2303-4-trunk.patch, HDFS-2303-5-modcommon-trunk.patch, > HDFS-2303-5-modcommon-trunk.patch, HDFS-2303-5-modcommon-trunk.patch, > HDFS-2303-5-trunk.patch, HDFS-2303.patch.txt > > > It would be nice
[jira] [Commented] (HDFS-2303) Unbundle jsvc
[ https://issues.apache.org/jira/browse/HDFS-2303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13228234#comment-13228234 ] Hudson commented on HDFS-2303: -- Integrated in Hadoop-Hdfs-trunk-Commit #1944 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1944/]) HDFS-2303. Unbundle jsvc. Contributed by Roman Shaposhnik and Mingjie Lai. (Revision 1299963) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1299963 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs > Unbundle jsvc > - > > Key: HDFS-2303 > URL: https://issues.apache.org/jira/browse/HDFS-2303 > Project: Hadoop HDFS > Issue Type: Improvement > Components: build, scripts >Affects Versions: 0.23.0, 0.24.0 >Reporter: Roman Shaposhnik >Assignee: Mingjie Lai > Fix For: 0.23.3 > > Attachments: HDFS-2303-2.patch.txt, HDFS-2303-3-trunk.patch, > HDFS-2303-4-trunk.patch, HDFS-2303-5-modcommon-trunk.patch, > HDFS-2303-5-modcommon-trunk.patch, HDFS-2303-5-modcommon-trunk.patch, > HDFS-2303-5-trunk.patch, HDFS-2303.patch.txt > > > It would be nice to recompile jsvc as part of the native profile. This has a > number of benefits including an ability to re-generate all binary artifacts, > etc. Most of all, however, it will provide a way to generate jsvc on Linux > distributions that don't have matching libc -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2303) Unbundle jsvc
[ https://issues.apache.org/jira/browse/HDFS-2303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13228195#comment-13228195 ] Hudson commented on HDFS-2303: -- Integrated in Hadoop-Mapreduce-0.23-Commit #683 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/683/]) HDFS-2303. svn merge -c 1299963 from trunk (Revision 1299964) Result = ABORTED eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1299964 Files : * /hadoop/common/branches/branch-0.23 * /hadoop/common/branches/branch-0.23/hadoop-common-project * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/pom.xml * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-examples * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++ * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job * /hadoop/common/branches/branch-0.23/hadoop-project * /hadoop/common/branches/branch-0.23/hadoop-project/src/site > Unbundle jsvc > - > > Key: HDFS-2303 > URL: https://issues.apache.org/jira/browse/HDFS-2303 > Project: Hadoop HDFS > Issue Type: Improvement > Components: build, scripts >Affects Versions: 0.23.0, 0.24.0 >Reporter: Roman Shaposhnik >Assignee: Mingjie Lai > Fix For: 0.23.3 > > Attachments: HDFS-2303-2.patch.txt, HDFS-2303-3-trunk.patch, > HDFS-2303-4-trunk.patch, HDFS-2303-5-modcommon-trunk.patch, > HDFS-2303-5-modcommon-trunk.patch, HDFS-2303-5-modcommon-trunk.patch, > HDFS-2303-5-trunk.patch, HDFS-2303.patch.txt > > > It woul
[jira] [Commented] (HDFS-1512) BlockSender calls deprecated method getReplica
[ https://issues.apache.org/jira/browse/HDFS-1512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13228190#comment-13228190 ] Amin Bandeali commented on HDFS-1512: - I am out of town for next 2-3 weeks. I can take a look after. On Sat, Mar 10, 2012 at 8:26 AM, Uma Maheswara Rao G (Commented) (JIRA) < > BlockSender calls deprecated method getReplica > -- > > Key: HDFS-1512 > URL: https://issues.apache.org/jira/browse/HDFS-1512 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Reporter: Eli Collins >Assignee: Amin Bandeali > Labels: newbie > Attachments: HDFS-1512.patch, HDFS-1512.patch > > > HDFS-680 deprecated FSDatasetInterface#getReplica, however it is still used > by BlockSender which still maintains a Replica member. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2303) Unbundle jsvc
[ https://issues.apache.org/jira/browse/HDFS-2303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13228189#comment-13228189 ] Hudson commented on HDFS-2303: -- Integrated in Hadoop-Mapreduce-trunk-Commit #1878 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1878/]) HDFS-2303. Unbundle jsvc. Contributed by Roman Shaposhnik and Mingjie Lai. (Revision 1299963) Result = ABORTED eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1299963 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/packages/templates/conf/hadoop-env.sh * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/pom.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs > Unbundle jsvc > - > > Key: HDFS-2303 > URL: https://issues.apache.org/jira/browse/HDFS-2303 > Project: Hadoop HDFS > Issue Type: Improvement > Components: build, scripts >Affects Versions: 0.23.0, 0.24.0 >Reporter: Roman Shaposhnik >Assignee: Mingjie Lai > Fix For: 0.23.3 > > Attachments: HDFS-2303-2.patch.txt, HDFS-2303-3-trunk.patch, > HDFS-2303-4-trunk.patch, HDFS-2303-5-modcommon-trunk.patch, > HDFS-2303-5-modcommon-trunk.patch, HDFS-2303-5-modcommon-trunk.patch, > HDFS-2303-5-trunk.patch, HDFS-2303.patch.txt > > > It would be nice to recompile jsvc as part of the native profile. This has a > number of benefits including an ability to re-generate all binary artifacts, > etc. Most of all, however, it will provide a way to generate jsvc on Linux > distributions that don't have matching libc -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3004) Implement Recovery Mode
[ https://issues.apache.org/jira/browse/HDFS-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13228186#comment-13228186 ] Eli Collins commented on HDFS-3004: --- bq. The first choice isn't always skip-- sometimes it's "truncate." Why would a user choose "always choose 1st"? The user doesn't know if future errors are skippable or not-skippable so when they select "always choose first" on a skippable prompt they don't know that they're signing up for a future truncate. Seems like we need to make the order consistent if we're going to give people a "Yes to all" option. - Per above, What is the "TODO: attempt to resynchronize stream here" for? - Should the catch of Throwable catch IOException like it used to? We're not trying to catch new types of exceptions in the non-recovery case right? - Do we need to sanity check dfs.namenode.num.checkpoints.retained in recovery mode? Ie since we do roll the log is there anyway that we could load an image/log, truncate it in recovery mode, then not retain the old log? - TestRecoverTruncatedEditLog still doesn't check that we actually truncated the log, eg even if we didn't truncate the log the test would still pass because the directory would still be there - What testing have you done? Would be good to try this on a tarball build with various corrupt and non-corrupt images/logs. > Implement Recovery Mode > --- > > Key: HDFS-3004 > URL: https://issues.apache.org/jira/browse/HDFS-3004 > Project: Hadoop HDFS > Issue Type: New Feature > Components: tools >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-3004.010.patch, > HDFS-3004__namenode_recovery_tool.txt > > > When the NameNode metadata is corrupt for some reason, we want to be able to > fix it. Obviously, we would prefer never to get in this case. In a perfect > world, we never would. However, bad data on disk can happen from time to > time, because of hardware errors or misconfigurations. In the past we have > had to correct it manually, which is time-consuming and which can result in > downtime. > Recovery mode is initialized by the system administrator. When the NameNode > starts up in Recovery Mode, it will try to load the FSImage file, apply all > the edits from the edits log, and then write out a new image. Then it will > shut down. > Unlike in the normal startup process, the recovery mode startup process will > be interactive. When the NameNode finds something that is inconsistent, it > will prompt the operator as to what it should do. The operator can also > choose to take the first option for all prompts by starting up with the '-f' > flag, or typing 'a' at one of the prompts. > I have reused as much code as possible from the NameNode in this tool. > Hopefully, the effort that was spent developing this will also make the > NameNode editLog and image processing even more robust than it already is. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2303) Unbundle jsvc
[ https://issues.apache.org/jira/browse/HDFS-2303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-2303: -- Target Version/s: (was: 0.23.3) Fix Version/s: 0.23.3 > Unbundle jsvc > - > > Key: HDFS-2303 > URL: https://issues.apache.org/jira/browse/HDFS-2303 > Project: Hadoop HDFS > Issue Type: Improvement > Components: build, scripts >Affects Versions: 0.23.0, 0.24.0 >Reporter: Roman Shaposhnik >Assignee: Mingjie Lai > Fix For: 0.23.3 > > Attachments: HDFS-2303-2.patch.txt, HDFS-2303-3-trunk.patch, > HDFS-2303-4-trunk.patch, HDFS-2303-5-modcommon-trunk.patch, > HDFS-2303-5-modcommon-trunk.patch, HDFS-2303-5-modcommon-trunk.patch, > HDFS-2303-5-trunk.patch, HDFS-2303.patch.txt > > > It would be nice to recompile jsvc as part of the native profile. This has a > number of benefits including an ability to re-generate all binary artifacts, > etc. Most of all, however, it will provide a way to generate jsvc on Linux > distributions that don't have matching libc -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-2303) Unbundle jsvc
[ https://issues.apache.org/jira/browse/HDFS-2303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins resolved HDFS-2303. --- Resolution: Fixed Hadoop Flags: Incompatible change,Reviewed I've committed this and merged to branch-23. Thank you Roman and Mingjie! > Unbundle jsvc > - > > Key: HDFS-2303 > URL: https://issues.apache.org/jira/browse/HDFS-2303 > Project: Hadoop HDFS > Issue Type: Improvement > Components: build, scripts >Affects Versions: 0.23.0, 0.24.0 >Reporter: Roman Shaposhnik >Assignee: Mingjie Lai > Attachments: HDFS-2303-2.patch.txt, HDFS-2303-3-trunk.patch, > HDFS-2303-4-trunk.patch, HDFS-2303-5-modcommon-trunk.patch, > HDFS-2303-5-modcommon-trunk.patch, HDFS-2303-5-modcommon-trunk.patch, > HDFS-2303-5-trunk.patch, HDFS-2303.patch.txt > > > It would be nice to recompile jsvc as part of the native profile. This has a > number of benefits including an ability to re-generate all binary artifacts, > etc. Most of all, however, it will provide a way to generate jsvc on Linux > distributions that don't have matching libc -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2303) Unbundle jsvc
[ https://issues.apache.org/jira/browse/HDFS-2303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-2303: -- Issue Type: Improvement (was: Bug) Release Note: To run secure Datanodes users must install jsvc for their platform and set JSVC_HOME to point to the location of jsvc in their environment. Summary: Unbundle jsvc (was: jsvc needs to be recompilable) > Unbundle jsvc > - > > Key: HDFS-2303 > URL: https://issues.apache.org/jira/browse/HDFS-2303 > Project: Hadoop HDFS > Issue Type: Improvement > Components: build, scripts >Affects Versions: 0.23.0, 0.24.0 >Reporter: Roman Shaposhnik >Assignee: Mingjie Lai > Attachments: HDFS-2303-2.patch.txt, HDFS-2303-3-trunk.patch, > HDFS-2303-4-trunk.patch, HDFS-2303-5-modcommon-trunk.patch, > HDFS-2303-5-modcommon-trunk.patch, HDFS-2303-5-modcommon-trunk.patch, > HDFS-2303-5-trunk.patch, HDFS-2303.patch.txt > > > It would be nice to recompile jsvc as part of the native profile. This has a > number of benefits including an ability to re-generate all binary artifacts, > etc. Most of all, however, it will provide a way to generate jsvc on Linux > distributions that don't have matching libc -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2303) jsvc needs to be recompilable
[ https://issues.apache.org/jira/browse/HDFS-2303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13228169#comment-13228169 ] Eli Collins commented on HDFS-2303: --- bq. -1 core tests. The patch failed the unit tests build Must be a bug in jenkins, the test results page shows no failures, and this patch modifies shell scripts not used by the tests. > jsvc needs to be recompilable > - > > Key: HDFS-2303 > URL: https://issues.apache.org/jira/browse/HDFS-2303 > Project: Hadoop HDFS > Issue Type: Bug > Components: build, scripts >Affects Versions: 0.23.0, 0.24.0 >Reporter: Roman Shaposhnik >Assignee: Mingjie Lai > Attachments: HDFS-2303-2.patch.txt, HDFS-2303-3-trunk.patch, > HDFS-2303-4-trunk.patch, HDFS-2303-5-modcommon-trunk.patch, > HDFS-2303-5-modcommon-trunk.patch, HDFS-2303-5-modcommon-trunk.patch, > HDFS-2303-5-trunk.patch, HDFS-2303.patch.txt > > > It would be nice to recompile jsvc as part of the native profile. This has a > number of benefits including an ability to re-generate all binary artifacts, > etc. Most of all, however, it will provide a way to generate jsvc on Linux > distributions that don't have matching libc -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2303) jsvc needs to be recompilable
[ https://issues.apache.org/jira/browse/HDFS-2303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13228163#comment-13228163 ] Hadoop QA commented on HDFS-2303: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12518047/HDFS-2303-5-modcommon-trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed the unit tests build +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1989//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1989//console This message is automatically generated. > jsvc needs to be recompilable > - > > Key: HDFS-2303 > URL: https://issues.apache.org/jira/browse/HDFS-2303 > Project: Hadoop HDFS > Issue Type: Bug > Components: build, scripts >Affects Versions: 0.23.0, 0.24.0 >Reporter: Roman Shaposhnik >Assignee: Mingjie Lai > Attachments: HDFS-2303-2.patch.txt, HDFS-2303-3-trunk.patch, > HDFS-2303-4-trunk.patch, HDFS-2303-5-modcommon-trunk.patch, > HDFS-2303-5-modcommon-trunk.patch, HDFS-2303-5-modcommon-trunk.patch, > HDFS-2303-5-trunk.patch, HDFS-2303.patch.txt > > > It would be nice to recompile jsvc as part of the native profile. This has a > number of benefits including an ability to re-generate all binary artifacts, > etc. Most of all, however, it will provide a way to generate jsvc on Linux > distributions that don't have matching libc -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3005) ConcurrentModificationException in FSDataset$FSVolume.getDfsUsed(..)
[
https://issues.apache.org/jira/browse/HDFS-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated HDFS-3005:
-
Attachment: h3005_20120312.patch
Hi VINAYAKUMAR,
Thanks for posting a patch. ConcurrentHashMap won't work since the map is
being iterated.
h3005_20120312.patch: synchronizes dataset and changes the inner FSDataset
classes to static.
> ConcurrentModificationException in FSDataset$FSVolume.getDfsUsed(..)
>
>
> Key: HDFS-3005
> URL: https://issues.apache.org/jira/browse/HDFS-3005
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
>Affects Versions: 0.24.0
>Reporter: Tsz Wo (Nicholas), SZE
> Attachments: HDFS-3005.patch, h3005_20120312.patch
>
>
> Saw this in [build
> #1888|https://builds.apache.org/job/PreCommit-HDFS-Build/1888//testReport/org.apache.hadoop.hdfs.server.datanode/TestMulitipleNNDataBlockScanner/testBlockScannerAfterRestart/].
> {noformat}
> java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
> at java.util.HashMap$EntryIterator.next(HashMap.java:834)
> at java.util.HashMap$EntryIterator.next(HashMap.java:832)
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.getDfsUsed(FSDataset.java:557)
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolumeSet.getDfsUsed(FSDataset.java:809)
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolumeSet.access$1400(FSDataset.java:774)
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.getDfsUsed(FSDataset.java:1124)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.sendHeartBeat(BPOfferService.java:406)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.offerService(BPOfferService.java:490)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.run(BPOfferService.java:635)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-3005) ConcurrentModificationException in FSDataset$FSVolume.getDfsUsed(..)
[
https://issues.apache.org/jira/browse/HDFS-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE reassigned HDFS-3005:
Assignee: Tsz Wo (Nicholas), SZE
> ConcurrentModificationException in FSDataset$FSVolume.getDfsUsed(..)
>
>
> Key: HDFS-3005
> URL: https://issues.apache.org/jira/browse/HDFS-3005
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node
>Affects Versions: 0.24.0
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Attachments: HDFS-3005.patch, h3005_20120312.patch
>
>
> Saw this in [build
> #1888|https://builds.apache.org/job/PreCommit-HDFS-Build/1888//testReport/org.apache.hadoop.hdfs.server.datanode/TestMulitipleNNDataBlockScanner/testBlockScannerAfterRestart/].
> {noformat}
> java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
> at java.util.HashMap$EntryIterator.next(HashMap.java:834)
> at java.util.HashMap$EntryIterator.next(HashMap.java:832)
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.getDfsUsed(FSDataset.java:557)
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolumeSet.getDfsUsed(FSDataset.java:809)
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolumeSet.access$1400(FSDataset.java:774)
> at
> org.apache.hadoop.hdfs.server.datanode.FSDataset.getDfsUsed(FSDataset.java:1124)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.sendHeartBeat(BPOfferService.java:406)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.offerService(BPOfferService.java:490)
> at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.run(BPOfferService.java:635)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3082) Clean up FSDatasetInterface
[ https://issues.apache.org/jira/browse/HDFS-3082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-3082: - Status: Patch Available (was: Open) > Clean up FSDatasetInterface > --- > > Key: HDFS-3082 > URL: https://issues.apache.org/jira/browse/HDFS-3082 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: data-node >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Attachments: h3082_20120312.patch > > > Some methods defined in FSDatasetInterface are actually not needed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3082) Clean up FSDatasetInterface
[ https://issues.apache.org/jira/browse/HDFS-3082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-3082: - Attachment: h3082_20120312.patch h3082_20120312.patch: removes a few methods from FSDatasetInterface and changes DataNode.data to package private. > Clean up FSDatasetInterface > --- > > Key: HDFS-3082 > URL: https://issues.apache.org/jira/browse/HDFS-3082 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: data-node >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Attachments: h3082_20120312.patch > > > Some methods defined in FSDatasetInterface are actually not needed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3082) Clean up FSDatasetInterface
Clean up FSDatasetInterface --- Key: HDFS-3082 URL: https://issues.apache.org/jira/browse/HDFS-3082 Project: Hadoop HDFS Issue Type: Sub-task Components: data-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Some methods defined in FSDatasetInterface are actually not needed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-3078) 2NN https port setting is broken
[ https://issues.apache.org/jira/browse/HDFS-3078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins resolved HDFS-3078. --- Resolution: Fixed Thanks Todd. I've committed this to branch-1. > 2NN https port setting is broken > > > Key: HDFS-3078 > URL: https://issues.apache.org/jira/browse/HDFS-3078 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 1.0.0 >Reporter: Eli Collins >Assignee: Eli Collins > Attachments: hdfs-3078.txt > > > The code in SecondaryNameNode.java to set the https port is broken, if the > port is set it sets the bind addr to "addr:addr:port" which is bogus. Even if > it did work it uses port 0 instead of port 50490 (default listed in > ./src/packages/templates/conf/hdfs-site.xml). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3078) 2NN https port setting is broken
[ https://issues.apache.org/jira/browse/HDFS-3078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-3078: -- Attachment: hdfs-3078.txt > 2NN https port setting is broken > > > Key: HDFS-3078 > URL: https://issues.apache.org/jira/browse/HDFS-3078 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 1.0.0 >Reporter: Eli Collins >Assignee: Eli Collins > Attachments: hdfs-3078.txt > > > The code in SecondaryNameNode.java to set the https port is broken, if the > port is set it sets the bind addr to "addr:addr:port" which is bogus. Even if > it did work it uses port 0 instead of port 50490 (default listed in > ./src/packages/templates/conf/hdfs-site.xml). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (HDFS-3078) 2NN https port setting is broken
[ https://issues.apache.org/jira/browse/HDFS-3078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-3078 started by Eli Collins. > 2NN https port setting is broken > > > Key: HDFS-3078 > URL: https://issues.apache.org/jira/browse/HDFS-3078 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 1.0.0 >Reporter: Eli Collins >Assignee: Eli Collins > Attachments: hdfs-3078.txt > > > The code in SecondaryNameNode.java to set the https port is broken, if the > port is set it sets the bind addr to "addr:addr:port" which is bogus. Even if > it did work it uses port 0 instead of port 50490 (default listed in > ./src/packages/templates/conf/hdfs-site.xml). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3078) 2NN https port setting is broken
[ https://issues.apache.org/jira/browse/HDFS-3078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-3078: -- Attachment: (was: hdfs-3078.txt) > 2NN https port setting is broken > > > Key: HDFS-3078 > URL: https://issues.apache.org/jira/browse/HDFS-3078 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 1.0.0 >Reporter: Eli Collins >Assignee: Eli Collins > Attachments: hdfs-3078.txt > > > The code in SecondaryNameNode.java to set the https port is broken, if the > port is set it sets the bind addr to "addr:addr:port" which is bogus. Even if > it did work it uses port 0 instead of port 50490 (default listed in > ./src/packages/templates/conf/hdfs-site.xml). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3078) 2NN https port setting is broken
[ https://issues.apache.org/jira/browse/HDFS-3078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-3078: -- Hadoop Flags: Reviewed Summary: 2NN https port setting is broken (was: 2NN https port setting is broken in Hadoop 1.0) > 2NN https port setting is broken > > > Key: HDFS-3078 > URL: https://issues.apache.org/jira/browse/HDFS-3078 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 1.0.0 >Reporter: Eli Collins >Assignee: Eli Collins > Attachments: hdfs-3078.txt > > > The code in SecondaryNameNode.java to set the https port is broken, if the > port is set it sets the bind addr to "addr:addr:port" which is bogus. Even if > it did work it uses port 0 instead of port 50490 (default listed in > ./src/packages/templates/conf/hdfs-site.xml). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3070) hdfs balancer doesn't balance blocks between datanodes
[ https://issues.apache.org/jira/browse/HDFS-3070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13228068#comment-13228068 ] Eli Collins commented on HDFS-3070: --- Yea sounds like a bug in the method the balancer uses to determine the namenodes. > hdfs balancer doesn't balance blocks between datanodes > -- > > Key: HDFS-3070 > URL: https://issues.apache.org/jira/browse/HDFS-3070 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer >Affects Versions: 0.24.0 >Reporter: Stephen Chu > Attachments: unbalanced_nodes.png, unbalanced_nodes_inservice.png > > > I TeraGenerated data into DataNodes styx01 and styx02. Looking at the web UI, > both have over 3% disk usage. > Attached is a screenshot of the Live Nodes web UI. > On styx01, I run the _hdfs balancer_ command with threshold 1% and don't see > the blocks being balanced across all 4 datanodes (all blocks on styx01 and > styx02 stay put). > HA is currently enabled. > [schu@styx01 ~]$ hdfs haadmin -getServiceState nn1 > active > [schu@styx01 ~]$ hdfs balancer -threshold 1 > 12/03/08 10:10:32 INFO balancer.Balancer: Using a threshold of 1.0 > 12/03/08 10:10:32 INFO balancer.Balancer: namenodes = [] > 12/03/08 10:10:32 INFO balancer.Balancer: p = > Balancer.Parameters[BalancingPolicy.Node, threshold=1.0] > Time Stamp Iteration# Bytes Already Moved Bytes Left To Move > Bytes Being Moved > Balancing took 95.0 milliseconds > [schu@styx01 ~]$ > I believe with a threshold of 1% the balancer should trigger blocks being > moved across DataNodes, right? I am curious about the "namenode = []" from > the above output. > [schu@styx01 ~]$ hadoop version > Hadoop 0.24.0-SNAPSHOT > Subversion > git://styx01.sf.cloudera.com/home/schu/hadoop-common/hadoop-common-project/hadoop-common > -r f6a577d697bbcd04ffbc568167c97b79479ff319 > Compiled by schu on Thu Mar 8 15:32:50 PST 2012 > From source with checksum ec971a6e7316f7fbf471b617905856b8 > From > http://hadoop.apache.org/hdfs/docs/r0.21.0/api/org/apache/hadoop/hdfs/server/balancer/Balancer.html: > The threshold parameter is a fraction in the range of (0%, 100%) with a > default value of 10%. The threshold sets a target for whether the cluster is > balanced. A cluster is balanced if for each datanode, the utilization of the > node (ratio of used space at the node to total capacity of the node) differs > from the utilization of the (ratio of used space in the cluster to total > capacity of the cluster) by no more than the threshold value. The smaller the > threshold, the more balanced a cluster will become. It takes more time to run > the balancer for small threshold values. Also for a very small threshold the > cluster may not be able to reach the balanced state when applications write > and delete files concurrently. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3081) SshFenceByTcpPort uses netcat incorrectly
[
https://issues.apache.org/jira/browse/HDFS-3081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins updated HDFS-3081:
--
Target Version/s: 0.23.3 (was: 0.24.0)
> SshFenceByTcpPort uses netcat incorrectly
> -
>
> Key: HDFS-3081
> URL: https://issues.apache.org/jira/browse/HDFS-3081
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
>Affects Versions: 0.24.0
>Reporter: Philip Zeyliger
>Assignee: Todd Lipcon
>
> SshFencyByTcpPort currently assumes that the NN is listening on localhost.
> Typical setups have the namenode listening just on the hostname of the
> namenode, which would lead "nc -z" to not catch it.
> Here's an example in which the NN is running, listening on 8020, but doesn't
> respond to "localhost 8020".
> {noformat}
> [root@xxx ~]# lsof -P -p 5286 | grep -i listen
> java5286 root 110u IPv41772357 TCP xxx:8020
> (LISTEN)
> java5286 root 121u IPv41772397 TCP xxx:50070
> (LISTEN)
> [root@xxx ~]# nc -z localhost 8020
> [root@xxx ~]# nc -z xxx 8020
> Connection to xxx 8020 port [tcp/intu-ec-svcdisc] succeeded!
> {noformat}
> Here's the likely offending code:
> {code}
> LOG.info(
> "Indeterminate response from trying to kill service. " +
> "Verifying whether it is running using nc...");
> rc = execCommand(session, "nc -z localhost 8020");
> {code}
> Naively, we could rely on netcat to the correct hostname (since the NN ought
> to be listening on the hostname it's configured as), or just to use fuser.
> Fuser catches ports independently of what IPs they're bound to:
> {noformat}
> [root@xxx ~]# fuser 1234/tcp
> 1234/tcp: 6766 6768
> [root@xxx ~]# jobs
> [1]- Running nc -l localhost 1234 &
> [2]+ Running nc -l rhel56-18.ent.cloudera.com 1234 &
> [root@xxx ~]# sudo lsof -P | grep -i LISTEN | grep -i 1234
> nc 6766 root3u IPv42563626
> TCP localhost:1234 (LISTEN)
> nc 6768 root3u IPv42563671
> TCP xxx:1234 (LISTEN)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-3081) SshFenceByTcpPort uses netcat incorrectly
[
https://issues.apache.org/jira/browse/HDFS-3081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon reassigned HDFS-3081:
-
Assignee: Todd Lipcon
> SshFenceByTcpPort uses netcat incorrectly
> -
>
> Key: HDFS-3081
> URL: https://issues.apache.org/jira/browse/HDFS-3081
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha
>Affects Versions: 0.24.0
>Reporter: Philip Zeyliger
>Assignee: Todd Lipcon
>
> SshFencyByTcpPort currently assumes that the NN is listening on localhost.
> Typical setups have the namenode listening just on the hostname of the
> namenode, which would lead "nc -z" to not catch it.
> Here's an example in which the NN is running, listening on 8020, but doesn't
> respond to "localhost 8020".
> {noformat}
> [root@xxx ~]# lsof -P -p 5286 | grep -i listen
> java5286 root 110u IPv41772357 TCP xxx:8020
> (LISTEN)
> java5286 root 121u IPv41772397 TCP xxx:50070
> (LISTEN)
> [root@xxx ~]# nc -z localhost 8020
> [root@xxx ~]# nc -z xxx 8020
> Connection to xxx 8020 port [tcp/intu-ec-svcdisc] succeeded!
> {noformat}
> Here's the likely offending code:
> {code}
> LOG.info(
> "Indeterminate response from trying to kill service. " +
> "Verifying whether it is running using nc...");
> rc = execCommand(session, "nc -z localhost 8020");
> {code}
> Naively, we could rely on netcat to the correct hostname (since the NN ought
> to be listening on the hostname it's configured as), or just to use fuser.
> Fuser catches ports independently of what IPs they're bound to:
> {noformat}
> [root@xxx ~]# fuser 1234/tcp
> 1234/tcp: 6766 6768
> [root@xxx ~]# jobs
> [1]- Running nc -l localhost 1234 &
> [2]+ Running nc -l rhel56-18.ent.cloudera.com 1234 &
> [root@xxx ~]# sudo lsof -P | grep -i LISTEN | grep -i 1234
> nc 6766 root3u IPv42563626
> TCP localhost:1234 (LISTEN)
> nc 6768 root3u IPv42563671
> TCP xxx:1234 (LISTEN)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3081) SshFenceByTcpPort uses netcat incorrectly
SshFenceByTcpPort uses netcat incorrectly
-
Key: HDFS-3081
URL: https://issues.apache.org/jira/browse/HDFS-3081
Project: Hadoop HDFS
Issue Type: Bug
Components: ha
Affects Versions: 0.24.0
Reporter: Philip Zeyliger
SshFencyByTcpPort currently assumes that the NN is listening on localhost.
Typical setups have the namenode listening just on the hostname of the
namenode, which would lead "nc -z" to not catch it.
Here's an example in which the NN is running, listening on 8020, but doesn't
respond to "localhost 8020".
{noformat}
[root@xxx ~]# lsof -P -p 5286 | grep -i listen
java5286 root 110u IPv41772357 TCP xxx:8020
(LISTEN)
java5286 root 121u IPv41772397 TCP xxx:50070
(LISTEN)
[root@xxx ~]# nc -z localhost 8020
[root@xxx ~]# nc -z xxx 8020
Connection to xxx 8020 port [tcp/intu-ec-svcdisc] succeeded!
{noformat}
Here's the likely offending code:
{code}
LOG.info(
"Indeterminate response from trying to kill service. " +
"Verifying whether it is running using nc...");
rc = execCommand(session, "nc -z localhost 8020");
{code}
Naively, we could rely on netcat to the correct hostname (since the NN ought to
be listening on the hostname it's configured as), or just to use fuser. Fuser
catches ports independently of what IPs they're bound to:
{noformat}
[root@xxx ~]# fuser 1234/tcp
1234/tcp: 6766 6768
[root@xxx ~]# jobs
[1]- Running nc -l localhost 1234 &
[2]+ Running nc -l rhel56-18.ent.cloudera.com 1234 &
[root@xxx ~]# sudo lsof -P | grep -i LISTEN | grep -i 1234
nc 6766 root3u IPv42563626 TCP
localhost:1234 (LISTEN)
nc 6768 root3u IPv42563671 TCP
xxx:1234 (LISTEN)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3080) Move FSDataset interfaces and implementation to separated packages
Move FSDataset interfaces and implementation to separated packages -- Key: HDFS-3080 URL: https://issues.apache.org/jira/browse/HDFS-3080 Project: Hadoop HDFS Issue Type: Improvement Components: data-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE FSDataset interfaces and classes currently reside in the datanode package. It is not very easy to tell which are the FSDataset interfaces, which are the FSDataset implementation and which other datanode classes. It is easy to directly use FSDataset implementations by mistakes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2731) HA: Autopopulate standby name dirs if they're empty
[ https://issues.apache.org/jira/browse/HDFS-2731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13227870#comment-13227870 ] Hudson commented on HDFS-2731: -- Integrated in Hadoop-Mapreduce-0.23-Commit #682 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/682/]) HDFS-2731. Add command to bootstrap the Standby Node's name directories from the Active NameNode. Contributed by Todd Lipcon. (Revision 1299809) Result = ABORTED todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1299809 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/BootstrapStandby.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestBootstrapStandby.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/GenericTestUtils.java > HA: Autopopulate standby name dirs if they're empty > --- > > Key: HDFS-2731 > URL: https://issues.apache.org/jira/browse/HDFS-2731 > Project: Hadoop HDFS > Issue Type: New Feature > Components: ha >Affects Versions: 0.24.0 >Reporter: Aaron T. Myers >Assignee: Todd Lipcon > Fix For: 0.24.0, 0.23.3 > > Attachments: hdfs-2731.txt, hdfs-2731.txt, hdfs-2731.txt > > > To setup a SBN we currently format the primary then manually copy the name > dirs to the SBN. The SBN should do this automatically. Specifically, on NN > startup, if HA with a shared edits dir is configured and populated, if the > SBN has empty name dirs it should downloads the image and log from the > primary (as an optimization it could copy the logs from the shared dir). If > the other NN is still in standby then it should fail to start as it does > currently. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3078) 2NN https port setting is broken in Hadoop 1.0
[ https://issues.apache.org/jira/browse/HDFS-3078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13227868#comment-13227868 ] Todd Lipcon commented on HDFS-3078: --- +1 > 2NN https port setting is broken in Hadoop 1.0 > -- > > Key: HDFS-3078 > URL: https://issues.apache.org/jira/browse/HDFS-3078 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 1.0.0 >Reporter: Eli Collins >Assignee: Eli Collins > Attachments: hdfs-3078.txt > > > The code in SecondaryNameNode.java to set the https port is broken, if the > port is set it sets the bind addr to "addr:addr:port" which is bogus. Even if > it did work it uses port 0 instead of port 50490 (default listed in > ./src/packages/templates/conf/hdfs-site.xml). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2731) HA: Autopopulate standby name dirs if they're empty
[ https://issues.apache.org/jira/browse/HDFS-2731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13227855#comment-13227855 ] Hudson commented on HDFS-2731: -- Integrated in Hadoop-Mapreduce-trunk-Commit #1877 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1877/]) HDFS-2731. Add command to bootstrap the Standby Node's name directories from the Active NameNode. Contributed by Todd Lipcon. (Revision 1299807) Result = ABORTED todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1299807 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/BootstrapStandby.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestBootstrapStandby.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/GenericTestUtils.java > HA: Autopopulate standby name dirs if they're empty > --- > > Key: HDFS-2731 > URL: https://issues.apache.org/jira/browse/HDFS-2731 > Project: Hadoop HDFS > Issue Type: New Feature > Components: ha >Affects Versions: 0.24.0 >Reporter: Aaron T. Myers >Assignee: Todd Lipcon > Fix For: 0.24.0, 0.23.3 > > Attachments: hdfs-2731.txt, hdfs-2731.txt, hdfs-2731.txt > > > To setup a SBN we currently format the primary then manually copy the name > dirs to the SBN. The SBN should do this automatically. Specifically, on NN > startup, if HA with a shared edits dir is configured and populated, if the > SBN has empty name dirs it should downloads the image and log from the > primary (as an optimization it could copy the logs from the shared dir). If > the other NN is still in standby then it should fail to start as it does > currently. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2731) HA: Autopopulate standby name dirs if they're empty
[ https://issues.apache.org/jira/browse/HDFS-2731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13227832#comment-13227832 ] Hudson commented on HDFS-2731: -- Integrated in Hadoop-Common-0.23-Commit #674 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/674/]) HDFS-2731. Add command to bootstrap the Standby Node's name directories from the Active NameNode. Contributed by Todd Lipcon. (Revision 1299809) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1299809 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/BootstrapStandby.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestBootstrapStandby.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/GenericTestUtils.java > HA: Autopopulate standby name dirs if they're empty > --- > > Key: HDFS-2731 > URL: https://issues.apache.org/jira/browse/HDFS-2731 > Project: Hadoop HDFS > Issue Type: New Feature > Components: ha >Affects Versions: 0.24.0 >Reporter: Aaron T. Myers >Assignee: Todd Lipcon > Fix For: 0.24.0, 0.23.3 > > Attachments: hdfs-2731.txt, hdfs-2731.txt, hdfs-2731.txt > > > To setup a SBN we currently format the primary then manually copy the name > dirs to the SBN. The SBN should do this automatically. Specifically, on NN > startup, if HA with a shared edits dir is configured and populated, if the > SBN has empty name dirs it should downloads the image and log from the > primary (as an optimization it could copy the logs from the shared dir). If > the other NN is still in standby then it should fail to start as it does > currently. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3079) Document -bootstrapStandby flag for deploying HA
Document -bootstrapStandby flag for deploying HA Key: HDFS-3079 URL: https://issues.apache.org/jira/browse/HDFS-3079 Project: Hadoop HDFS Issue Type: Improvement Components: documentation, ha Affects Versions: 0.24.0, 0.23.3 Reporter: Todd Lipcon Priority: Minor HDFS-2731 added a new flag to make it easier to bootstrap a new standby node in an HA cluster. But, I forgot to update the docs. We should improve the docs to mention this new, easier method of setting up HA. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2731) HA: Autopopulate standby name dirs if they're empty
[ https://issues.apache.org/jira/browse/HDFS-2731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13227831#comment-13227831 ] Hudson commented on HDFS-2731: -- Integrated in Hadoop-Hdfs-0.23-Commit #665 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/665/]) HDFS-2731. Add command to bootstrap the Standby Node's name directories from the Active NameNode. Contributed by Todd Lipcon. (Revision 1299809) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1299809 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/BootstrapStandby.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestBootstrapStandby.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/GenericTestUtils.java > HA: Autopopulate standby name dirs if they're empty > --- > > Key: HDFS-2731 > URL: https://issues.apache.org/jira/browse/HDFS-2731 > Project: Hadoop HDFS > Issue Type: New Feature > Components: ha >Affects Versions: 0.24.0 >Reporter: Aaron T. Myers >Assignee: Todd Lipcon > Fix For: 0.24.0, 0.23.3 > > Attachments: hdfs-2731.txt, hdfs-2731.txt, hdfs-2731.txt > > > To setup a SBN we currently format the primary then manually copy the name > dirs to the SBN. The SBN should do this automatically. Specifically, on NN > startup, if HA with a shared edits dir is configured and populated, if the > SBN has empty name dirs it should downloads the image and log from the > primary (as an optimization it could copy the logs from the shared dir). If > the other NN is still in standby then it should fail to start as it does > currently. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3044) fsck move should be non-destructive by default
[
https://issues.apache.org/jira/browse/HDFS-3044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-3044:
---
Attachment: HDFS-3044.002.patch
* change options parsing code a bit
* rename some functions
> fsck move should be non-destructive by default
> --
>
> Key: HDFS-3044
> URL: https://issues.apache.org/jira/browse/HDFS-3044
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
>Reporter: Eli Collins
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-3044.002.patch
>
>
> The fsck move behavior in the code and originally articulated in HADOOP-101
> is:
> {quote}Current failure modes for DFS involve blocks that are completely
> missing. The only way to "fix" them would be to recover chains of blocks and
> put them into lost+found{quote}
> A directory is created with the file name, the blocks that are accessible are
> created as individual files in this directory, then the original file is
> removed.
> I suspect the rationale for this behavior was that you can't use files that
> are missing locations, and copying the block as files at least makes part of
> the files accessible. However this behavior can also result in permanent
> dataloss. Eg:
> - Some datanodes don't come up (eg due to a HW issues) and checkin on cluster
> startup, files with blocks where all replicas are on these set of datanodes
> are marked corrupt
> - Admin does fsck move, which deletes the "corrupt" files, saves whatever
> blocks were available
> - The HW issues with datanodes are resolved, they are started and join the
> cluster. The NN tells them to delete their blocks for the corrupt files since
> the file was deleted.
> I think we should:
> - Make fsck move non-destructive by default (eg just does a move into
> lost+found)
> - Make the destructive behavior optional (eg "--destructive" so admins think
> about what they're doing)
> - Provide better sanity checks and warnings, eg if you're running fsck and
> not all the slaves have checked in (if using dfs.hosts) then fsck should
> print a warning indicating this that an admin should have to override if they
> want to do something destructive
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3044) fsck move should be non-destructive by default
[
https://issues.apache.org/jira/browse/HDFS-3044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-3044:
---
Attachment: (was: HDFS-3044.001.patch)
> fsck move should be non-destructive by default
> --
>
> Key: HDFS-3044
> URL: https://issues.apache.org/jira/browse/HDFS-3044
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
>Reporter: Eli Collins
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-3044.002.patch
>
>
> The fsck move behavior in the code and originally articulated in HADOOP-101
> is:
> {quote}Current failure modes for DFS involve blocks that are completely
> missing. The only way to "fix" them would be to recover chains of blocks and
> put them into lost+found{quote}
> A directory is created with the file name, the blocks that are accessible are
> created as individual files in this directory, then the original file is
> removed.
> I suspect the rationale for this behavior was that you can't use files that
> are missing locations, and copying the block as files at least makes part of
> the files accessible. However this behavior can also result in permanent
> dataloss. Eg:
> - Some datanodes don't come up (eg due to a HW issues) and checkin on cluster
> startup, files with blocks where all replicas are on these set of datanodes
> are marked corrupt
> - Admin does fsck move, which deletes the "corrupt" files, saves whatever
> blocks were available
> - The HW issues with datanodes are resolved, they are started and join the
> cluster. The NN tells them to delete their blocks for the corrupt files since
> the file was deleted.
> I think we should:
> - Make fsck move non-destructive by default (eg just does a move into
> lost+found)
> - Make the destructive behavior optional (eg "--destructive" so admins think
> about what they're doing)
> - Provide better sanity checks and warnings, eg if you're running fsck and
> not all the slaves have checked in (if using dfs.hosts) then fsck should
> print a warning indicating this that an admin should have to override if they
> want to do something destructive
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2731) HA: Autopopulate standby name dirs if they're empty
[ https://issues.apache.org/jira/browse/HDFS-2731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13227823#comment-13227823 ] Hudson commented on HDFS-2731: -- Integrated in Hadoop-Common-trunk-Commit #1868 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1868/]) HDFS-2731. Add command to bootstrap the Standby Node's name directories from the Active NameNode. Contributed by Todd Lipcon. (Revision 1299807) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1299807 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/BootstrapStandby.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestBootstrapStandby.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/GenericTestUtils.java > HA: Autopopulate standby name dirs if they're empty > --- > > Key: HDFS-2731 > URL: https://issues.apache.org/jira/browse/HDFS-2731 > Project: Hadoop HDFS > Issue Type: New Feature > Components: ha >Affects Versions: 0.24.0 >Reporter: Aaron T. Myers >Assignee: Todd Lipcon > Fix For: 0.24.0, 0.23.3 > > Attachments: hdfs-2731.txt, hdfs-2731.txt, hdfs-2731.txt > > > To setup a SBN we currently format the primary then manually copy the name > dirs to the SBN. The SBN should do this automatically. Specifically, on NN > startup, if HA with a shared edits dir is configured and populated, if the > SBN has empty name dirs it should downloads the image and log from the > primary (as an optimization it could copy the logs from the shared dir). If > the other NN is still in standby then it should fail to start as it does > currently. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2731) HA: Autopopulate standby name dirs if they're empty
[ https://issues.apache.org/jira/browse/HDFS-2731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13227822#comment-13227822 ] Hudson commented on HDFS-2731: -- Integrated in Hadoop-Hdfs-trunk-Commit #1943 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1943/]) HDFS-2731. Add command to bootstrap the Standby Node's name directories from the Active NameNode. Contributed by Todd Lipcon. (Revision 1299807) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1299807 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/BootstrapStandby.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSck.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestBootstrapStandby.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/GenericTestUtils.java > HA: Autopopulate standby name dirs if they're empty > --- > > Key: HDFS-2731 > URL: https://issues.apache.org/jira/browse/HDFS-2731 > Project: Hadoop HDFS > Issue Type: New Feature > Components: ha >Affects Versions: 0.24.0 >Reporter: Aaron T. Myers >Assignee: Todd Lipcon > Fix For: 0.24.0, 0.23.3 > > Attachments: hdfs-2731.txt, hdfs-2731.txt, hdfs-2731.txt > > > To setup a SBN we currently format the primary then manually copy the name > dirs to the SBN. The SBN should do this automatically. Specifically, on NN > startup, if HA with a shared edits dir is configured and populated, if the > SBN has empty name dirs it should downloads the image and log from the > primary (as an optimization it could copy the logs from the shared dir). If > the other NN is still in standby then it should fail to start as it does > currently. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2731) HA: Autopopulate standby name dirs if they're empty
[ https://issues.apache.org/jira/browse/HDFS-2731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2731: -- Resolution: Fixed Fix Version/s: 0.23.3 0.24.0 Release Note: The HA NameNode may now be started with the "-bootstrapStandby" flag. This causes it to copy the namespace information and most recent checkpoint from its HA pair, and save it to local storage, allowing an HA setup to be bootstrapped without use of rsync or external tools. Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to 23.3 and trunk, thanks for the reviews Aaron. > HA: Autopopulate standby name dirs if they're empty > --- > > Key: HDFS-2731 > URL: https://issues.apache.org/jira/browse/HDFS-2731 > Project: Hadoop HDFS > Issue Type: New Feature > Components: ha >Affects Versions: 0.24.0 >Reporter: Aaron T. Myers >Assignee: Todd Lipcon > Fix For: 0.24.0, 0.23.3 > > Attachments: hdfs-2731.txt, hdfs-2731.txt, hdfs-2731.txt > > > To setup a SBN we currently format the primary then manually copy the name > dirs to the SBN. The SBN should do this automatically. Specifically, on NN > startup, if HA with a shared edits dir is configured and populated, if the > SBN has empty name dirs it should downloads the image and log from the > primary (as an optimization it could copy the logs from the shared dir). If > the other NN is still in standby then it should fail to start as it does > currently. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3078) 2NN https port setting is broken in Hadoop 1.0
[ https://issues.apache.org/jira/browse/HDFS-3078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-3078: -- Fix Version/s: (was: 1.1.0) > 2NN https port setting is broken in Hadoop 1.0 > -- > > Key: HDFS-3078 > URL: https://issues.apache.org/jira/browse/HDFS-3078 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 1.0.0 >Reporter: Eli Collins >Assignee: Eli Collins > Attachments: hdfs-3078.txt > > > The code in SecondaryNameNode.java to set the https port is broken, if the > port is set it sets the bind addr to "addr:addr:port" which is bogus. Even if > it did work it uses port 0 instead of port 50490 (default listed in > ./src/packages/templates/conf/hdfs-site.xml). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3078) 2NN https port setting is broken in Hadoop 1.0
[ https://issues.apache.org/jira/browse/HDFS-3078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-3078: -- Attachment: hdfs-3078.txt Patch attached. > 2NN https port setting is broken in Hadoop 1.0 > -- > > Key: HDFS-3078 > URL: https://issues.apache.org/jira/browse/HDFS-3078 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 1.0.0 >Reporter: Eli Collins >Assignee: Eli Collins > Fix For: 1.1.0 > > Attachments: hdfs-3078.txt > > > The code in SecondaryNameNode.java to set the https port is broken, if the > port is set it sets the bind addr to "addr:addr:port" which is bogus. Even if > it did work it uses port 0 instead of port 50490 (default listed in > ./src/packages/templates/conf/hdfs-site.xml). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3078) 2NN https port setting is broken in Hadoop 1.0
2NN https port setting is broken in Hadoop 1.0 -- Key: HDFS-3078 URL: https://issues.apache.org/jira/browse/HDFS-3078 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 1.0.0 Reporter: Eli Collins Assignee: Eli Collins Fix For: 1.1.0 The code in SecondaryNameNode.java to set the https port is broken, if the port is set it sets the bind addr to "addr:addr:port" which is bogus. Even if it did work it uses port 0 instead of port 50490 (default listed in ./src/packages/templates/conf/hdfs-site.xml). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3077) Quorum-based protocol for reading and writing edit logs
[ https://issues.apache.org/jira/browse/HDFS-3077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13227780#comment-13227780 ] Todd Lipcon commented on HDFS-3077: --- I plan to post a more thorough design doc in the next week or two, but my current rough thinking on the design is: - Implement a standalone daemon which hosts the JournalProtocol interface, or something similar to it. This daemon is responsible for accepting edits from the active master and writing them locally. We will need to extend the interface to also allow reading from the logs, and a recovery operation (see below) - The user would configure an odd number of these logger daemons (could be collocated with the NNs or even with DNs, given a dedicated disk) - Create a JournalManager implementation which uses a quorum protocol to commit edits to the logger daemons. As for the particular quorum protocol, I am leaning towards implementing ZAB -- its properties align very closely to the semantics we need. I see the following advantages over the BookKeeper approach: - Re-uses existing Hadoop subsystems like IPC, security, and the file-based edit logging code. This means that it will be easier to maintain for the Hadoop development community, and easier to deploy for Hadoop operations. - Doesn't introduce a new dependency on an external project. If there is a bug discovered in this code, we can fix it with a new Hadoop release without having to wait on a new release of ZooKeeper. Since ZK and HDFS may be managed by different ops teams, this also simplifies upgrade. - BookKeeper is a general system, whereas this is a specific system. Since BK tries to be quite general, it has extra complexity that we don't need. For example, it handles the interleaving of up to thousands of distinct edit logs into a single on-disk layout. These complexities are useful for a general "write-ahead log as a service" project, but not for our use case where even very large clusters have only a handful of distinct logs. - BookKeeper's commit protocol waits for all replicas to commit. This means that, should one of the bookies fail, one must wait for a rather lengthy timeout before continuing. Additionally, the latency of a commit is the maximum of the latency of the bookies, meaning that it's much less feasible to collocate bookies with other machines under load like DataNodes. A quorum commit protocol instead has a latency equal to the median of its replicas' latencies, allowing it to ride over transient slowness on the part of one of its replicas. The BK approach is still interesting for some deployments, and I'd advocate that we explore both options in parallel. If one approach as implemented turns out to have significant advantages, we can at that time choose to support only one, but in the meantime, I think it's worth developing both approaches. > Quorum-based protocol for reading and writing edit logs > --- > > Key: HDFS-3077 > URL: https://issues.apache.org/jira/browse/HDFS-3077 > Project: Hadoop HDFS > Issue Type: New Feature > Components: ha, name-node >Reporter: Todd Lipcon >Assignee: Todd Lipcon > > Currently, one of the weak points of the HA design is that it relies on > shared storage such as an NFS filer for the shared edit log. One alternative > that has been proposed is to depend on BookKeeper, a ZooKeeper subproject > which provides a highly available replicated edit log on commodity hardware. > This JIRA is to implement another alternative, based on a quorum commit > protocol, integrated more tightly in HDFS and with the requirements driven > only by HDFS's needs rather than more generic use cases. More details to > follow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3070) hdfs balancer doesn't balance blocks between datanodes
[ https://issues.apache.org/jira/browse/HDFS-3070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13227775#comment-13227775 ] Aaron T. Myers commented on HDFS-3070: -- If the lack of having servicerpc-address configured caused this, then I would still consider that a bug. The balancer should work even if only the normal NN RPC address is configured. > hdfs balancer doesn't balance blocks between datanodes > -- > > Key: HDFS-3070 > URL: https://issues.apache.org/jira/browse/HDFS-3070 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer >Affects Versions: 0.24.0 >Reporter: Stephen Chu > Attachments: unbalanced_nodes.png, unbalanced_nodes_inservice.png > > > I TeraGenerated data into DataNodes styx01 and styx02. Looking at the web UI, > both have over 3% disk usage. > Attached is a screenshot of the Live Nodes web UI. > On styx01, I run the _hdfs balancer_ command with threshold 1% and don't see > the blocks being balanced across all 4 datanodes (all blocks on styx01 and > styx02 stay put). > HA is currently enabled. > [schu@styx01 ~]$ hdfs haadmin -getServiceState nn1 > active > [schu@styx01 ~]$ hdfs balancer -threshold 1 > 12/03/08 10:10:32 INFO balancer.Balancer: Using a threshold of 1.0 > 12/03/08 10:10:32 INFO balancer.Balancer: namenodes = [] > 12/03/08 10:10:32 INFO balancer.Balancer: p = > Balancer.Parameters[BalancingPolicy.Node, threshold=1.0] > Time Stamp Iteration# Bytes Already Moved Bytes Left To Move > Bytes Being Moved > Balancing took 95.0 milliseconds > [schu@styx01 ~]$ > I believe with a threshold of 1% the balancer should trigger blocks being > moved across DataNodes, right? I am curious about the "namenode = []" from > the above output. > [schu@styx01 ~]$ hadoop version > Hadoop 0.24.0-SNAPSHOT > Subversion > git://styx01.sf.cloudera.com/home/schu/hadoop-common/hadoop-common-project/hadoop-common > -r f6a577d697bbcd04ffbc568167c97b79479ff319 > Compiled by schu on Thu Mar 8 15:32:50 PST 2012 > From source with checksum ec971a6e7316f7fbf471b617905856b8 > From > http://hadoop.apache.org/hdfs/docs/r0.21.0/api/org/apache/hadoop/hdfs/server/balancer/Balancer.html: > The threshold parameter is a fraction in the range of (0%, 100%) with a > default value of 10%. The threshold sets a target for whether the cluster is > balanced. A cluster is balanced if for each datanode, the utilization of the > node (ratio of used space at the node to total capacity of the node) differs > from the utilization of the (ratio of used space in the cluster to total > capacity of the cluster) by no more than the threshold value. The smaller the > threshold, the more balanced a cluster will become. It takes more time to run > the balancer for small threshold values. Also for a very small threshold the > cluster may not be able to reach the balanced state when applications write > and delete files concurrently. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3070) hdfs balancer doesn't balance blocks between datanodes
[
https://issues.apache.org/jira/browse/HDFS-3070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13227770#comment-13227770
]
Stephen Chu commented on HDFS-3070:
---
Eli, servicerpc-address was not configured in hdfs-site.xml.
dfs.namenode.rpc-address:
{noformat}
dfs.namenode.rpc-address.ha-nn-uri.nn1
styx01.sf.cloudera.com:12020
dfs.namenode.rpc-address.ha-nn-uri.nn2
styx02.sf.cloudera.com:12020
{noformat}
> hdfs balancer doesn't balance blocks between datanodes
> --
>
> Key: HDFS-3070
> URL: https://issues.apache.org/jira/browse/HDFS-3070
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer
>Affects Versions: 0.24.0
>Reporter: Stephen Chu
> Attachments: unbalanced_nodes.png, unbalanced_nodes_inservice.png
>
>
> I TeraGenerated data into DataNodes styx01 and styx02. Looking at the web UI,
> both have over 3% disk usage.
> Attached is a screenshot of the Live Nodes web UI.
> On styx01, I run the _hdfs balancer_ command with threshold 1% and don't see
> the blocks being balanced across all 4 datanodes (all blocks on styx01 and
> styx02 stay put).
> HA is currently enabled.
> [schu@styx01 ~]$ hdfs haadmin -getServiceState nn1
> active
> [schu@styx01 ~]$ hdfs balancer -threshold 1
> 12/03/08 10:10:32 INFO balancer.Balancer: Using a threshold of 1.0
> 12/03/08 10:10:32 INFO balancer.Balancer: namenodes = []
> 12/03/08 10:10:32 INFO balancer.Balancer: p =
> Balancer.Parameters[BalancingPolicy.Node, threshold=1.0]
> Time Stamp Iteration# Bytes Already Moved Bytes Left To Move
> Bytes Being Moved
> Balancing took 95.0 milliseconds
> [schu@styx01 ~]$
> I believe with a threshold of 1% the balancer should trigger blocks being
> moved across DataNodes, right? I am curious about the "namenode = []" from
> the above output.
> [schu@styx01 ~]$ hadoop version
> Hadoop 0.24.0-SNAPSHOT
> Subversion
> git://styx01.sf.cloudera.com/home/schu/hadoop-common/hadoop-common-project/hadoop-common
> -r f6a577d697bbcd04ffbc568167c97b79479ff319
> Compiled by schu on Thu Mar 8 15:32:50 PST 2012
> From source with checksum ec971a6e7316f7fbf471b617905856b8
> From
> http://hadoop.apache.org/hdfs/docs/r0.21.0/api/org/apache/hadoop/hdfs/server/balancer/Balancer.html:
> The threshold parameter is a fraction in the range of (0%, 100%) with a
> default value of 10%. The threshold sets a target for whether the cluster is
> balanced. A cluster is balanced if for each datanode, the utilization of the
> node (ratio of used space at the node to total capacity of the node) differs
> from the utilization of the (ratio of used space in the cluster to total
> capacity of the cluster) by no more than the threshold value. The smaller the
> threshold, the more balanced a cluster will become. It takes more time to run
> the balancer for small threshold values. Also for a very small threshold the
> cluster may not be able to reach the balanced state when applications write
> and delete files concurrently.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3077) Quorum-based protocol for reading and writing edit logs
Quorum-based protocol for reading and writing edit logs --- Key: HDFS-3077 URL: https://issues.apache.org/jira/browse/HDFS-3077 Project: Hadoop HDFS Issue Type: New Feature Components: ha, name-node Reporter: Todd Lipcon Assignee: Todd Lipcon Currently, one of the weak points of the HA design is that it relies on shared storage such as an NFS filer for the shared edit log. One alternative that has been proposed is to depend on BookKeeper, a ZooKeeper subproject which provides a highly available replicated edit log on commodity hardware. This JIRA is to implement another alternative, based on a quorum commit protocol, integrated more tightly in HDFS and with the requirements driven only by HDFS's needs rather than more generic use cases. More details to follow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2839) HA: Remove need for client configuration of nameservice ID
[ https://issues.apache.org/jira/browse/HDFS-2839?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13227729#comment-13227729 ] Sanjay Radia commented on HDFS-2839: Lets look at the bigger picture of the various connections to the NN, the requirements and possible solutions. *Requirements* # A DfsClient must be redirected when failover occurs # Http client must be redirected when failover occurs # Delegation tokens ** delegation tokens should work for the new active, even if issued by previous active ** renewal - renewer must be able to renew at the new active transparently # Cross cluster access must be supported # URI compatibility when transitioning nonHA-to-HA and HA-to-nonHA: A URI to NN that contains a name (not ip address) should continue to work. # Solution must work in ** Experimental seeting where DNS changes may not be permitted and also ** Production setting where the customer wants to use DNS to locate NNs. *Solutions* *Solution A: IP Failover* * Ip Failover satisfies all requirements as long as clients (DFSClient, Token renewer etc) reconnect on failure *Solution B:* * NN has a logical name that is authority in URI (hdfs://logicalName/path) * Logical name is mapped to Active and Standby’s IP (issues discussed below) * *Addressing the requirements in Solution B* *# DNS Client uses the multiple IPs *# HTTP - HTTP Proxy or DNS Round robin with standby redirecting *# Delegation token *** Delegation token secret is shared with standby *** Renewer – logical name is CanonicalService name and a proxy provider object maps to the IPs. *# Cross cluster – The logical to IP mapping must be available across clusters *# The non-HA NN’s DNS name is the logical name of NN. *# Supply 2 resolvers – config file resolver and DNS resolver. Later we can add ZK resolver. *What is the Logical Name?* Since the logical name is being exposed it should be the volumeName. Note this is distinct from NameServiceId since the NameServiceId was supposed to allow a name server to store multiple Volumes. * Proposal: Add VolumeName to config ** Currently a 1-1 mapping between Volume Name and NameServiceId since a NameService can host only one volume. ** Why do this now? the logical name is being exposed via the URI - we should select the right terminology now ** How are Volume names mapped to IPs? (See below) *Is the notion of Logical name used when using the IP failover Solution* Yes – the LogicalName is the DNS name that maps to a single IP which failsover *Where is the LogicalName -to-IPs Mapping Stored?* * DNS-Resolver - DNS-Name to IP or IPs (single IP in case of IP-Failover) * ConfigFile-resolver - the mapping in the config file - this config file will need to be be available in all clusters, for all clusters to allow cross cluster access. This configuration file will be in addition to the client side mount tables. May want to consider having a single conf file that resolves mount points as well as logical names? * Zookeeper-resolver (Volume management at some point should move to Zookeeper ) Scalability of zookeeper may need to be addressed. > HA: Remove need for client configuration of nameservice ID > -- > > Key: HDFS-2839 > URL: https://issues.apache.org/jira/browse/HDFS-2839 > Project: Hadoop HDFS > Issue Type: New Feature > Components: ha, hdfs client, name-node >Affects Versions: 0.24.0 >Reporter: Jitendra Nath Pandey >Assignee: Jitendra Nath Pandey > > The fully qualified path from an ha cluster, won't be usable from a different > cluster that doesn't know about that particular namespace id. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2303) jsvc needs to be recompilable
[ https://issues.apache.org/jira/browse/HDFS-2303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-2303: -- Attachment: HDFS-2303-5-modcommon-trunk.patch Patch attached again to kick jenkins. > jsvc needs to be recompilable > - > > Key: HDFS-2303 > URL: https://issues.apache.org/jira/browse/HDFS-2303 > Project: Hadoop HDFS > Issue Type: Bug > Components: build, scripts >Affects Versions: 0.23.0, 0.24.0 >Reporter: Roman Shaposhnik >Assignee: Roman Shaposhnik > Attachments: HDFS-2303-2.patch.txt, HDFS-2303-3-trunk.patch, > HDFS-2303-4-trunk.patch, HDFS-2303-5-modcommon-trunk.patch, > HDFS-2303-5-modcommon-trunk.patch, HDFS-2303-5-modcommon-trunk.patch, > HDFS-2303-5-trunk.patch, HDFS-2303.patch.txt > > > It would be nice to recompile jsvc as part of the native profile. This has a > number of benefits including an ability to re-generate all binary artifacts, > etc. Most of all, however, it will provide a way to generate jsvc on Linux > distributions that don't have matching libc -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-2303) jsvc needs to be recompilable
[ https://issues.apache.org/jira/browse/HDFS-2303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins reassigned HDFS-2303: - Assignee: Mingjie Lai (was: Roman Shaposhnik) > jsvc needs to be recompilable > - > > Key: HDFS-2303 > URL: https://issues.apache.org/jira/browse/HDFS-2303 > Project: Hadoop HDFS > Issue Type: Bug > Components: build, scripts >Affects Versions: 0.23.0, 0.24.0 >Reporter: Roman Shaposhnik >Assignee: Mingjie Lai > Attachments: HDFS-2303-2.patch.txt, HDFS-2303-3-trunk.patch, > HDFS-2303-4-trunk.patch, HDFS-2303-5-modcommon-trunk.patch, > HDFS-2303-5-modcommon-trunk.patch, HDFS-2303-5-modcommon-trunk.patch, > HDFS-2303-5-trunk.patch, HDFS-2303.patch.txt > > > It would be nice to recompile jsvc as part of the native profile. This has a > number of benefits including an ability to re-generate all binary artifacts, > etc. Most of all, however, it will provide a way to generate jsvc on Linux > distributions that don't have matching libc -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2303) jsvc needs to be recompilable
[ https://issues.apache.org/jira/browse/HDFS-2303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-2303: -- Status: Open (was: Patch Available) > jsvc needs to be recompilable > - > > Key: HDFS-2303 > URL: https://issues.apache.org/jira/browse/HDFS-2303 > Project: Hadoop HDFS > Issue Type: Bug > Components: build, scripts >Affects Versions: 0.23.0, 0.24.0 >Reporter: Roman Shaposhnik >Assignee: Roman Shaposhnik > Attachments: HDFS-2303-2.patch.txt, HDFS-2303-3-trunk.patch, > HDFS-2303-4-trunk.patch, HDFS-2303-5-modcommon-trunk.patch, > HDFS-2303-5-modcommon-trunk.patch, HDFS-2303-5-modcommon-trunk.patch, > HDFS-2303-5-trunk.patch, HDFS-2303.patch.txt > > > It would be nice to recompile jsvc as part of the native profile. This has a > number of benefits including an ability to re-generate all binary artifacts, > etc. Most of all, however, it will provide a way to generate jsvc on Linux > distributions that don't have matching libc -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1623) High Availability Framework for HDFS NN
[ https://issues.apache.org/jira/browse/HDFS-1623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13227535#comment-13227535 ] Hudson commented on HDFS-1623: -- Integrated in Hadoop-Mapreduce-trunk #1017 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1017/]) Moving HDFS-1623 and HADOOP-7454 to 0.23.3 section in CHANGES.txt files (Revision 1299417) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1299417 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > High Availability Framework for HDFS NN > --- > > Key: HDFS-1623 > URL: https://issues.apache.org/jira/browse/HDFS-1623 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Sanjay Radia > Fix For: 0.24.0, 0.23.3 > > Attachments: HA-tests.pdf, HDFS-1623.rel23.patch, > HDFS-1623.trunk.patch, HDFS-High-Availability.pdf, NameNode HA_v2.pdf, > NameNode HA_v2_1.pdf, Namenode HA Framework.pdf, dfsio-results.tsv, > ha-testplan.pdf, ha-testplan.tex > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-693) java.net.SocketTimeoutException: 480000 millis timeout while waiting for channel to be ready for write exceptions were cast when trying to read file via StreamFile.
[ https://issues.apache.org/jira/browse/HDFS-693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13227534#comment-13227534 ] Inder SIngh commented on HDFS-693: -- Folks, What's the recommendation here. Does increasing the timeout resolve this? Does it have any other side effects.We are hitting the same on medium size cluster ~60 nodes. > java.net.SocketTimeoutException: 48 millis timeout while waiting for > channel to be ready for write exceptions were cast when trying to read file > via StreamFile. > > > Key: HDFS-693 > URL: https://issues.apache.org/jira/browse/HDFS-693 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node >Affects Versions: 0.20.1 >Reporter: Yajun Dong > Attachments: HDFS-693.log > > > To exclude the case of network problem, I found the count of dataXceiver is > about 30. Also, I could see the output of netstate -a | grep 50075 has many > TIME_WAIT status when this happened. > partial log in attachment. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1623) High Availability Framework for HDFS NN
[ https://issues.apache.org/jira/browse/HDFS-1623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13227498#comment-13227498 ] Hudson commented on HDFS-1623: -- Integrated in Hadoop-Mapreduce-0.23-Build #223 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/223/]) HDFS-1623. Merging change r1296534 from trunk to 0.23 (Revision 1299412) Result = FAILURE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1299412 Files : * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.HDFS-1623.txt * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/pom.xml * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs/src/documentation/content/xdocs/service_level_auth.xml * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ActiveStandbyElector.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/BadFencingConfigurationException.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/FailoverController.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/FailoverFailedException.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/FenceMethod.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HAAdmin.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HAServiceProtocol.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HAServiceProtocolHelper.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HealthCheckFailedException.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/NodeFencer.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ServiceFailedException.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ShellCommandFencer.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/SshFenceByTcpPort.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/StreamPumper.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/protocolPB * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/protocolPB/HAServiceProtocolClientSideTranslatorPB.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/protocolPB/HAServiceProtocolPB.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/protocolPB/HAServiceProtocolServerSideTranslatorPB.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/DefaultFailoverProxyProvider.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/FailoverProxyProvider.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryInvocationHandler.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicies.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicy.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtocolTranslator.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RPC.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/h
[jira] [Commented] (HDFS-1624) Append: The condition is incorrect for checking whether the last block is full
[
https://issues.apache.org/jira/browse/HDFS-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13227499#comment-13227499
]
Hudson commented on HDFS-1624:
--
Integrated in Hadoop-Mapreduce-0.23-Build #223 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/223/])
HDFS-1624, HADOOP-7454 - Merge r1296540 from trunk to 0.23 to fix
CHANGES.txt (Revision 1299415)
Result = FAILURE
suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1299415
Files :
*
/hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.HDFS-1623.txt
*
/hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.HDFS-1623.txt
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> Append: The condition is incorrect for checking whether the last block is full
> --
>
> Key: HDFS-1624
> URL: https://issues.apache.org/jira/browse/HDFS-1624
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 0.21.0, 0.22.0, 0.23.0
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h1624_20110211.patch
>
>
> When the last block is full, the free space should be 0 but not equal to
> block size.
> {code}
> //In DFSOutputStream.DataStreamer.DataStreamer(..),
> if (freeInLastBlock == blockSize) {
> throw new IOException("The last block for file " +
> src + " is full.");
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1624) Append: The condition is incorrect for checking whether the last block is full
[
https://issues.apache.org/jira/browse/HDFS-1624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13227476#comment-13227476
]
Hudson commented on HDFS-1624:
--
Integrated in Hadoop-Hdfs-0.23-Build #195 (See
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/195/])
HDFS-1624, HADOOP-7454 - Merge r1296540 from trunk to 0.23 to fix
CHANGES.txt (Revision 1299415)
Result = SUCCESS
suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1299415
Files :
*
/hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.HDFS-1623.txt
*
/hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.HDFS-1623.txt
*
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> Append: The condition is incorrect for checking whether the last block is full
> --
>
> Key: HDFS-1624
> URL: https://issues.apache.org/jira/browse/HDFS-1624
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 0.21.0, 0.22.0, 0.23.0
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Attachments: h1624_20110211.patch
>
>
> When the last block is full, the free space should be 0 but not equal to
> block size.
> {code}
> //In DFSOutputStream.DataStreamer.DataStreamer(..),
> if (freeInLastBlock == blockSize) {
> throw new IOException("The last block for file " +
> src + " is full.");
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1623) High Availability Framework for HDFS NN
[ https://issues.apache.org/jira/browse/HDFS-1623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13227475#comment-13227475 ] Hudson commented on HDFS-1623: -- Integrated in Hadoop-Hdfs-0.23-Build #195 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/195/]) HDFS-1623. Merging change r1296534 from trunk to 0.23 (Revision 1299412) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1299412 Files : * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.HDFS-1623.txt * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/pom.xml * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs/src/documentation/content/xdocs/service_level_auth.xml * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeys.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ActiveStandbyElector.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/BadFencingConfigurationException.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/FailoverController.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/FailoverFailedException.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/FenceMethod.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HAAdmin.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HAServiceProtocol.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HAServiceProtocolHelper.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/HealthCheckFailedException.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/NodeFencer.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ServiceFailedException.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ShellCommandFencer.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/SshFenceByTcpPort.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/StreamPumper.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/protocolPB * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/protocolPB/HAServiceProtocolClientSideTranslatorPB.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/protocolPB/HAServiceProtocolPB.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/protocolPB/HAServiceProtocolServerSideTranslatorPB.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/DefaultFailoverProxyProvider.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/FailoverProxyProvider.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryInvocationHandler.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicies.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicy.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtocolTranslator.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RPC.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-comm
[jira] [Commented] (HDFS-1623) High Availability Framework for HDFS NN
[ https://issues.apache.org/jira/browse/HDFS-1623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13227469#comment-13227469 ] Hudson commented on HDFS-1623: -- Integrated in Hadoop-Hdfs-trunk #982 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/982/]) Moving HDFS-1623 and HADOOP-7454 to 0.23.3 section in CHANGES.txt files (Revision 1299417) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1299417 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > High Availability Framework for HDFS NN > --- > > Key: HDFS-1623 > URL: https://issues.apache.org/jira/browse/HDFS-1623 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Sanjay Radia > Fix For: 0.24.0, 0.23.3 > > Attachments: HA-tests.pdf, HDFS-1623.rel23.patch, > HDFS-1623.trunk.patch, HDFS-High-Availability.pdf, NameNode HA_v2.pdf, > NameNode HA_v2_1.pdf, Namenode HA Framework.pdf, dfsio-results.tsv, > ha-testplan.pdf, ha-testplan.tex > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
