[jira] [Commented] (HDFS-3240) Drop log level of "heartbeat: ..." in BPServiceActor to DEBUG
[
https://issues.apache.org/jira/browse/HDFS-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250465#comment-13250465
]
Hudson commented on HDFS-3240:
--
Integrated in Hadoop-Mapreduce-trunk-Commit #2049 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2049/])
HDFS-3240. Drop log level of "heartbeat: ..." in BPServiceActor to DEBUG.
Contributed by Todd Lipcon. (Revision 1311577)
Result = SUCCESS
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311577
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java
> Drop log level of "heartbeat: ..." in BPServiceActor to DEBUG
> -
>
> Key: HDFS-3240
> URL: https://issues.apache.org/jira/browse/HDFS-3240
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
>Affects Versions: 2.0.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Trivial
> Fix For: 2.0.0
>
> Attachments: hdfs-3240.txt
>
>
> The following log message is at INFO level, but should be at DEBUG level:
> {code}
> LOG.info("heartbeat: " + this);
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3240) Drop log level of "heartbeat: ..." in BPServiceActor to DEBUG
[
https://issues.apache.org/jira/browse/HDFS-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250462#comment-13250462
]
Hudson commented on HDFS-3240:
--
Integrated in Hadoop-Common-trunk-Commit #2036 (See
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2036/])
HDFS-3240. Drop log level of "heartbeat: ..." in BPServiceActor to DEBUG.
Contributed by Todd Lipcon. (Revision 1311577)
Result = SUCCESS
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311577
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java
> Drop log level of "heartbeat: ..." in BPServiceActor to DEBUG
> -
>
> Key: HDFS-3240
> URL: https://issues.apache.org/jira/browse/HDFS-3240
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
>Affects Versions: 2.0.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Trivial
> Fix For: 2.0.0
>
> Attachments: hdfs-3240.txt
>
>
> The following log message is at INFO level, but should be at DEBUG level:
> {code}
> LOG.info("heartbeat: " + this);
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3240) Drop log level of "heartbeat: ..." in BPServiceActor to DEBUG
[
https://issues.apache.org/jira/browse/HDFS-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250461#comment-13250461
]
Hudson commented on HDFS-3240:
--
Integrated in Hadoop-Hdfs-trunk-Commit #2110 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2110/])
HDFS-3240. Drop log level of "heartbeat: ..." in BPServiceActor to DEBUG.
Contributed by Todd Lipcon. (Revision 1311577)
Result = SUCCESS
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311577
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java
> Drop log level of "heartbeat: ..." in BPServiceActor to DEBUG
> -
>
> Key: HDFS-3240
> URL: https://issues.apache.org/jira/browse/HDFS-3240
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
>Affects Versions: 2.0.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Trivial
> Fix For: 2.0.0
>
> Attachments: hdfs-3240.txt
>
>
> The following log message is at INFO level, but should be at DEBUG level:
> {code}
> LOG.info("heartbeat: " + this);
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3222) DFSInputStream#openInfo should not silently get the length as 0 when locations length is zero for last partial block.
[
https://issues.apache.org/jira/browse/HDFS-3222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250460#comment-13250460
]
Uma Maheswara Rao G commented on HDFS-3222:
---
Before moving.
{quote}
The other thought is that, after a restart, a block that was previously being
written would be in the under construction state, but with no expectedTargets.
This differs from the case where a block has been allocated but not yet written
to replicas. We could use that to set a new flag in the LocatedBlock response
indicating that it's not a 0-length, but instead that it's corrupt.
{quote}
On restart case, block is there and there is no locations means there must be a
sync call before. So, some data must have been written to it. If there is no
sync means, that partial block itself will not be noted by NN.
In your above comment, what is the other case you are pointing?
So, we need not set any flag from namenode right? I can decide directly from
client it self.
> DFSInputStream#openInfo should not silently get the length as 0 when
> locations length is zero for last partial block.
> -
>
> Key: HDFS-3222
> URL: https://issues.apache.org/jira/browse/HDFS-3222
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
>Affects Versions: 1.0.3, 2.0.0, 3.0.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
> Attachments: HDFS-3222-Test.patch
>
>
> I have seen one situation with Hbase cluster.
> Scenario is as follows:
> 1)1.5 blocks has been written and synced.
> 2)Suddenly cluster has been restarted.
> Reader opened the file and trying to get the length., By this time partial
> block contained DNs are not reported to NN. So, locations for this partial
> block would be 0. In this case, DFSInputStream assumes that, 1 block size as
> final size.
> But reader also assuming that, 1 block size is the final length and setting
> his end marker. Finally reader ending up reading only partial data. Due to
> this, HMaster could not replay the complete edits.
> Actually this happend with 20 version. Looking at the code, same should
> present in trunk as well.
> {code}
> int replicaNotFoundCount = locatedblock.getLocations().length;
>
> for(DatanodeInfo datanode : locatedblock.getLocations()) {
> ..
> ..
> // Namenode told us about these locations, but none know about the replica
> // means that we hit the race between pipeline creation start and end.
> // we require all 3 because some other exception could have happened
> // on a DN that has it. we want to report that error
> if (replicaNotFoundCount == 0) {
> return 0;
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3240) Drop log level of "heartbeat: ..." in BPServiceActor to DEBUG
[
https://issues.apache.org/jira/browse/HDFS-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated HDFS-3240:
--
Resolution: Fixed
Fix Version/s: 2.0.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
> Drop log level of "heartbeat: ..." in BPServiceActor to DEBUG
> -
>
> Key: HDFS-3240
> URL: https://issues.apache.org/jira/browse/HDFS-3240
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
>Affects Versions: 2.0.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Trivial
> Fix For: 2.0.0
>
> Attachments: hdfs-3240.txt
>
>
> The following log message is at INFO level, but should be at DEBUG level:
> {code}
> LOG.info("heartbeat: " + this);
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2696) fuse-dfs build problems
[ https://issues.apache.org/jira/browse/HDFS-2696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250441#comment-13250441 ] Hadoop QA commented on HDFS-2696: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522061/HDFS-2696-maven.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2241//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2241//console This message is automatically generated. > fuse-dfs build problems > --- > > Key: HDFS-2696 > URL: https://issues.apache.org/jira/browse/HDFS-2696 > Project: Hadoop HDFS > Issue Type: Bug > Components: build, contrib/fuse-dfs >Affects Versions: 0.23.0 > Environment: linux >Reporter: Petru Dimulescu >Assignee: Bruno Mahé > Labels: bigtop > Attachments: HDFS-2696-complete.patch, HDFS-2696-maven.patch, > HDFS-2696-plus.patch, HDFS-2696-rewrite.patch, dfsfuse-build.patch.zip > > > fuse-dfs has some compilation problems on the 0.23 and 0.24(head) branches. > Some details on how to compile fuse-dfs on 0.23 is here : > http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-user/201112.mbox/%3c4ee23195.7030...@gmail.com%3E > and a wiki page follows. > The attached patch removes some problems (an inexistnant > ivy/library.properties, detecting jni.h location). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3240) Drop log level of "heartbeat: ..." in BPServiceActor to DEBUG
[
https://issues.apache.org/jira/browse/HDFS-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250434#comment-13250434
]
Hadoop QA commented on HDFS-3240:
-
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12522056/hdfs-3240.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/2240//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2240//console
This message is automatically generated.
> Drop log level of "heartbeat: ..." in BPServiceActor to DEBUG
> -
>
> Key: HDFS-3240
> URL: https://issues.apache.org/jira/browse/HDFS-3240
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
>Affects Versions: 2.0.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Trivial
> Attachments: hdfs-3240.txt
>
>
> The following log message is at INFO level, but should be at DEBUG level:
> {code}
> LOG.info("heartbeat: " + this);
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3239) Hadoop-1.0.2 is taking the wrong class path during setup
[ https://issues.apache.org/jira/browse/HDFS-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250432#comment-13250432 ] Uma Maheswara Rao G commented on HDFS-3239: --- @Hadoopuser, no problem :-). here is the mailing list address, hdfs-u...@hadoop.apache.org, common-u...@hadoop.apache.org. We are always happy to address your issues. We can file the confirmed bugs in JIRA. > Hadoop-1.0.2 is taking the wrong class path during setup > - > > Key: HDFS-3239 > URL: https://issues.apache.org/jira/browse/HDFS-3239 > Project: Hadoop HDFS > Issue Type: Bug > Environment: I am using 32 bit ubuntu 11.10 >Reporter: Hadoopuser > > /usr/libexec/../bin/hadoop: line 321: /usr/lib/jvm/java-6-sun/bin/java: No > such file or directory > /usr/libexec/../bin/hadoop: line 387: /usr/lib/jvm/java-6-sun/bin/java: No > such file or directory -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3235) MiniDFSClusterManager doesn't correctly support -format option
[ https://issues.apache.org/jira/browse/HDFS-3235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250429#comment-13250429 ] Hudson commented on HDFS-3235: -- Integrated in Hadoop-Mapreduce-trunk-Commit #2048 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2048/]) HDFS-3235. MiniDFSClusterManager doesn't correctly support -format option. Contributed by Henry Robinson. (Revision 1311556) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311556 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/MiniDFSClusterManager.java > MiniDFSClusterManager doesn't correctly support -format option > -- > > Key: HDFS-3235 > URL: https://issues.apache.org/jira/browse/HDFS-3235 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Henry Robinson >Assignee: Henry Robinson >Priority: Minor > Fix For: 3.0.0 > > Attachments: HDFS-3235.0.patch > > > MiniDFSClusterManager.java correctly honours -format for setting > StartupOption.FORMAT, but does not set .format(true) on the > MiniDFSClusterBuilder. This means the datanodes' data dirs will be formatted > every time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3235) MiniDFSClusterManager doesn't correctly support -format option
[ https://issues.apache.org/jira/browse/HDFS-3235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250427#comment-13250427 ] Hudson commented on HDFS-3235: -- Integrated in Hadoop-Common-trunk-Commit #2035 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2035/]) HDFS-3235. MiniDFSClusterManager doesn't correctly support -format option. Contributed by Henry Robinson. (Revision 1311556) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311556 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/MiniDFSClusterManager.java > MiniDFSClusterManager doesn't correctly support -format option > -- > > Key: HDFS-3235 > URL: https://issues.apache.org/jira/browse/HDFS-3235 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Henry Robinson >Assignee: Henry Robinson >Priority: Minor > Fix For: 3.0.0 > > Attachments: HDFS-3235.0.patch > > > MiniDFSClusterManager.java correctly honours -format for setting > StartupOption.FORMAT, but does not set .format(true) on the > MiniDFSClusterBuilder. This means the datanodes' data dirs will be formatted > every time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3238) ServerCommand and friends don't need to be writables
[ https://issues.apache.org/jira/browse/HDFS-3238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250424#comment-13250424 ] Hadoop QA commented on HDFS-3238: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522055/hdfs-3238.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2239//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2239//console This message is automatically generated. > ServerCommand and friends don't need to be writables > > > Key: HDFS-3238 > URL: https://issues.apache.org/jira/browse/HDFS-3238 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Eli Collins >Assignee: Eli Collins > Attachments: hdfs-3238.txt > > > We can remove writable infrastructure from the ServerCommand classes as > they're not uses across clients and we're PB within the server side. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3123) BNN is getting Nullpointer execption and shuttingdown When NameNode got formatted
[
https://issues.apache.org/jira/browse/HDFS-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250423#comment-13250423
]
Uma Maheswara Rao G commented on HDFS-3123:
---
Brahma, HADOOP-8202 fixed close problem in latest builds in trunk and branch-2.
> BNN is getting Nullpointer execption and shuttingdown When NameNode got
> formatted
> --
>
> Key: HDFS-3123
> URL: https://issues.apache.org/jira/browse/HDFS-3123
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 0.24.0, 0.23.4
>Reporter: Brahma Reddy Battula
>Assignee: Uma Maheswara Rao G
>
> Scenario 1
> ==
> Start NN and BNN
> stop NN and BNN
> Format NN and start only BNN
> Then BNN as getting Nullpointer and getting shutdown
> {noformat}
> 12/03/20 21:26:05 ERROR ipc.RPC: Tried to call RPC.stopProxy on an object
> that is not a proxy.
> java.lang.IllegalArgumentException: not a proxy instance
> at java.lang.reflect.Proxy.getInvocationHandler(Proxy.java:637)
> at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:591)
> at
> org.apache.hadoop.hdfs.server.namenode.BackupNode.stop(BackupNode.java:194)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:547)
> at
> org.apache.hadoop.hdfs.server.namenode.BackupNode.(BackupNode.java:86)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:847)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:908)
> 12/03/20 21:26:05 ERROR ipc.RPC: Could not get invocation handler null for
> proxy class class
> org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB, or invocation
> handler is not closeable.
> 12/03/20 21:26:05 ERROR namenode.NameNode: Exception in namenode join
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getFSImage(NameNode.java:609)
> at
> org.apache.hadoop.hdfs.server.namenode.BackupNode.stop(BackupNode.java:205)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:547)
> at
> org.apache.hadoop.hdfs.server.namenode.BackupNode.(BackupNode.java:86)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:847)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:908)
> 12/03/20 21:26:05 INFO namenode.NameNode: SHUTDOWN_MSG:
> /
> SHUTDOWN_MSG: Shutting down NameNode at HOST-10-18-40-233/10.18.40.233
> /
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3236) NameNode does not initialize generic conf keys when started with -initializeSharedEditsDir
[ https://issues.apache.org/jira/browse/HDFS-3236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250422#comment-13250422 ] Hudson commented on HDFS-3236: -- Integrated in Hadoop-Mapreduce-trunk-Commit #2047 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2047/]) HDFS-3236. NameNode does not initialize generic conf keys when started with -initializeSharedEditsDir. Contributed by Aaron T. Myers. (Revision 1311554) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311554 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestInitializeSharedEdits.java > NameNode does not initialize generic conf keys when started with > -initializeSharedEditsDir > -- > > Key: HDFS-3236 > URL: https://issues.apache.org/jira/browse/HDFS-3236 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha, name-node >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers >Priority: Minor > Fix For: 2.0.0 > > Attachments: HDFS-3236.patch > > > This means that configurations that scope the location of the > name/edits/shared edits dirs by nameserice or namenode won't work with `hdfs > namenode -initializeSharedEdits'. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-2768) BackupNode stop can not close proxy connections because it is not a proxy instance.
[ https://issues.apache.org/jira/browse/HDFS-2768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G resolved HDFS-2768. --- Resolution: Fixed Seems, this was already fixed in trunk as part of HADOOP-8202 > BackupNode stop can not close proxy connections because it is not a proxy > instance. > --- > > Key: HDFS-2768 > URL: https://issues.apache.org/jira/browse/HDFS-2768 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.24.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 2.0.0 > > Attachments: HDFS-2768.patch, HDFS-2768.patch > > > Observe this from BackupNode tests: > java.lang.IllegalArgumentException: not a proxy instance > at java.lang.reflect.Proxy.getInvocationHandler(Unknown Source) > at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:557) > at > org.apache.hadoop.hdfs.server.namenode.BackupNode.stop(BackupNode.java:194) > at > org.apache.hadoop.hdfs.server.namenode.TestBackupNode.testCheckpoint(TestBackupNode.java:355) > at > org.apache.hadoop.hdfs.server.namenode.TestBackupNode.testBackupNode(TestBackupNode.java:241) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) > at java.lang.reflect.Method.invoke(Unknown Source) > at junit.framework.TestCase.runTest(TestCase.java:168) > at junit.framework.TestCase.runBare(TestCase.java:134) > at junit.framework.TestResult$1.protect(TestResult.java:110) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3235) MiniDFSClusterManager doesn't correctly support -format option
[ https://issues.apache.org/jira/browse/HDFS-3235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250417#comment-13250417 ] Hudson commented on HDFS-3235: -- Integrated in Hadoop-Hdfs-trunk-Commit #2109 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2109/]) HDFS-3235. MiniDFSClusterManager doesn't correctly support -format option. Contributed by Henry Robinson. (Revision 1311556) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311556 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/test/MiniDFSClusterManager.java > MiniDFSClusterManager doesn't correctly support -format option > -- > > Key: HDFS-3235 > URL: https://issues.apache.org/jira/browse/HDFS-3235 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Henry Robinson >Assignee: Henry Robinson >Priority: Minor > Fix For: 3.0.0 > > Attachments: HDFS-3235.0.patch > > > MiniDFSClusterManager.java correctly honours -format for setting > StartupOption.FORMAT, but does not set .format(true) on the > MiniDFSClusterBuilder. This means the datanodes' data dirs will be formatted > every time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3236) NameNode does not initialize generic conf keys when started with -initializeSharedEditsDir
[ https://issues.apache.org/jira/browse/HDFS-3236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250416#comment-13250416 ] Hudson commented on HDFS-3236: -- Integrated in Hadoop-Hdfs-trunk-Commit #2109 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2109/]) HDFS-3236. NameNode does not initialize generic conf keys when started with -initializeSharedEditsDir. Contributed by Aaron T. Myers. (Revision 1311554) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311554 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestInitializeSharedEdits.java > NameNode does not initialize generic conf keys when started with > -initializeSharedEditsDir > -- > > Key: HDFS-3236 > URL: https://issues.apache.org/jira/browse/HDFS-3236 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha, name-node >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers >Priority: Minor > Fix For: 2.0.0 > > Attachments: HDFS-3236.patch > > > This means that configurations that scope the location of the > name/edits/shared edits dirs by nameserice or namenode won't work with `hdfs > namenode -initializeSharedEdits'. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3236) NameNode does not initialize generic conf keys when started with -initializeSharedEditsDir
[ https://issues.apache.org/jira/browse/HDFS-3236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250414#comment-13250414 ] Hudson commented on HDFS-3236: -- Integrated in Hadoop-Common-trunk-Commit #2034 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2034/]) HDFS-3236. NameNode does not initialize generic conf keys when started with -initializeSharedEditsDir. Contributed by Aaron T. Myers. (Revision 1311554) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311554 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestInitializeSharedEdits.java > NameNode does not initialize generic conf keys when started with > -initializeSharedEditsDir > -- > > Key: HDFS-3236 > URL: https://issues.apache.org/jira/browse/HDFS-3236 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha, name-node >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers >Priority: Minor > Fix For: 2.0.0 > > Attachments: HDFS-3236.patch > > > This means that configurations that scope the location of the > name/edits/shared edits dirs by nameserice or namenode won't work with `hdfs > namenode -initializeSharedEdits'. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3235) MiniDFSClusterManager doesn't correctly support -format option
[ https://issues.apache.org/jira/browse/HDFS-3235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-3235: - Resolution: Fixed Fix Version/s: 3.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I've just committed this fix to trunk. Thanks a lot for the contribution, Hank. > MiniDFSClusterManager doesn't correctly support -format option > -- > > Key: HDFS-3235 > URL: https://issues.apache.org/jira/browse/HDFS-3235 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Henry Robinson >Assignee: Henry Robinson >Priority: Minor > Fix For: 3.0.0 > > Attachments: HDFS-3235.0.patch > > > MiniDFSClusterManager.java correctly honours -format for setting > StartupOption.FORMAT, but does not set .format(true) on the > MiniDFSClusterBuilder. This means the datanodes' data dirs will be formatted > every time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3235) MiniDFSClusterManager doesn't correctly support -format option
[ https://issues.apache.org/jira/browse/HDFS-3235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250408#comment-13250408 ] Aaron T. Myers commented on HDFS-3235: -- +1, the patch looks good to me. I'm going to commit this momentarily. > MiniDFSClusterManager doesn't correctly support -format option > -- > > Key: HDFS-3235 > URL: https://issues.apache.org/jira/browse/HDFS-3235 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Henry Robinson >Assignee: Henry Robinson >Priority: Minor > Attachments: HDFS-3235.0.patch > > > MiniDFSClusterManager.java correctly honours -format for setting > StartupOption.FORMAT, but does not set .format(true) on the > MiniDFSClusterBuilder. This means the datanodes' data dirs will be formatted > every time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3236) NameNode does not initialize generic conf keys when started with -initializeSharedEditsDir
[ https://issues.apache.org/jira/browse/HDFS-3236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-3236: - Resolution: Fixed Fix Version/s: 2.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks a lot for the review, Todd. I've just committed this to trunk and branch-2. > NameNode does not initialize generic conf keys when started with > -initializeSharedEditsDir > -- > > Key: HDFS-3236 > URL: https://issues.apache.org/jira/browse/HDFS-3236 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha, name-node >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers >Priority: Minor > Fix For: 2.0.0 > > Attachments: HDFS-3236.patch > > > This means that configurations that scope the location of the > name/edits/shared edits dirs by nameserice or namenode won't work with `hdfs > namenode -initializeSharedEdits'. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3239) Hadoop-1.0.2 is taking the wrong class path during setup
[ https://issues.apache.org/jira/browse/HDFS-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250404#comment-13250404 ] Hadoopuser commented on HDFS-3239: -- @amit my java home is set to /usr/lib/jvm/jdk1.6.0_27/bin/java @uma Mashwara Rao Sorry i will be careful the next time > Hadoop-1.0.2 is taking the wrong class path during setup > - > > Key: HDFS-3239 > URL: https://issues.apache.org/jira/browse/HDFS-3239 > Project: Hadoop HDFS > Issue Type: Bug > Environment: I am using 32 bit ubuntu 11.10 >Reporter: Hadoopuser > > /usr/libexec/../bin/hadoop: line 321: /usr/lib/jvm/java-6-sun/bin/java: No > such file or directory > /usr/libexec/../bin/hadoop: line 387: /usr/lib/jvm/java-6-sun/bin/java: No > such file or directory -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2696) fuse-dfs build problems
[ https://issues.apache.org/jira/browse/HDFS-2696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruno Mahé updated HDFS-2696: - Status: Patch Available (was: Open) > fuse-dfs build problems > --- > > Key: HDFS-2696 > URL: https://issues.apache.org/jira/browse/HDFS-2696 > Project: Hadoop HDFS > Issue Type: Bug > Components: build, contrib/fuse-dfs >Affects Versions: 0.23.0 > Environment: linux >Reporter: Petru Dimulescu >Assignee: Bruno Mahé > Labels: bigtop > Attachments: HDFS-2696-complete.patch, HDFS-2696-maven.patch, > HDFS-2696-plus.patch, HDFS-2696-rewrite.patch, dfsfuse-build.patch.zip > > > fuse-dfs has some compilation problems on the 0.23 and 0.24(head) branches. > Some details on how to compile fuse-dfs on 0.23 is here : > http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-user/201112.mbox/%3c4ee23195.7030...@gmail.com%3E > and a wiki page follows. > The attached patch removes some problems (an inexistnant > ivy/library.properties, detecting jni.h location). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-3239) Hadoop-1.0.2 is taking the wrong class path during setup
[ https://issues.apache.org/jira/browse/HDFS-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G resolved HDFS-3239. --- Resolution: Invalid This is like a usage problem. This kind of questions can be in mailing lists. > Hadoop-1.0.2 is taking the wrong class path during setup > - > > Key: HDFS-3239 > URL: https://issues.apache.org/jira/browse/HDFS-3239 > Project: Hadoop HDFS > Issue Type: Bug > Environment: I am using 32 bit ubuntu 11.10 >Reporter: Hadoopuser > > /usr/libexec/../bin/hadoop: line 321: /usr/lib/jvm/java-6-sun/bin/java: No > such file or directory > /usr/libexec/../bin/hadoop: line 387: /usr/lib/jvm/java-6-sun/bin/java: No > such file or directory -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2696) fuse-dfs build problems
[
https://issues.apache.org/jira/browse/HDFS-2696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bruno Mahé updated HDFS-2696:
-
Attachment: HDFS-2696-maven.patch
Here is another version of the patch a little bit more maven friendly.
The previous version of that patch could trigger some issues depending on the
execution order of the different maven modules.
Tested with:
{noformat}mvn clean install -DskipTests -Pnative -DskipTest -Pfuse{noformat}
The -DskipTest is there to go around a typo issue in some other unrelated
native build in some yarn module
> fuse-dfs build problems
> ---
>
> Key: HDFS-2696
> URL: https://issues.apache.org/jira/browse/HDFS-2696
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build, contrib/fuse-dfs
>Affects Versions: 0.23.0
> Environment: linux
>Reporter: Petru Dimulescu
>Assignee: Bruno Mahé
> Labels: bigtop
> Attachments: HDFS-2696-complete.patch, HDFS-2696-maven.patch,
> HDFS-2696-plus.patch, HDFS-2696-rewrite.patch, dfsfuse-build.patch.zip
>
>
> fuse-dfs has some compilation problems on the 0.23 and 0.24(head) branches.
> Some details on how to compile fuse-dfs on 0.23 is here :
> http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-user/201112.mbox/%3c4ee23195.7030...@gmail.com%3E
> and a wiki page follows.
> The attached patch removes some problems (an inexistnant
> ivy/library.properties, detecting jni.h location).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-2768) BackupNode stop can not close proxy connections because it is not a proxy instance.
[ https://issues.apache.org/jira/browse/HDFS-2768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G reopened HDFS-2768: --- Seems this changes has been reverted with HA merge. > BackupNode stop can not close proxy connections because it is not a proxy > instance. > --- > > Key: HDFS-2768 > URL: https://issues.apache.org/jira/browse/HDFS-2768 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.24.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 2.0.0 > > Attachments: HDFS-2768.patch, HDFS-2768.patch > > > Observe this from BackupNode tests: > java.lang.IllegalArgumentException: not a proxy instance > at java.lang.reflect.Proxy.getInvocationHandler(Unknown Source) > at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:557) > at > org.apache.hadoop.hdfs.server.namenode.BackupNode.stop(BackupNode.java:194) > at > org.apache.hadoop.hdfs.server.namenode.TestBackupNode.testCheckpoint(TestBackupNode.java:355) > at > org.apache.hadoop.hdfs.server.namenode.TestBackupNode.testBackupNode(TestBackupNode.java:241) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) > at java.lang.reflect.Method.invoke(Unknown Source) > at junit.framework.TestCase.runTest(TestCase.java:168) > at junit.framework.TestCase.runBare(TestCase.java:134) > at junit.framework.TestResult$1.protect(TestResult.java:110) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3235) MiniDFSClusterManager doesn't correctly support -format option
[ https://issues.apache.org/jira/browse/HDFS-3235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250398#comment-13250398 ] Hadoop QA commented on HDFS-3235: - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522039/HDFS-3235.0.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2238//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2238//console This message is automatically generated. > MiniDFSClusterManager doesn't correctly support -format option > -- > > Key: HDFS-3235 > URL: https://issues.apache.org/jira/browse/HDFS-3235 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Henry Robinson >Assignee: Henry Robinson >Priority: Minor > Attachments: HDFS-3235.0.patch > > > MiniDFSClusterManager.java correctly honours -format for setting > StartupOption.FORMAT, but does not set .format(true) on the > MiniDFSClusterBuilder. This means the datanodes' data dirs will be formatted > every time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3239) Hadoop-1.0.2 is taking the wrong class path during setup
[ https://issues.apache.org/jira/browse/HDFS-3239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250394#comment-13250394 ] amith commented on HDFS-3239: - Hi Hadoopuser What is the java home configured in your local box? > Hadoop-1.0.2 is taking the wrong class path during setup > - > > Key: HDFS-3239 > URL: https://issues.apache.org/jira/browse/HDFS-3239 > Project: Hadoop HDFS > Issue Type: Bug > Environment: I am using 32 bit ubuntu 11.10 >Reporter: Hadoopuser > > /usr/libexec/../bin/hadoop: line 321: /usr/lib/jvm/java-6-sun/bin/java: No > such file or directory > /usr/libexec/../bin/hadoop: line 387: /usr/lib/jvm/java-6-sun/bin/java: No > such file or directory -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2799) Trim fs.checkpoint.dir values
[
https://issues.apache.org/jira/browse/HDFS-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250391#comment-13250391
]
amith commented on HDFS-2799:
-
@Eli can u review the patch once
> Trim fs.checkpoint.dir values
> -
>
> Key: HDFS-2799
> URL: https://issues.apache.org/jira/browse/HDFS-2799
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.23.0
>Reporter: Eli Collins
> Labels: noob
> Attachments: HDFS-2799.patch
>
>
> fs.checkpoint.dir values need to be trimmed like dfs.name.dir and
> dfs.data.dir values so eg the following works. This currently results in the
> directory "HADOOP_HOME/?/home/eli/hadoop/dirs3/dfs/chkpoint1" being created.
> {noformat}
>
> fs.checkpoint.dir
>
> /home/eli/hadoop/dirs3/dfs/chkpoint1,
> /home/eli/hadoop/dirs3/dfs/chkpoint2
>
>
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2799) Trim fs.checkpoint.dir values
[
https://issues.apache.org/jira/browse/HDFS-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250390#comment-13250390
]
amith commented on HDFS-2799:
-
Hi Harsh,
I have given a patch as u expected.
I have moved the getCheckPointDirs to DFSUtil so this can be used by others in
case required :)
With this movement it is much easy to write a new test also :)
> Trim fs.checkpoint.dir values
> -
>
> Key: HDFS-2799
> URL: https://issues.apache.org/jira/browse/HDFS-2799
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.23.0
>Reporter: Eli Collins
> Labels: noob
> Attachments: HDFS-2799.patch
>
>
> fs.checkpoint.dir values need to be trimmed like dfs.name.dir and
> dfs.data.dir values so eg the following works. This currently results in the
> directory "HADOOP_HOME/?/home/eli/hadoop/dirs3/dfs/chkpoint1" being created.
> {noformat}
>
> fs.checkpoint.dir
>
> /home/eli/hadoop/dirs3/dfs/chkpoint1,
> /home/eli/hadoop/dirs3/dfs/chkpoint2
>
>
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3241) BackupNode can't start if its storage directory is not manually hacked
BackupNode can't start if its storage directory is not manually hacked
--
Key: HDFS-3241
URL: https://issues.apache.org/jira/browse/HDFS-3241
Project: Hadoop HDFS
Issue Type: Improvement
Components: name-node
Affects Versions: 3.0.0
Reporter: Brandon Li
Priority: Minor
BackupNode can't start if its storage dir is not formatted (because it extends
NameNode).
After I use "/bin/hadoop namenode -format" to format its storage directory, it
still can't start because the inconsistent namespace with primary namenode.
{noformat}
2012-04-09 18:33:16,464 ERROR namenode.NameNode (NameNode.java:main(958)) -
Exception in namenode join
java.io.IOException: Inconsistent namespace information:
NamespaceInfo has:
LV=-40;NS=1165721067;cTime=0;CID=CID-b8ced26a-4675-476d-b5fb-c9ad337be34a;BPID=BP-1658271424-10.10.10.191-1333658489893.
Storage has:
LV=-40;NS=403924869;cTime=0;CID=CID-57ce694a-172c-4da1-9cd6-d498615e4f1e;BPID=BP-4558626-10.10.10.48-1334021589986.
at
org.apache.hadoop.hdfs.server.protocol.NamespaceInfo.validateStorage(NamespaceInfo.java:109)
at
org.apache.hadoop.hdfs.server.namenode.BackupNode.registerWith(BackupNode.java:332)
at
org.apache.hadoop.hdfs.server.namenode.BackupNode.initialize(BackupNode.java:161)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:546)
at
org.apache.hadoop.hdfs.server.namenode.BackupNode.(BackupNode.java:86)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:893)
{noformat}
The backup node started after I copied the VERSION file from the primary
namenode storage directory to that of the backup node.
This manual hack can be avoided if the backup node can format its local
directory by requesting VERSION info from the primary namenode.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3236) NameNode does not initialize generic conf keys when started with -initializeSharedEditsDir
[ https://issues.apache.org/jira/browse/HDFS-3236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250388#comment-13250388 ] Hadoop QA commented on HDFS-3236: - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522042/HDFS-3236.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2237//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2237//console This message is automatically generated. > NameNode does not initialize generic conf keys when started with > -initializeSharedEditsDir > -- > > Key: HDFS-3236 > URL: https://issues.apache.org/jira/browse/HDFS-3236 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha, name-node >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers >Priority: Minor > Attachments: HDFS-3236.patch > > > This means that configurations that scope the location of the > name/edits/shared edits dirs by nameserice or namenode won't work with `hdfs > namenode -initializeSharedEdits'. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3234) Accidentally left log message in GetConf after HDFS-3226
[ https://issues.apache.org/jira/browse/HDFS-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250382#comment-13250382 ] Hudson commented on HDFS-3234: -- Integrated in Hadoop-Hdfs-trunk-Commit #2108 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2108/]) HDFS-3234. Accidentally left log message in GetConf after HDFS-3226. Contributed by Todd Lipcon. (Revision 1311541) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311541 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/GetConf.java > Accidentally left log message in GetConf after HDFS-3226 > > > Key: HDFS-3234 > URL: https://issues.apache.org/jira/browse/HDFS-3234 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 2.0.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Trivial > Fix For: 2.0.0 > > Attachments: hdfs-3234.txt > > > I accidentally left a debug printout in. It doesn't cause a functionality > regression, but it does cause noisy output on the command line: > $ ./bin/hdfs getconf -confKey fs.defaultFS > key: fs.defaultFS > hdfs://nameserviceId1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3226) Allow GetConf tool to print arbitrary keys
[ https://issues.apache.org/jira/browse/HDFS-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250383#comment-13250383 ] Hudson commented on HDFS-3226: -- Integrated in Hadoop-Hdfs-trunk-Commit #2108 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2108/]) HDFS-3234. Accidentally left log message in GetConf after HDFS-3226. Contributed by Todd Lipcon. (Revision 1311541) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311541 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/GetConf.java > Allow GetConf tool to print arbitrary keys > -- > > Key: HDFS-3226 > URL: https://issues.apache.org/jira/browse/HDFS-3226 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Affects Versions: 2.0.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 2.0.0 > > Attachments: hdfs-3226.txt > > > Currently the "hdfs getconf" tool can only print out certain keys, like the > list of NNs, etc. It would be handy to allow it to fetch an arbitrary > configuration. For example, users may wish to write shell scripts that > interact with their hadoop cluster, and it is useful to be able to fetch > configs like the name of the superuser, or the state of whether HA is enabled. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3240) Drop log level of "heartbeat: ..." in BPServiceActor to DEBUG
[
https://issues.apache.org/jira/browse/HDFS-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250379#comment-13250379
]
Aaron T. Myers commented on HDFS-3240:
--
+1 pending Jenkins.
> Drop log level of "heartbeat: ..." in BPServiceActor to DEBUG
> -
>
> Key: HDFS-3240
> URL: https://issues.apache.org/jira/browse/HDFS-3240
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
>Affects Versions: 2.0.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Trivial
> Attachments: hdfs-3240.txt
>
>
> The following log message is at INFO level, but should be at DEBUG level:
> {code}
> LOG.info("heartbeat: " + this);
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3234) Accidentally left log message in GetConf after HDFS-3226
[ https://issues.apache.org/jira/browse/HDFS-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250377#comment-13250377 ] Hudson commented on HDFS-3234: -- Integrated in Hadoop-Common-trunk-Commit #2033 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2033/]) HDFS-3234. Accidentally left log message in GetConf after HDFS-3226. Contributed by Todd Lipcon. (Revision 1311541) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311541 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/GetConf.java > Accidentally left log message in GetConf after HDFS-3226 > > > Key: HDFS-3234 > URL: https://issues.apache.org/jira/browse/HDFS-3234 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 2.0.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Trivial > Fix For: 2.0.0 > > Attachments: hdfs-3234.txt > > > I accidentally left a debug printout in. It doesn't cause a functionality > regression, but it does cause noisy output on the command line: > $ ./bin/hdfs getconf -confKey fs.defaultFS > key: fs.defaultFS > hdfs://nameserviceId1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3226) Allow GetConf tool to print arbitrary keys
[ https://issues.apache.org/jira/browse/HDFS-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250378#comment-13250378 ] Hudson commented on HDFS-3226: -- Integrated in Hadoop-Common-trunk-Commit #2033 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2033/]) HDFS-3234. Accidentally left log message in GetConf after HDFS-3226. Contributed by Todd Lipcon. (Revision 1311541) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311541 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/GetConf.java > Allow GetConf tool to print arbitrary keys > -- > > Key: HDFS-3226 > URL: https://issues.apache.org/jira/browse/HDFS-3226 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Affects Versions: 2.0.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 2.0.0 > > Attachments: hdfs-3226.txt > > > Currently the "hdfs getconf" tool can only print out certain keys, like the > list of NNs, etc. It would be handy to allow it to fetch an arbitrary > configuration. For example, users may wish to write shell scripts that > interact with their hadoop cluster, and it is useful to be able to fetch > configs like the name of the superuser, or the state of whether HA is enabled. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3240) Drop log level of "heartbeat: ..." in BPServiceActor to DEBUG
[
https://issues.apache.org/jira/browse/HDFS-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated HDFS-3240:
--
Status: Patch Available (was: Open)
> Drop log level of "heartbeat: ..." in BPServiceActor to DEBUG
> -
>
> Key: HDFS-3240
> URL: https://issues.apache.org/jira/browse/HDFS-3240
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
>Affects Versions: 2.0.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Trivial
> Attachments: hdfs-3240.txt
>
>
> The following log message is at INFO level, but should be at DEBUG level:
> {code}
> LOG.info("heartbeat: " + this);
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3240) Drop log level of "heartbeat: ..." in BPServiceActor to DEBUG
[
https://issues.apache.org/jira/browse/HDFS-3240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated HDFS-3240:
--
Attachment: hdfs-3240.txt
> Drop log level of "heartbeat: ..." in BPServiceActor to DEBUG
> -
>
> Key: HDFS-3240
> URL: https://issues.apache.org/jira/browse/HDFS-3240
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
>Affects Versions: 2.0.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Trivial
> Attachments: hdfs-3240.txt
>
>
> The following log message is at INFO level, but should be at DEBUG level:
> {code}
> LOG.info("heartbeat: " + this);
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3240) Drop log level of "heartbeat: ..." in BPServiceActor to DEBUG
Drop log level of "heartbeat: ..." in BPServiceActor to DEBUG
-
Key: HDFS-3240
URL: https://issues.apache.org/jira/browse/HDFS-3240
Project: Hadoop HDFS
Issue Type: Improvement
Components: data-node
Affects Versions: 2.0.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Trivial
Attachments: hdfs-3240.txt
The following log message is at INFO level, but should be at DEBUG level:
{code}
LOG.info("heartbeat: " + this);
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3222) DFSInputStream#openInfo should not silently get the length as 0 when locations length is zero for last partial block.
[
https://issues.apache.org/jira/browse/HDFS-3222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250369#comment-13250369
]
Todd Lipcon commented on HDFS-3222:
---
Sounds good to me.
> DFSInputStream#openInfo should not silently get the length as 0 when
> locations length is zero for last partial block.
> -
>
> Key: HDFS-3222
> URL: https://issues.apache.org/jira/browse/HDFS-3222
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
>Affects Versions: 1.0.3, 2.0.0, 3.0.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
> Attachments: HDFS-3222-Test.patch
>
>
> I have seen one situation with Hbase cluster.
> Scenario is as follows:
> 1)1.5 blocks has been written and synced.
> 2)Suddenly cluster has been restarted.
> Reader opened the file and trying to get the length., By this time partial
> block contained DNs are not reported to NN. So, locations for this partial
> block would be 0. In this case, DFSInputStream assumes that, 1 block size as
> final size.
> But reader also assuming that, 1 block size is the final length and setting
> his end marker. Finally reader ending up reading only partial data. Due to
> this, HMaster could not replay the complete edits.
> Actually this happend with 20 version. Looking at the code, same should
> present in trunk as well.
> {code}
> int replicaNotFoundCount = locatedblock.getLocations().length;
>
> for(DatanodeInfo datanode : locatedblock.getLocations()) {
> ..
> ..
> // Namenode told us about these locations, but none know about the replica
> // means that we hit the race between pipeline creation start and end.
> // we require all 3 because some other exception could have happened
> // on a DN that has it. we want to report that error
> if (replicaNotFoundCount == 0) {
> return 0;
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3226) Allow GetConf tool to print arbitrary keys
[ https://issues.apache.org/jira/browse/HDFS-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250368#comment-13250368 ] Hudson commented on HDFS-3226: -- Integrated in Hadoop-Mapreduce-trunk-Commit #2045 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2045/]) HDFS-3234. Accidentally left log message in GetConf after HDFS-3226. Contributed by Todd Lipcon. (Revision 1311541) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311541 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/GetConf.java > Allow GetConf tool to print arbitrary keys > -- > > Key: HDFS-3226 > URL: https://issues.apache.org/jira/browse/HDFS-3226 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Affects Versions: 2.0.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 2.0.0 > > Attachments: hdfs-3226.txt > > > Currently the "hdfs getconf" tool can only print out certain keys, like the > list of NNs, etc. It would be handy to allow it to fetch an arbitrary > configuration. For example, users may wish to write shell scripts that > interact with their hadoop cluster, and it is useful to be able to fetch > configs like the name of the superuser, or the state of whether HA is enabled. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3234) Accidentally left log message in GetConf after HDFS-3226
[ https://issues.apache.org/jira/browse/HDFS-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250367#comment-13250367 ] Hudson commented on HDFS-3234: -- Integrated in Hadoop-Mapreduce-trunk-Commit #2045 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2045/]) HDFS-3234. Accidentally left log message in GetConf after HDFS-3226. Contributed by Todd Lipcon. (Revision 1311541) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311541 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/GetConf.java > Accidentally left log message in GetConf after HDFS-3226 > > > Key: HDFS-3234 > URL: https://issues.apache.org/jira/browse/HDFS-3234 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 2.0.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Trivial > Fix For: 2.0.0 > > Attachments: hdfs-3234.txt > > > I accidentally left a debug printout in. It doesn't cause a functionality > regression, but it does cause noisy output on the command line: > $ ./bin/hdfs getconf -confKey fs.defaultFS > key: fs.defaultFS > hdfs://nameserviceId1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2983) Relax the build version check to permit rolling upgrades within a release
[ https://issues.apache.org/jira/browse/HDFS-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250366#comment-13250366 ] Todd Lipcon commented on HDFS-2983: --- bq. The scenario that scares me is if somebody does a snapshot, then several rolling upgrades, and then decides to rollback. This may be possible, but seems to be very much error-prone. Why is this scenario different than if somebody does a snapshot, then several _non-rolling_ upgrades, then decides to rollback? In both cases, we have the case of a newer version trying to do a rollback to an older version snapshot. Right? > Relax the build version check to permit rolling upgrades within a release > - > > Key: HDFS-2983 > URL: https://issues.apache.org/jira/browse/HDFS-2983 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Eli Collins >Assignee: Aaron T. Myers > Attachments: HDFS-2983.patch, HDFS-2983.patch, HDFS-2983.patch, > HDFS-2983.patch, HDFS-2983.patch, HDFS-2983.patch, HDFS-2983.patch > > > Currently the version check for DN/NN communication is strict (it checks the > exact svn revision or git hash, Storage#getBuildVersion calls > VersionInfo#getRevision), which prevents rolling upgrades across any > releases. Once we have the PB-base RPC in place (coming soon to branch-23) > we'll have the necessary pieces in place to loosen this restriction, though > perhaps it takes another 23 minor release or so before we're ready to commit > to making the minor versions compatible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3239) Hadoop-1.0.2 is taking the wrong class path during setup
Hadoop-1.0.2 is taking the wrong class path during setup - Key: HDFS-3239 URL: https://issues.apache.org/jira/browse/HDFS-3239 Project: Hadoop HDFS Issue Type: Bug Environment: I am using 32 bit ubuntu 11.10 Reporter: Srikanth Kommineni /usr/libexec/../bin/hadoop: line 321: /usr/lib/jvm/java-6-sun/bin/java: No such file or directory /usr/libexec/../bin/hadoop: line 387: /usr/lib/jvm/java-6-sun/bin/java: No such file or directory -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2983) Relax the build version check to permit rolling upgrades within a release
[ https://issues.apache.org/jira/browse/HDFS-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250364#comment-13250364 ] Konstantin Shvachko commented on HDFS-2983: --- Yes, ctime will prevent DNs from previous versions connecting to NN started with -upgrade. Can we add a condition to allow rolling-upgrades only if there are no upgrades-snapshots in progress. That is one should first finalize or rollback the upgrade-snapshot, then do rolling upgrades. The scenario that scares me is if somebody does a snapshot, then several rolling upgrades, and then decides to rollback. This may be possible, but seems to be very much error-prone. Also that way we will have clear separation between snapshots and rolling upgrades. > Relax the build version check to permit rolling upgrades within a release > - > > Key: HDFS-2983 > URL: https://issues.apache.org/jira/browse/HDFS-2983 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Eli Collins >Assignee: Aaron T. Myers > Attachments: HDFS-2983.patch, HDFS-2983.patch, HDFS-2983.patch, > HDFS-2983.patch, HDFS-2983.patch, HDFS-2983.patch, HDFS-2983.patch > > > Currently the version check for DN/NN communication is strict (it checks the > exact svn revision or git hash, Storage#getBuildVersion calls > VersionInfo#getRevision), which prevents rolling upgrades across any > releases. Once we have the PB-base RPC in place (coming soon to branch-23) > we'll have the necessary pieces in place to loosen this restriction, though > perhaps it takes another 23 minor release or so before we're ready to commit > to making the minor versions compatible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3238) ServerCommand and friends don't need to be writables
[ https://issues.apache.org/jira/browse/HDFS-3238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250359#comment-13250359 ] Todd Lipcon commented on HDFS-3238: --- +1 pending jenkins results > ServerCommand and friends don't need to be writables > > > Key: HDFS-3238 > URL: https://issues.apache.org/jira/browse/HDFS-3238 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Eli Collins >Assignee: Eli Collins > Attachments: hdfs-3238.txt > > > We can remove writable infrastructure from the ServerCommand classes as > they're not uses across clients and we're PB within the server side. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3234) Accidentally left log message in GetConf after HDFS-3226
[ https://issues.apache.org/jira/browse/HDFS-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-3234: -- Resolution: Fixed Fix Version/s: 2.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk and branch-2, thx. > Accidentally left log message in GetConf after HDFS-3226 > > > Key: HDFS-3234 > URL: https://issues.apache.org/jira/browse/HDFS-3234 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 2.0.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Trivial > Fix For: 2.0.0 > > Attachments: hdfs-3234.txt > > > I accidentally left a debug printout in. It doesn't cause a functionality > regression, but it does cause noisy output on the command line: > $ ./bin/hdfs getconf -confKey fs.defaultFS > key: fs.defaultFS > hdfs://nameserviceId1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3238) ServerCommand and friends don't need to be writables
ServerCommand and friends don't need to be writables Key: HDFS-3238 URL: https://issues.apache.org/jira/browse/HDFS-3238 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Eli Collins Assignee: Eli Collins Attachments: hdfs-3238.txt We can remove writable infrastructure from the ServerCommand classes as they're not uses across clients and we're PB within the server side. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3238) ServerCommand and friends don't need to be writables
[ https://issues.apache.org/jira/browse/HDFS-3238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-3238: -- Attachment: hdfs-3238.txt Patch attached. > ServerCommand and friends don't need to be writables > > > Key: HDFS-3238 > URL: https://issues.apache.org/jira/browse/HDFS-3238 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Eli Collins >Assignee: Eli Collins > Attachments: hdfs-3238.txt > > > We can remove writable infrastructure from the ServerCommand classes as > they're not uses across clients and we're PB within the server side. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3238) ServerCommand and friends don't need to be writables
[ https://issues.apache.org/jira/browse/HDFS-3238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-3238: -- Status: Patch Available (was: Open) > ServerCommand and friends don't need to be writables > > > Key: HDFS-3238 > URL: https://issues.apache.org/jira/browse/HDFS-3238 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Eli Collins >Assignee: Eli Collins > Attachments: hdfs-3238.txt > > > We can remove writable infrastructure from the ServerCommand classes as > they're not uses across clients and we're PB within the server side. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3234) Accidentally left log message in GetConf after HDFS-3226
[ https://issues.apache.org/jira/browse/HDFS-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250344#comment-13250344 ] Hadoop QA commented on HDFS-3234: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522037/hdfs-3234.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2236//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2236//console This message is automatically generated. > Accidentally left log message in GetConf after HDFS-3226 > > > Key: HDFS-3234 > URL: https://issues.apache.org/jira/browse/HDFS-3234 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 2.0.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Trivial > Attachments: hdfs-3234.txt > > > I accidentally left a debug printout in. It doesn't cause a functionality > regression, but it does cause noisy output on the command line: > $ ./bin/hdfs getconf -confKey fs.defaultFS > key: fs.defaultFS > hdfs://nameserviceId1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3094) add -nonInteractive and -force option to namenode -format command
[ https://issues.apache.org/jira/browse/HDFS-3094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250341#comment-13250341 ] Arpit Gupta commented on HDFS-3094: --- i believe test failure is unrelated. > add -nonInteractive and -force option to namenode -format command > - > > Key: HDFS-3094 > URL: https://issues.apache.org/jira/browse/HDFS-3094 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.24.0, 1.0.2 >Reporter: Arpit Gupta >Assignee: Arpit Gupta > Attachments: HDFS-3094.branch-1.0.patch, HDFS-3094.branch-1.0.patch, > HDFS-3094.branch-1.0.patch, HDFS-3094.branch-1.0.patch, > HDFS-3094.branch-1.0.patch, HDFS-3094.branch-1.0.patch, > HDFS-3094.branch-1.0.patch, HDFS-3094.branch-1.0.patch, HDFS-3094.patch, > HDFS-3094.patch, HDFS-3094.patch, HDFS-3094.patch, HDFS-3094.patch, > HDFS-3094.patch, HDFS-3094.patch > > > Currently the bin/hadoop namenode -format prompts the user for a Y/N to setup > the directories in the local file system. > -force : namenode formats the directories without prompting > -nonInterActive : namenode format will return with an exit code of 1 if the > dir exists. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3094) add -nonInteractive and -force option to namenode -format command
[ https://issues.apache.org/jira/browse/HDFS-3094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250340#comment-13250340 ] Hadoop QA commented on HDFS-3094: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522032/HDFS-3094.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestFileAppend4 +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2235//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2235//console This message is automatically generated. > add -nonInteractive and -force option to namenode -format command > - > > Key: HDFS-3094 > URL: https://issues.apache.org/jira/browse/HDFS-3094 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.24.0, 1.0.2 >Reporter: Arpit Gupta >Assignee: Arpit Gupta > Attachments: HDFS-3094.branch-1.0.patch, HDFS-3094.branch-1.0.patch, > HDFS-3094.branch-1.0.patch, HDFS-3094.branch-1.0.patch, > HDFS-3094.branch-1.0.patch, HDFS-3094.branch-1.0.patch, > HDFS-3094.branch-1.0.patch, HDFS-3094.branch-1.0.patch, HDFS-3094.patch, > HDFS-3094.patch, HDFS-3094.patch, HDFS-3094.patch, HDFS-3094.patch, > HDFS-3094.patch, HDFS-3094.patch > > > Currently the bin/hadoop namenode -format prompts the user for a Y/N to setup > the directories in the local file system. > -force : namenode formats the directories without prompting > -nonInterActive : namenode format will return with an exit code of 1 if the > dir exists. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3236) NameNode does not initialize generic conf keys when started with -initializeSharedEditsDir
[ https://issues.apache.org/jira/browse/HDFS-3236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250312#comment-13250312 ] Todd Lipcon commented on HDFS-3236: --- +1 pending jenkins > NameNode does not initialize generic conf keys when started with > -initializeSharedEditsDir > -- > > Key: HDFS-3236 > URL: https://issues.apache.org/jira/browse/HDFS-3236 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha, name-node >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers >Priority: Minor > Attachments: HDFS-3236.patch > > > This means that configurations that scope the location of the > name/edits/shared edits dirs by nameserice or namenode won't work with `hdfs > namenode -initializeSharedEdits'. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-3159) Document NN auto-failover setup and configuration
[ https://issues.apache.org/jira/browse/HDFS-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon reassigned HDFS-3159: - Assignee: Todd Lipcon > Document NN auto-failover setup and configuration > - > > Key: HDFS-3159 > URL: https://issues.apache.org/jira/browse/HDFS-3159 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: auto-failover, documentation, ha >Affects Versions: Auto failover (HDFS-3042) >Reporter: Todd Lipcon >Assignee: Todd Lipcon > > We should document how to configure, set up, and monitor an automatic > failover setup. This will require adding the new configs to the *-default.xml > and adding prose to the "apt" docs as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3237) DatanodeInfo should have a DatanodeID rather than extend it
DatanodeInfo should have a DatanodeID rather than extend it --- Key: HDFS-3237 URL: https://issues.apache.org/jira/browse/HDFS-3237 Project: Hadoop HDFS Issue Type: Improvement Reporter: Eli Collins Assignee: Eli Collins Priority: Minor DatanodeInfo currently extends DatanodeID, the code would be more clear if it had a DatanodeID member instead, as DatanodeInfo is private within the server side and DatanodeID is passed to clients. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3236) NameNode does not initialize generic conf keys when started with -initializeSharedEditsDir
[ https://issues.apache.org/jira/browse/HDFS-3236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-3236: - Status: Patch Available (was: Open) > NameNode does not initialize generic conf keys when started with > -initializeSharedEditsDir > -- > > Key: HDFS-3236 > URL: https://issues.apache.org/jira/browse/HDFS-3236 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha, name-node >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers >Priority: Minor > Attachments: HDFS-3236.patch > > > This means that configurations that scope the location of the > name/edits/shared edits dirs by nameserice or namenode won't work with `hdfs > namenode -initializeSharedEdits'. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3236) NameNode does not initialize generic conf keys when started with -initializeSharedEditsDir
NameNode does not initialize generic conf keys when started with -initializeSharedEditsDir -- Key: HDFS-3236 URL: https://issues.apache.org/jira/browse/HDFS-3236 Project: Hadoop HDFS Issue Type: Bug Components: ha, name-node Affects Versions: 2.0.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Priority: Minor Attachments: HDFS-3236.patch This means that configurations that scope the location of the name/edits/shared edits dirs by nameserice or namenode won't work with `hdfs namenode -initializeSharedEdits'. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3236) NameNode does not initialize generic conf keys when started with -initializeSharedEditsDir
[ https://issues.apache.org/jira/browse/HDFS-3236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-3236: - Attachment: HDFS-3236.patch Here's a patch which addresses the issue. > NameNode does not initialize generic conf keys when started with > -initializeSharedEditsDir > -- > > Key: HDFS-3236 > URL: https://issues.apache.org/jira/browse/HDFS-3236 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha, name-node >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers >Priority: Minor > Attachments: HDFS-3236.patch > > > This means that configurations that scope the location of the > name/edits/shared edits dirs by nameserice or namenode won't work with `hdfs > namenode -initializeSharedEdits'. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3235) MiniDFSClusterManager doesn't correctly support -format option
[ https://issues.apache.org/jira/browse/HDFS-3235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250305#comment-13250305 ] Henry Robinson commented on HDFS-3235: -- sure thing. I'll probably only need another thirty or so of these to get it right. > MiniDFSClusterManager doesn't correctly support -format option > -- > > Key: HDFS-3235 > URL: https://issues.apache.org/jira/browse/HDFS-3235 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Henry Robinson >Assignee: Henry Robinson >Priority: Minor > Attachments: HDFS-3235.0.patch > > > MiniDFSClusterManager.java correctly honours -format for setting > StartupOption.FORMAT, but does not set .format(true) on the > MiniDFSClusterBuilder. This means the datanodes' data dirs will be formatted > every time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3235) MiniDFSClusterManager doesn't correctly support -format option
[ https://issues.apache.org/jira/browse/HDFS-3235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-3235: - Target Version/s: 3.0.0 Affects Version/s: 3.0.0 Fix Version/s: (was: 3.0.0) Hey Henry, in the future, please set the "affects version" and "target versions" fields when filing a JIRA, and only set the "fix version" field when the JIRA is resolved. > MiniDFSClusterManager doesn't correctly support -format option > -- > > Key: HDFS-3235 > URL: https://issues.apache.org/jira/browse/HDFS-3235 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Henry Robinson >Assignee: Henry Robinson >Priority: Minor > Attachments: HDFS-3235.0.patch > > > MiniDFSClusterManager.java correctly honours -format for setting > StartupOption.FORMAT, but does not set .format(true) on the > MiniDFSClusterBuilder. This means the datanodes' data dirs will be formatted > every time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3235) MiniDFSClusterManager doesn't correctly support -format option
[ https://issues.apache.org/jira/browse/HDFS-3235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Henry Robinson updated HDFS-3235: - Status: Patch Available (was: Open) > MiniDFSClusterManager doesn't correctly support -format option > -- > > Key: HDFS-3235 > URL: https://issues.apache.org/jira/browse/HDFS-3235 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Henry Robinson >Assignee: Henry Robinson >Priority: Minor > Fix For: 3.0.0 > > Attachments: HDFS-3235.0.patch > > > MiniDFSClusterManager.java correctly honours -format for setting > StartupOption.FORMAT, but does not set .format(true) on the > MiniDFSClusterBuilder. This means the datanodes' data dirs will be formatted > every time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3234) Accidentally left log message in GetConf after HDFS-3226
[ https://issues.apache.org/jira/browse/HDFS-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250299#comment-13250299 ] Aaron T. Myers commented on HDFS-3234: -- +1 pending Jenkins. > Accidentally left log message in GetConf after HDFS-3226 > > > Key: HDFS-3234 > URL: https://issues.apache.org/jira/browse/HDFS-3234 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 2.0.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Trivial > Attachments: hdfs-3234.txt > > > I accidentally left a debug printout in. It doesn't cause a functionality > regression, but it does cause noisy output on the command line: > $ ./bin/hdfs getconf -confKey fs.defaultFS > key: fs.defaultFS > hdfs://nameserviceId1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3235) MiniDFSClusterManager doesn't correctly support -format option
[ https://issues.apache.org/jira/browse/HDFS-3235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Henry Robinson updated HDFS-3235: - Attachment: HDFS-3235.0.patch Patch against trunk > MiniDFSClusterManager doesn't correctly support -format option > -- > > Key: HDFS-3235 > URL: https://issues.apache.org/jira/browse/HDFS-3235 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Henry Robinson >Assignee: Henry Robinson >Priority: Minor > Fix For: 3.0.0 > > Attachments: HDFS-3235.0.patch > > > MiniDFSClusterManager.java correctly honours -format for setting > StartupOption.FORMAT, but does not set .format(true) on the > MiniDFSClusterBuilder. This means the datanodes' data dirs will be formatted > every time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3235) MiniDFSClusterManager doesn't correctly support -format option
MiniDFSClusterManager doesn't correctly support -format option -- Key: HDFS-3235 URL: https://issues.apache.org/jira/browse/HDFS-3235 Project: Hadoop HDFS Issue Type: Bug Reporter: Henry Robinson Assignee: Henry Robinson Priority: Minor Fix For: 3.0.0 MiniDFSClusterManager.java correctly honours -format for setting StartupOption.FORMAT, but does not set .format(true) on the MiniDFSClusterBuilder. This means the datanodes' data dirs will be formatted every time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2983) Relax the build version check to permit rolling upgrades within a release
[ https://issues.apache.org/jira/browse/HDFS-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250297#comment-13250297 ] Hadoop QA commented on HDFS-2983: - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522027/HDFS-2983.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2234//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2234//console This message is automatically generated. > Relax the build version check to permit rolling upgrades within a release > - > > Key: HDFS-2983 > URL: https://issues.apache.org/jira/browse/HDFS-2983 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Eli Collins >Assignee: Aaron T. Myers > Attachments: HDFS-2983.patch, HDFS-2983.patch, HDFS-2983.patch, > HDFS-2983.patch, HDFS-2983.patch, HDFS-2983.patch, HDFS-2983.patch > > > Currently the version check for DN/NN communication is strict (it checks the > exact svn revision or git hash, Storage#getBuildVersion calls > VersionInfo#getRevision), which prevents rolling upgrades across any > releases. Once we have the PB-base RPC in place (coming soon to branch-23) > we'll have the necessary pieces in place to loosen this restriction, though > perhaps it takes another 23 minor release or so before we're ready to commit > to making the minor versions compatible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3234) Accidentally left log message in GetConf after HDFS-3226
[ https://issues.apache.org/jira/browse/HDFS-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-3234: -- Status: Patch Available (was: Open) > Accidentally left log message in GetConf after HDFS-3226 > > > Key: HDFS-3234 > URL: https://issues.apache.org/jira/browse/HDFS-3234 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 2.0.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Trivial > Attachments: hdfs-3234.txt > > > I accidentally left a debug printout in. It doesn't cause a functionality > regression, but it does cause noisy output on the command line: > $ ./bin/hdfs getconf -confKey fs.defaultFS > key: fs.defaultFS > hdfs://nameserviceId1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3234) Accidentally left log message in GetConf after HDFS-3226
[ https://issues.apache.org/jira/browse/HDFS-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-3234: -- Attachment: hdfs-3234.txt Attached trivial patch removes the log message. No test since it's just an error printout. > Accidentally left log message in GetConf after HDFS-3226 > > > Key: HDFS-3234 > URL: https://issues.apache.org/jira/browse/HDFS-3234 > Project: Hadoop HDFS > Issue Type: Bug > Components: tools >Affects Versions: 2.0.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Trivial > Attachments: hdfs-3234.txt > > > I accidentally left a debug printout in. It doesn't cause a functionality > regression, but it does cause noisy output on the command line: > $ ./bin/hdfs getconf -confKey fs.defaultFS > key: fs.defaultFS > hdfs://nameserviceId1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3234) Accidentally left log message in GetConf after HDFS-3226
Accidentally left log message in GetConf after HDFS-3226 Key: HDFS-3234 URL: https://issues.apache.org/jira/browse/HDFS-3234 Project: Hadoop HDFS Issue Type: Bug Components: tools Affects Versions: 2.0.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Trivial I accidentally left a debug printout in. It doesn't cause a functionality regression, but it does cause noisy output on the command line: $ ./bin/hdfs getconf -confKey fs.defaultFS key: fs.defaultFS hdfs://nameserviceId1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3094) add -nonInteractive and -force option to namenode -format command
[ https://issues.apache.org/jira/browse/HDFS-3094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Gupta updated HDFS-3094: -- Attachment: HDFS-3094.patch updated patch for trunk. Todd since force option is being used for recovery, i changed the vars for this patch to isForceFormat and isInteractiveFormat and the appropriate methods. > add -nonInteractive and -force option to namenode -format command > - > > Key: HDFS-3094 > URL: https://issues.apache.org/jira/browse/HDFS-3094 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.24.0, 1.0.2 >Reporter: Arpit Gupta >Assignee: Arpit Gupta > Attachments: HDFS-3094.branch-1.0.patch, HDFS-3094.branch-1.0.patch, > HDFS-3094.branch-1.0.patch, HDFS-3094.branch-1.0.patch, > HDFS-3094.branch-1.0.patch, HDFS-3094.branch-1.0.patch, > HDFS-3094.branch-1.0.patch, HDFS-3094.branch-1.0.patch, HDFS-3094.patch, > HDFS-3094.patch, HDFS-3094.patch, HDFS-3094.patch, HDFS-3094.patch, > HDFS-3094.patch, HDFS-3094.patch > > > Currently the bin/hadoop namenode -format prompts the user for a Y/N to setup > the directories in the local file system. > -force : namenode formats the directories without prompting > -nonInterActive : namenode format will return with an exit code of 1 if the > dir exists. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3229) add JournalProtocol RPCs to list finalized edit segments, and read edit segment file from JournalNode.
[ https://issues.apache.org/jira/browse/HDFS-3229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250251#comment-13250251 ] Todd Lipcon commented on HDFS-3229: --- bq. However, if we believe we need web UI for JournalNode, we need the port anyways. I think it's a good idea, since we have other endpoints in our default HTTP server that are very useful for ops -- for example the /jmx servlet and the /conf servlet can both be very handy. I also think exposing a basic web UI is helpful to operators who might try to understand the current state of the system. bq. Suppose we used HTTP server to synchronize the lagging JournalNode by downloading missed edit logs from another Journal Node. Firstly, the lagging JN needs to get (e.g., by asking for NN) a list of JNs with full set of edit logs. Then, it downloads the missed logs from a good JN through http, while it could accept streamed logs from NN through rpc at the same time. Given the two servers are working on different file sets(finalized logs vs in-progress log), synchronizing them seems not a concern. Right - this is the same process that the 2NN uses to synchronize finalized log segments from the NN. See SecondaryNameNode.downloadCheckpointFiles for the code. > add JournalProtocol RPCs to list finalized edit segments, and read edit > segment file from JournalNode. > --- > > Key: HDFS-3229 > URL: https://issues.apache.org/jira/browse/HDFS-3229 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Reporter: Brandon Li >Assignee: Brandon Li > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3229) add JournalProtocol RPCs to list finalized edit segments, and read edit segment file from JournalNode.
[ https://issues.apache.org/jira/browse/HDFS-3229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250249#comment-13250249 ] Brandon Li commented on HDFS-3229: -- Todd, Reusing existing code has certain benefits for sure. Have an HTTP server for JournalNode requires additional port. In a large cluster, it may not be trivial to manager one more port. However, if we believe we need web UI for JournalNode, we need the port anyways. Suppose we used HTTP server to synchronize the lagging JournalNode by downloading missed edit logs from another Journal Node. Firstly, the lagging JN needs to get (e.g., by asking for NN) a list of JNs with full set of edit logs. Then, it downloads the missed logs from a good JN through http, while it could accept streamed logs from NN through rpc at the same time. Given the two servers are working on different file sets(finalized logs vs in-progress log), synchronizing them seems not a concern. Debug-ability in this case has more to do with the developer's familiarity with the code/protocol, and seems not a good enough reason to me now. Please let me know what you think. > add JournalProtocol RPCs to list finalized edit segments, and read edit > segment file from JournalNode. > --- > > Key: HDFS-3229 > URL: https://issues.apache.org/jira/browse/HDFS-3229 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Reporter: Brandon Li >Assignee: Brandon Li > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2983) Relax the build version check to permit rolling upgrades within a release
[ https://issues.apache.org/jira/browse/HDFS-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-2983: - Attachment: HDFS-2983.patch Thanks a lot for the review, Todd. Here's an updated patch which addresses your latest two comments. PS. I thought you'd like that javadoc. I put a lot of thought into it. :) > Relax the build version check to permit rolling upgrades within a release > - > > Key: HDFS-2983 > URL: https://issues.apache.org/jira/browse/HDFS-2983 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Eli Collins >Assignee: Aaron T. Myers > Attachments: HDFS-2983.patch, HDFS-2983.patch, HDFS-2983.patch, > HDFS-2983.patch, HDFS-2983.patch, HDFS-2983.patch, HDFS-2983.patch > > > Currently the version check for DN/NN communication is strict (it checks the > exact svn revision or git hash, Storage#getBuildVersion calls > VersionInfo#getRevision), which prevents rolling upgrades across any > releases. Once we have the PB-base RPC in place (coming soon to branch-23) > we'll have the necessary pieces in place to loosen this restriction, though > perhaps it takes another 23 minor release or so before we're ready to commit > to making the minor versions compatible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3094) add -nonInteractive and -force option to namenode -format command
[ https://issues.apache.org/jira/browse/HDFS-3094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250235#comment-13250235 ] Todd Lipcon commented on HDFS-3094: --- Sorry again Arpit - looks like the commit of HDFS-3004 caused another conflict here just a couple hours ago... > add -nonInteractive and -force option to namenode -format command > - > > Key: HDFS-3094 > URL: https://issues.apache.org/jira/browse/HDFS-3094 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.24.0, 1.0.2 >Reporter: Arpit Gupta >Assignee: Arpit Gupta > Attachments: HDFS-3094.branch-1.0.patch, HDFS-3094.branch-1.0.patch, > HDFS-3094.branch-1.0.patch, HDFS-3094.branch-1.0.patch, > HDFS-3094.branch-1.0.patch, HDFS-3094.branch-1.0.patch, > HDFS-3094.branch-1.0.patch, HDFS-3094.branch-1.0.patch, HDFS-3094.patch, > HDFS-3094.patch, HDFS-3094.patch, HDFS-3094.patch, HDFS-3094.patch, > HDFS-3094.patch > > > Currently the bin/hadoop namenode -format prompts the user for a Y/N to setup > the directories in the local file system. > -force : namenode formats the directories without prompting > -nonInterActive : namenode format will return with an exit code of 1 if the > dir exists. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3094) add -nonInteractive and -force option to namenode -format command
[ https://issues.apache.org/jira/browse/HDFS-3094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250222#comment-13250222 ] Arpit Gupta commented on HDFS-3094: --- test failure does not seem to be related to the patch. > add -nonInteractive and -force option to namenode -format command > - > > Key: HDFS-3094 > URL: https://issues.apache.org/jira/browse/HDFS-3094 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.24.0, 1.0.2 >Reporter: Arpit Gupta >Assignee: Arpit Gupta > Attachments: HDFS-3094.branch-1.0.patch, HDFS-3094.branch-1.0.patch, > HDFS-3094.branch-1.0.patch, HDFS-3094.branch-1.0.patch, > HDFS-3094.branch-1.0.patch, HDFS-3094.branch-1.0.patch, > HDFS-3094.branch-1.0.patch, HDFS-3094.branch-1.0.patch, HDFS-3094.patch, > HDFS-3094.patch, HDFS-3094.patch, HDFS-3094.patch, HDFS-3094.patch, > HDFS-3094.patch > > > Currently the bin/hadoop namenode -format prompts the user for a Y/N to setup > the directories in the local file system. > -force : namenode formats the directories without prompting > -nonInterActive : namenode format will return with an exit code of 1 if the > dir exists. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2983) Relax the build version check to permit rolling upgrades within a release
[
https://issues.apache.org/jira/browse/HDFS-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250211#comment-13250211
]
Todd Lipcon commented on HDFS-2983:
---
{code}
+if (!dnVersion.equals(nnVersion)) {
+ LOG.info("Reported DataNode version '" + dnVersion + "' does not match "
+
+ "NameNode version '" + nnVersion + "' but is within acceptable " +
+ "limits. Note: This is normal during a rolling upgrade.");
+}
{code}
Can you also please include the DN IP address in this log message?
- Nice lengthy javadoc on VersionUtil.compareVersions. Can you please add
something like:
"This method of comparison is similar to the method used by package versioning
systems like deb and RPM"
and also maybe give one example of what you mean? eg add "For example, Hadoop
0.3 < Hadoop 0.20 even though naive string comparison would consider it larger."
Otherwise, looks great. +1 from my standpoint. Konstantin/Sanjay - can you
please comment regarding the above discussion? While I agree that there are
more improvements to be made, I don't think this patch will hurt things. Or, if
you are nervous about it, can we commit this with a flag to allow rolling
upgrade if the operator permits it?
> Relax the build version check to permit rolling upgrades within a release
> -
>
> Key: HDFS-2983
> URL: https://issues.apache.org/jira/browse/HDFS-2983
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Eli Collins
>Assignee: Aaron T. Myers
> Attachments: HDFS-2983.patch, HDFS-2983.patch, HDFS-2983.patch,
> HDFS-2983.patch, HDFS-2983.patch, HDFS-2983.patch
>
>
> Currently the version check for DN/NN communication is strict (it checks the
> exact svn revision or git hash, Storage#getBuildVersion calls
> VersionInfo#getRevision), which prevents rolling upgrades across any
> releases. Once we have the PB-base RPC in place (coming soon to branch-23)
> we'll have the necessary pieces in place to loosen this restriction, though
> perhaps it takes another 23 minor release or so before we're ready to commit
> to making the minor versions compatible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3055) Implement recovery mode for branch-1
[ https://issues.apache.org/jira/browse/HDFS-3055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250194#comment-13250194 ] Todd Lipcon commented on HDFS-3055: --- OK. +1, patch looks good. Please run all the branch-1 unit tests so we don't introduce any other failures - should be OK but best to be safe on the stable branch. When you report back, I'll commit. > Implement recovery mode for branch-1 > > > Key: HDFS-3055 > URL: https://issues.apache.org/jira/browse/HDFS-3055 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > Fix For: 1.0.0 > > Attachments: HDFS-3055-b1.001.patch, HDFS-3055-b1.002.patch, > HDFS-3055-b1.003.patch, HDFS-3055-b1.004.patch, HDFS-3055-b1.005.patch, > HDFS-3055-b1.006.patch > > > Implement recovery mode for branch-1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-2995) start-dfs.sh should only start the 2NN for namenodes with dfs.namenode.secondary.http-address configured
[ https://issues.apache.org/jira/browse/HDFS-2995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon resolved HDFS-2995. --- Resolution: Fixed Fix Version/s: 2.0.0 Hadoop Flags: Reviewed Looks like this got committed to trunk and branch-2 last week. > start-dfs.sh should only start the 2NN for namenodes with > dfs.namenode.secondary.http-address configured > > > Key: HDFS-2995 > URL: https://issues.apache.org/jira/browse/HDFS-2995 > Project: Hadoop HDFS > Issue Type: Bug > Components: scripts >Affects Versions: HA branch (HDFS-1623) >Reporter: Todd Lipcon >Assignee: Eli Collins > Fix For: 2.0.0 > > Attachments: hdfs-2995.txt > > > When I run "start-dfs.sh" it tries to start a 2NN on every node in the > cluster. This despite: > [todd@c1120 hadoop-active]$ ./bin/hdfs getconf -secondaryNameNodes > Incorrect configuration: secondary namenode address > dfs.namenode.secondary.http-address is not configured. > Thankfully they do not start :) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3216) DatanodeID should support multiple IP addresses
[
https://issues.apache.org/jira/browse/HDFS-3216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250181#comment-13250181
]
Todd Lipcon commented on HDFS-3216:
---
bq. #2 Yes, when reading/writing DatanodeInfos to/from streams (same as before
when creating a DatanodeID w/o a name)
When do we read/write DatanodeInfo from streams, now that we are pb-ified? i.e
is the writable interface even used anymore?
{code}
+ * Return the canonical IP address for this DatanodeID. Not all uses
+ * of DatanodeID are multi-IP aware, or would multiple IPs, therefore
+ * we use the first address as the canonical one.
{code}
ENOTASENTENCE
bq. #1 We still need the notion of canonical IP, mostly for cases that don't
care about multiple IP addresses. Updated the javadoc.
How is it ensured that the "canonical" IP is kept consistent across DN
restarts, for example? It's just whichever one is listed first in the DN-side
configuration?
bq. Fixed the cast, now casts to String and serializes/deserializes the IPs,
the test does check this (was failing now passes).
That's a little strange, to serialize it into a comma-separated list inside
JSON. It's not possible to get Jackson to serialize it as a proper JSON array?
Perhaps using a List inside the map?
> DatanodeID should support multiple IP addresses
> ---
>
> Key: HDFS-3216
> URL: https://issues.apache.org/jira/browse/HDFS-3216
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Eli Collins
>Assignee: Eli Collins
> Attachments: hdfs-3216.txt, hdfs-3216.txt
>
>
> The DatanodeID has a single field for the IP address, for HDFS-3146 we need
> to extend it to support multiple addresses.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3094) add -nonInteractive and -force option to namenode -format command
[ https://issues.apache.org/jira/browse/HDFS-3094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250171#comment-13250171 ] Hadoop QA commented on HDFS-3094: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521996/HDFS-3094.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestDatanodeBlockScanner +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2233//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2233//console This message is automatically generated. > add -nonInteractive and -force option to namenode -format command > - > > Key: HDFS-3094 > URL: https://issues.apache.org/jira/browse/HDFS-3094 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.24.0, 1.0.2 >Reporter: Arpit Gupta >Assignee: Arpit Gupta > Attachments: HDFS-3094.branch-1.0.patch, HDFS-3094.branch-1.0.patch, > HDFS-3094.branch-1.0.patch, HDFS-3094.branch-1.0.patch, > HDFS-3094.branch-1.0.patch, HDFS-3094.branch-1.0.patch, > HDFS-3094.branch-1.0.patch, HDFS-3094.branch-1.0.patch, HDFS-3094.patch, > HDFS-3094.patch, HDFS-3094.patch, HDFS-3094.patch, HDFS-3094.patch, > HDFS-3094.patch > > > Currently the bin/hadoop namenode -format prompts the user for a Y/N to setup > the directories in the local file system. > -force : namenode formats the directories without prompting > -nonInterActive : namenode format will return with an exit code of 1 if the > dir exists. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3055) Implement recovery mode for branch-1
[ https://issues.apache.org/jira/browse/HDFS-3055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-3055: --- Attachment: HDFS-3055-b1.006.patch * TestNameNodeRecovery: use StringUtils instead of StringWriter to serialize exception > Implement recovery mode for branch-1 > > > Key: HDFS-3055 > URL: https://issues.apache.org/jira/browse/HDFS-3055 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > Fix For: 1.0.0 > > Attachments: HDFS-3055-b1.001.patch, HDFS-3055-b1.002.patch, > HDFS-3055-b1.003.patch, HDFS-3055-b1.004.patch, HDFS-3055-b1.005.patch, > HDFS-3055-b1.006.patch > > > Implement recovery mode for branch-1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3055) Implement recovery mode for branch-1
[ https://issues.apache.org/jira/browse/HDFS-3055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250155#comment-13250155 ] Colin Patrick McCabe commented on HDFS-3055: > can you explain the changes in FSNamesystem.java? That change fixes error handling in FSNamesystem. Previously, we did not call FSNamesystem::shutdown() when initialization failed. This led to the MBeans staying registered. This is irrelevant when running the NameNode normally, since the MBeans are destroyed when the entire process goes away. However, when run from a test context, the next attempt to create a MiniDFSCluster instance will fail with "port in use" or some such error. > Can you update the logging in the test cases to use > StringUtils.stringifyException to match trunk? Ok. > Did you run all the existing tests in branch-1? I ran these tests: TestCheckpoint, TestEditLog, TestNameNodeRecovery, TestEditLogLoading, TestNameNodeMXBean, TestSaveNamespace, TestSecurityTokenEditLog, TestStorageDirectoryFailure, TestStorageRestore > Implement recovery mode for branch-1 > > > Key: HDFS-3055 > URL: https://issues.apache.org/jira/browse/HDFS-3055 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > Fix For: 1.0.0 > > Attachments: HDFS-3055-b1.001.patch, HDFS-3055-b1.002.patch, > HDFS-3055-b1.003.patch, HDFS-3055-b1.004.patch, HDFS-3055-b1.005.patch > > > Implement recovery mode for branch-1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2983) Relax the build version check to permit rolling upgrades within a release
[
https://issues.apache.org/jira/browse/HDFS-2983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250152#comment-13250152
]
Todd Lipcon commented on HDFS-2983:
---
I did a little investigation to try to answer Konstantin's questions above.
First, I'll summarize our current behavior, verified on 0.23.1 release (I
didn't understand this thoroughly before trying it out):
- In a running cluster, if you restart the NN without the {{-upgrade}} flag,
then the DataNodes will happily re-register without exiting.
- If you restart the NN with {{-upgrade}}, then when the DN next heartbeats, it
will fail the {{verifyRequest()}} check, since the registration ID's namespace
fields no longer match (the ctime has been incremented by the upgrade). This
causes the DataNode to exit.
- Of course, restarting the DN at this point makes it take the snapshot and
participate in the upgrade as expected.
So, to try to respond to Konstantin's questions, here are a couple example
scenarios:
*Scenario 1*: rolling upgrade without doing a "snapshot" upgrade (for emergency
bug fixes, hot fixes, MR fixes, other fixes which we don't expect to affect
data reliability):
- Leave the NN running, on the old version.
- On each DN, in succession: (1) shutdown DN, (2) upgrade software to the new
version, (3) start DN
The above is sufficient if the changes are scoped only to DNs. If the change
also affects the NN, then you will need to add the following step, either at
the beginning or end of the process:
- shutdown NN. upgrade installed software. start NN on new version
In the case of an HA setup, we can do the NN upgrade without downtime:
- shutdown SBN. upgrade SBN software. start SBN.
- failover to SBN running new version.
- Shutdown previous active. Upgrade software. Start previous active
- Optionally fail back
*Scenario 2*: upgrade to a version with a new layout version (LV)
In this case, a "snapshot" style upgrade is required -- the NN will not restart
without the "-upgrade" flag, and a DN will not connect to a NN with a different
LV. So the scenario is the same as today:
- Shutdown entire cluster
- Upgrade all software in teh clsuter
- Start cluster with {{-upgrade}} flag
-- any nodes that missed the software upgrade will fail to connect, since their
LV does not match (this patch retains that behavior)
*Scenario 3*: upgrade to a version with same layout version, but some data risk
(for example upgrading to a version with bug fixes pertaining to replication
policies, corrupt block detection, etc)
In this scenario, the NN does not mandate a {{-upgrade}} flag, but as Sanjay
mentioned above, it can still be useful for data protection. As with today, if
the user does not want the extra protection, this scenario can be treated
identically to scenario 1. If the user does want the protection, it can be
treated identically to scenario 2. Scenario 2 remains safe because of the check
against the NameNode's {{ctime}} matching the DN's {{ctime}}. As soon as you
restart the NN with the {{-upgrade}} flag, all running DNs will exit. Any newly
started DN will noticethe new namespace ctime and take part in the snapshot
upgrade.
Does the above description address your concerns? Another idea would be to add
a new configuration option like {{dfs.allow.rolling.upgrades}} which enables
the new behavior, so an admin who prefers not to use the feature can disallow
it completely.
> Relax the build version check to permit rolling upgrades within a release
> -
>
> Key: HDFS-2983
> URL: https://issues.apache.org/jira/browse/HDFS-2983
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.0.0
>Reporter: Eli Collins
>Assignee: Aaron T. Myers
> Attachments: HDFS-2983.patch, HDFS-2983.patch, HDFS-2983.patch,
> HDFS-2983.patch, HDFS-2983.patch, HDFS-2983.patch
>
>
> Currently the version check for DN/NN communication is strict (it checks the
> exact svn revision or git hash, Storage#getBuildVersion calls
> VersionInfo#getRevision), which prevents rolling upgrades across any
> releases. Once we have the PB-base RPC in place (coming soon to branch-23)
> we'll have the necessary pieces in place to loosen this restriction, though
> perhaps it takes another 23 minor release or so before we're ready to commit
> to making the minor versions compatible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3216) DatanodeID should support multiple IP addresses
[ https://issues.apache.org/jira/browse/HDFS-3216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-3216: -- Attachment: hdfs-3216.txt #1 We still need the notion of canonical IP, mostly for cases that don't care about multiple IP addresses. Updated the javadoc. #2 Yes, when reading/writing DatanodeInfos to/from streams (same as before when creating a DatanodeID w/o a name) #3 Done #4 It's tested by TestWebHdfsFileSystemContract#testGetFileBlockLocations. It currently only tests the first IP vs the whole array since LocatedBlock (HDFS) is multi-IP aware but BlockLocation (common) is not. Fixed the cast, now casts to String and serializes/deserializes the IPs, the test does check this (was failing now passes). #5 We expect this field to always contain IPs, see HDFS-3144, we no longer overload this field. Filed HDFS-3230 to cleanup the tests, will do it on trunk since its independent of this change. > DatanodeID should support multiple IP addresses > --- > > Key: HDFS-3216 > URL: https://issues.apache.org/jira/browse/HDFS-3216 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Eli Collins >Assignee: Eli Collins > Attachments: hdfs-3216.txt, hdfs-3216.txt > > > The DatanodeID has a single field for the IP address, for HDFS-3146 we need > to extend it to support multiple addresses. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3004) Implement Recovery Mode
[ https://issues.apache.org/jira/browse/HDFS-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250148#comment-13250148 ] Hudson commented on HDFS-3004: -- Integrated in Hadoop-Mapreduce-trunk-Commit #2041 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2041/]) HDFS-3004. Implement Recovery Mode. Contributed by Colin Patrick McCabe (Revision 1311394) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311394 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/docs/src/documentation/content/xdocs/hdfs_user_guide.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Checkpointer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageTransactionalStorageInspector.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileJournalManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/MetaRecoveryContext.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineEditsViewer/OfflineEditsXmlLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogRace.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeRecovery.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSecurityTokenEditLog.java > Implement Recovery Mode > --- > > Key: HDFS-3004 > URL: https://issues.apache.org/jira/browse/HDFS-3004 > Project: Hadoop HDFS > Issue Type: New Feature > Components: tools >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Fix For: 2.0.0 > > Attachments: HDFS-3004.010.
[jira] [Created] (HDFS-3233) Move IP to FQDN conversion from DatanodeJSPHelper to DatanodeID
Move IP to FQDN conversion from DatanodeJSPHelper to DatanodeID --- Key: HDFS-3233 URL: https://issues.apache.org/jira/browse/HDFS-3233 Project: Hadoop HDFS Issue Type: Improvement Reporter: Eli Collins Assignee: Eli Collins Priority: Minor In a handful of places DatanodeJSPHelper looks up the IP for a DN and then determines a FQDN for the IP. We should move this code to a single place, a new DatanodeID to return the FQDN for a DatanodeID. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3232) Cleanup DatanodeInfo vs DatanodeID handling in DN servlets
Cleanup DatanodeInfo vs DatanodeID handling in DN servlets
--
Key: HDFS-3232
URL: https://issues.apache.org/jira/browse/HDFS-3232
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Eli Collins
Assignee: Eli Collins
Priority: Minor
The DN servlets currently have code like the following:
{code}
final String hostname = host instanceof DatanodeInfo
? ((DatanodeInfo)host).getHostName() : host.getIpAddr();
{code}
I believe this outdated, that we now always get one or the other (at least when
not running the tests). Need to verify that. We should clean this code up as
well, eg always use the IP (which we'll lookup the FQDN for) since the hostname
isn't necessarily valid to put in a URL (the DN hostname isn't necesarily a
FQDN).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2799) Trim fs.checkpoint.dir values
[
https://issues.apache.org/jira/browse/HDFS-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250142#comment-13250142
]
Harsh J commented on HDFS-2799:
---
I think a simple fix of porting the Config.get… method to the right one would
be much better. Testing it too should be simple enough (run with a config and
see what dir it created?)
> Trim fs.checkpoint.dir values
> -
>
> Key: HDFS-2799
> URL: https://issues.apache.org/jira/browse/HDFS-2799
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.23.0
>Reporter: Eli Collins
> Labels: noob
> Attachments: HDFS-2799.patch
>
>
> fs.checkpoint.dir values need to be trimmed like dfs.name.dir and
> dfs.data.dir values so eg the following works. This currently results in the
> directory "HADOOP_HOME/?/home/eli/hadoop/dirs3/dfs/chkpoint1" being created.
> {noformat}
>
> fs.checkpoint.dir
>
> /home/eli/hadoop/dirs3/dfs/chkpoint1,
> /home/eli/hadoop/dirs3/dfs/chkpoint2
>
>
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3231) NN Host2NodesMap should use hostnames
NN Host2NodesMap should use hostnames - Key: HDFS-3231 URL: https://issues.apache.org/jira/browse/HDFS-3231 Project: Hadoop HDFS Issue Type: Improvement Reporter: Eli Collins Assignee: Eli Collins The NN's Host2NodesMap maps "host names" to datanode descriptors. It actually uses IP addresses and should use hostnames instead, as hostnames are a better key (eg a Datanode has one hostname but may have multiple IPs). Per HDFS-3216 there's actually a bug in that it's sometimes accessed with IP:port instead of IP, so that jira should be fixed before this one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3055) Implement recovery mode for branch-1
[
https://issues.apache.org/jira/browse/HDFS-3055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250131#comment-13250131
]
Todd Lipcon commented on HDFS-3055:
---
- can you explain the changes in FSNamesystem.java?
- Can you update the logging in the test cases to use
StringUtils.stringifyException to match trunk?
- Did you run all the existing tests in branch-1? The one difference that I can
see that might cause a failure is that the IOException thrown during a failed
startup used to retain the exception {{t}} as its cause, but no longer does.
Otherwise looks good.
> Implement recovery mode for branch-1
>
>
> Key: HDFS-3055
> URL: https://issues.apache.org/jira/browse/HDFS-3055
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>Priority: Minor
> Fix For: 1.0.0
>
> Attachments: HDFS-3055-b1.001.patch, HDFS-3055-b1.002.patch,
> HDFS-3055-b1.003.patch, HDFS-3055-b1.004.patch, HDFS-3055-b1.005.patch
>
>
> Implement recovery mode for branch-1
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3230) Cleanup DatanodeID creation in the tests
Cleanup DatanodeID creation in the tests Key: HDFS-3230 URL: https://issues.apache.org/jira/browse/HDFS-3230 Project: Hadoop HDFS Issue Type: Improvement Components: test Reporter: Eli Collins Assignee: Eli Collins Priority: Minor A lot of tests create dummy DatanodeIDs for testing, often use bogus values when creating the objects (eg hostname in the IP field), which they can get away with because the IDs aren't actually used. Let's add a test utility method for creating a DatanodeID for testing and use it throughout. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3165) HDFS Balancer scripts are refering to wrong path of hadoop-daemon.sh
[ https://issues.apache.org/jira/browse/HDFS-3165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250120#comment-13250120 ] Uma Maheswara Rao G commented on HDFS-3165: --- +1 lgtm > HDFS Balancer scripts are refering to wrong path of hadoop-daemon.sh > > > Key: HDFS-3165 > URL: https://issues.apache.org/jira/browse/HDFS-3165 > Project: Hadoop HDFS > Issue Type: Bug > Components: scripts >Affects Versions: 2.0.0, 3.0.0 > Environment: HDFS Balancer >Reporter: amith >Priority: Minor > Attachments: HDFS-3165.patch, HDFS-3165.patch, HDFS-3165.patch, > HDFS-3165_1.patch > > Original Estimate: 1m > Remaining Estimate: 1m > > HDFS Balancer scripts are refering to wrong path of hadoop-daemon.sh > HOST-xx-xx-xx-xx:/home/amith/V1R2/namenode1/sbin # ./start-balancer.sh > ./start-balancer.sh: line 27: > /home/amith/V1R2/namenode1/bin/hadoop-daemon.sh: No such file or directory > currently hadoop-daemon.sh script is in sbin and not bin. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3119) Overreplicated block is not deleted even after the replication factor is reduced after sync follwed by closing that file
[ https://issues.apache.org/jira/browse/HDFS-3119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250121#comment-13250121 ] Hudson commented on HDFS-3119: -- Integrated in Hadoop-Mapreduce-trunk-Commit #2040 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2040/]) HDFS-3119. Overreplicated block is not deleted even after the replication factor is reduced after sync follwed by closing that file. Contributed by Ashish Singhi. (Revision 1311380) Result = ABORTED umamahesh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311380 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestOverReplicatedBlocks.java > Overreplicated block is not deleted even after the replication factor is > reduced after sync follwed by closing that file > > > Key: HDFS-3119 > URL: https://issues.apache.org/jira/browse/HDFS-3119 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.24.0 >Reporter: J.Andreina >Assignee: Ashish Singhi >Priority: Minor > Labels: patch > Fix For: 0.24.0, 2.0.0 > > Attachments: HDFS-3119-1.patch, HDFS-3119-1.patch, HDFS-3119.patch > > > cluster setup: > -- > 1NN,2 DN,replication factor 2,block report interval 3sec ,block size-256MB > step1: write a file "filewrite.txt" of size 90bytes with sync(not closed) > step2: change the replication factor to 1 using the command: "./hdfs dfs > -setrep 1 /filewrite.txt" > step3: close the file > * At the NN side the file "Decreasing replication from 2 to 1 for > /filewrite.txt" , logs has occured but the overreplicated blocks are not > deleted even after the block report is sent from DN > * while listing the file in the console using "./hdfs dfs -ls " the > replication factor for that file is mentioned as 1 > * In fsck report for that files displays that the file is replicated to 2 > datanodes -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3165) HDFS Balancer scripts are refering to wrong path of hadoop-daemon.sh
[ https://issues.apache.org/jira/browse/HDFS-3165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-3165: -- Component/s: (was: balancer) scripts Target Version/s: 2.0.0, 3.0.0 Affects Version/s: (was: 1.0.1) 3.0.0 2.0.0 > HDFS Balancer scripts are refering to wrong path of hadoop-daemon.sh > > > Key: HDFS-3165 > URL: https://issues.apache.org/jira/browse/HDFS-3165 > Project: Hadoop HDFS > Issue Type: Bug > Components: scripts >Affects Versions: 2.0.0, 3.0.0 > Environment: HDFS Balancer >Reporter: amith >Priority: Minor > Attachments: HDFS-3165.patch, HDFS-3165.patch, HDFS-3165.patch, > HDFS-3165_1.patch > > Original Estimate: 1m > Remaining Estimate: 1m > > HDFS Balancer scripts are refering to wrong path of hadoop-daemon.sh > HOST-xx-xx-xx-xx:/home/amith/V1R2/namenode1/sbin # ./start-balancer.sh > ./start-balancer.sh: line 27: > /home/amith/V1R2/namenode1/bin/hadoop-daemon.sh: No such file or directory > currently hadoop-daemon.sh script is in sbin and not bin. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3004) Implement Recovery Mode
[ https://issues.apache.org/jira/browse/HDFS-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250112#comment-13250112 ] Todd Lipcon commented on HDFS-3004: --- Can you add a release note field for this issue, with brief description of the new feature and a pointer to the docs that describe it? > Implement Recovery Mode > --- > > Key: HDFS-3004 > URL: https://issues.apache.org/jira/browse/HDFS-3004 > Project: Hadoop HDFS > Issue Type: New Feature > Components: tools >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Fix For: 2.0.0 > > Attachments: HDFS-3004.010.patch, HDFS-3004.011.patch, > HDFS-3004.012.patch, HDFS-3004.013.patch, HDFS-3004.015.patch, > HDFS-3004.016.patch, HDFS-3004.017.patch, HDFS-3004.018.patch, > HDFS-3004.019.patch, HDFS-3004.020.patch, HDFS-3004.022.patch, > HDFS-3004.023.patch, HDFS-3004.024.patch, HDFS-3004.026.patch, > HDFS-3004.027.patch, HDFS-3004.029.patch, HDFS-3004.030.patch, > HDFS-3004.031.patch, HDFS-3004.032.patch, HDFS-3004.033.patch, > HDFS-3004.034.patch, HDFS-3004.035.patch, HDFS-3004.036.patch, > HDFS-3004.037.patch, HDFS-3004.038.patch, HDFS-3004.039.patch, > HDFS-3004.040.patch, HDFS-3004.041.patch, HDFS-3004.042.patch, > HDFS-3004.042.patch, HDFS-3004.042.patch, HDFS-3004.043.patch, > HDFS-3004__namenode_recovery_tool.txt > > > When the NameNode metadata is corrupt for some reason, we want to be able to > fix it. Obviously, we would prefer never to get in this case. In a perfect > world, we never would. However, bad data on disk can happen from time to > time, because of hardware errors or misconfigurations. In the past we have > had to correct it manually, which is time-consuming and which can result in > downtime. > Recovery mode is initialized by the system administrator. When the NameNode > starts up in Recovery Mode, it will try to load the FSImage file, apply all > the edits from the edits log, and then write out a new image. Then it will > shut down. > Unlike in the normal startup process, the recovery mode startup process will > be interactive. When the NameNode finds something that is inconsistent, it > will prompt the operator as to what it should do. The operator can also > choose to take the first option for all prompts by starting up with the '-f' > flag, or typing 'a' at one of the prompts. > I have reused as much code as possible from the NameNode in this tool. > Hopefully, the effort that was spent developing this will also make the > NameNode editLog and image processing even more robust than it already is. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3004) Implement Recovery Mode
[ https://issues.apache.org/jira/browse/HDFS-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250110#comment-13250110 ] Hudson commented on HDFS-3004: -- Integrated in Hadoop-Hdfs-trunk-Commit #2105 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2105/]) HDFS-3004. Implement Recovery Mode. Contributed by Colin Patrick McCabe (Revision 1311394) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311394 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/docs/src/documentation/content/xdocs/hdfs_user_guide.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Checkpointer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageTransactionalStorageInspector.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileJournalManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/MetaRecoveryContext.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineEditsViewer/OfflineEditsXmlLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogRace.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeRecovery.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSecurityTokenEditLog.java > Implement Recovery Mode > --- > > Key: HDFS-3004 > URL: https://issues.apache.org/jira/browse/HDFS-3004 > Project: Hadoop HDFS > Issue Type: New Feature > Components: tools >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Fix For: 2.0.0 > > Attachments: HDFS-3004.010.patch, HDF
[jira] [Commented] (HDFS-3165) HDFS Balancer scripts are refering to wrong path of hadoop-daemon.sh
[ https://issues.apache.org/jira/browse/HDFS-3165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250111#comment-13250111 ] Hadoop QA commented on HDFS-3165: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521982/HDFS-3165_1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2231//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2231//console This message is automatically generated. > HDFS Balancer scripts are refering to wrong path of hadoop-daemon.sh > > > Key: HDFS-3165 > URL: https://issues.apache.org/jira/browse/HDFS-3165 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer >Affects Versions: 1.0.1 > Environment: HDFS Balancer >Reporter: amith >Priority: Minor > Attachments: HDFS-3165.patch, HDFS-3165.patch, HDFS-3165.patch, > HDFS-3165_1.patch > > Original Estimate: 1m > Remaining Estimate: 1m > > HDFS Balancer scripts are refering to wrong path of hadoop-daemon.sh > HOST-xx-xx-xx-xx:/home/amith/V1R2/namenode1/sbin # ./start-balancer.sh > ./start-balancer.sh: line 27: > /home/amith/V1R2/namenode1/bin/hadoop-daemon.sh: No such file or directory > currently hadoop-daemon.sh script is in sbin and not bin. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3004) Implement Recovery Mode
[ https://issues.apache.org/jira/browse/HDFS-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-3004: -- Resolution: Fixed Fix Version/s: 2.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I've committed this and merged to branch-2. Nice work Colin! > Implement Recovery Mode > --- > > Key: HDFS-3004 > URL: https://issues.apache.org/jira/browse/HDFS-3004 > Project: Hadoop HDFS > Issue Type: New Feature > Components: tools >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Fix For: 2.0.0 > > Attachments: HDFS-3004.010.patch, HDFS-3004.011.patch, > HDFS-3004.012.patch, HDFS-3004.013.patch, HDFS-3004.015.patch, > HDFS-3004.016.patch, HDFS-3004.017.patch, HDFS-3004.018.patch, > HDFS-3004.019.patch, HDFS-3004.020.patch, HDFS-3004.022.patch, > HDFS-3004.023.patch, HDFS-3004.024.patch, HDFS-3004.026.patch, > HDFS-3004.027.patch, HDFS-3004.029.patch, HDFS-3004.030.patch, > HDFS-3004.031.patch, HDFS-3004.032.patch, HDFS-3004.033.patch, > HDFS-3004.034.patch, HDFS-3004.035.patch, HDFS-3004.036.patch, > HDFS-3004.037.patch, HDFS-3004.038.patch, HDFS-3004.039.patch, > HDFS-3004.040.patch, HDFS-3004.041.patch, HDFS-3004.042.patch, > HDFS-3004.042.patch, HDFS-3004.042.patch, HDFS-3004.043.patch, > HDFS-3004__namenode_recovery_tool.txt > > > When the NameNode metadata is corrupt for some reason, we want to be able to > fix it. Obviously, we would prefer never to get in this case. In a perfect > world, we never would. However, bad data on disk can happen from time to > time, because of hardware errors or misconfigurations. In the past we have > had to correct it manually, which is time-consuming and which can result in > downtime. > Recovery mode is initialized by the system administrator. When the NameNode > starts up in Recovery Mode, it will try to load the FSImage file, apply all > the edits from the edits log, and then write out a new image. Then it will > shut down. > Unlike in the normal startup process, the recovery mode startup process will > be interactive. When the NameNode finds something that is inconsistent, it > will prompt the operator as to what it should do. The operator can also > choose to take the first option for all prompts by starting up with the '-f' > flag, or typing 'a' at one of the prompts. > I have reused as much code as possible from the NameNode in this tool. > Hopefully, the effort that was spent developing this will also make the > NameNode editLog and image processing even more robust than it already is. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3004) Implement Recovery Mode
[ https://issues.apache.org/jira/browse/HDFS-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250108#comment-13250108 ] Hudson commented on HDFS-3004: -- Integrated in Hadoop-Common-trunk-Commit #2030 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2030/]) HDFS-3004. Implement Recovery Mode. Contributed by Colin Patrick McCabe (Revision 1311394) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311394 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/dev-support/findbugsExcludeFile.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperEditLogInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogTestUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/docs/src/documentation/content/xdocs/hdfs_user_guide.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Checkpointer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogBackupInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogFileInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogInputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EditLogOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageTransactionalStorageInspector.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileJournalManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/MetaRecoveryContext.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineEditsViewer/OfflineEditsXmlLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogRace.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeRecovery.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSecurityTokenEditLog.java > Implement Recovery Mode > --- > > Key: HDFS-3004 > URL: https://issues.apache.org/jira/browse/HDFS-3004 > Project: Hadoop HDFS > Issue Type: New Feature > Components: tools >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-3004.010.patch, HDFS-3004.011.patch, > HDFS-3
[jira] [Commented] (HDFS-3004) Implement Recovery Mode
[ https://issues.apache.org/jira/browse/HDFS-3004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250104#comment-13250104 ] Eli Collins commented on HDFS-3004: --- +1 diff'd latest patch against the previous, addresses Nicholas' feedback. > Implement Recovery Mode > --- > > Key: HDFS-3004 > URL: https://issues.apache.org/jira/browse/HDFS-3004 > Project: Hadoop HDFS > Issue Type: New Feature > Components: tools >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-3004.010.patch, HDFS-3004.011.patch, > HDFS-3004.012.patch, HDFS-3004.013.patch, HDFS-3004.015.patch, > HDFS-3004.016.patch, HDFS-3004.017.patch, HDFS-3004.018.patch, > HDFS-3004.019.patch, HDFS-3004.020.patch, HDFS-3004.022.patch, > HDFS-3004.023.patch, HDFS-3004.024.patch, HDFS-3004.026.patch, > HDFS-3004.027.patch, HDFS-3004.029.patch, HDFS-3004.030.patch, > HDFS-3004.031.patch, HDFS-3004.032.patch, HDFS-3004.033.patch, > HDFS-3004.034.patch, HDFS-3004.035.patch, HDFS-3004.036.patch, > HDFS-3004.037.patch, HDFS-3004.038.patch, HDFS-3004.039.patch, > HDFS-3004.040.patch, HDFS-3004.041.patch, HDFS-3004.042.patch, > HDFS-3004.042.patch, HDFS-3004.042.patch, HDFS-3004.043.patch, > HDFS-3004__namenode_recovery_tool.txt > > > When the NameNode metadata is corrupt for some reason, we want to be able to > fix it. Obviously, we would prefer never to get in this case. In a perfect > world, we never would. However, bad data on disk can happen from time to > time, because of hardware errors or misconfigurations. In the past we have > had to correct it manually, which is time-consuming and which can result in > downtime. > Recovery mode is initialized by the system administrator. When the NameNode > starts up in Recovery Mode, it will try to load the FSImage file, apply all > the edits from the edits log, and then write out a new image. Then it will > shut down. > Unlike in the normal startup process, the recovery mode startup process will > be interactive. When the NameNode finds something that is inconsistent, it > will prompt the operator as to what it should do. The operator can also > choose to take the first option for all prompts by starting up with the '-f' > flag, or typing 'a' at one of the prompts. > I have reused as much code as possible from the NameNode in this tool. > Hopefully, the effort that was spent developing this will also make the > NameNode editLog and image processing even more robust than it already is. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
