[jira] [Resolved] (HDFS-2766) HA: test for case where standby partially reads log and then performs checkpoint
[ https://issues.apache.org/jira/browse/HDFS-2766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers resolved HDFS-2766. -- Resolution: Fixed Hadoop Flags: Reviewed Thanks a lot for the review, Todd. I've just committed this. HA: test for case where standby partially reads log and then performs checkpoint Key: HDFS-2766 URL: https://issues.apache.org/jira/browse/HDFS-2766 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, name-node Affects Versions: HA branch (HDFS-1623) Reporter: Todd Lipcon Assignee: Aaron T. Myers Attachments: HDFS-2766-HDFS-1623.patch, HDFS-2766-HDFS-1623.patch Here's a potential bug case that we don't currently test for: - SBN is reading a finalized edits file when NFS disappears halfway through (or some intermittent error happens) - SBN performs a checkpoint and uploads it to the NN - NN receives a checkpoint that doesn't correspond to the end of any log segment - Both NN and SBN should be able to restart at this point. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-2738) FSEditLog.selectinputStreams is reading through in-progress streams even when non-in-progress are requested
[
https://issues.apache.org/jira/browse/HDFS-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers resolved HDFS-2738.
--
Resolution: Fixed
Hadoop Flags: Reviewed
Thanks a lot for the review, Todd. I've just committed this.
FSEditLog.selectinputStreams is reading through in-progress streams even when
non-in-progress are requested
---
Key: HDFS-2738
URL: https://issues.apache.org/jira/browse/HDFS-2738
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ha, name-node
Affects Versions: HA branch (HDFS-1623)
Reporter: Todd Lipcon
Assignee: Aaron T. Myers
Priority: Blocker
Attachments: HDFS-2738-HDFS-1623.patch, HDFS-2738-HDFS-1623.patch,
HDFS-2738-HDFS-1623.patch
The new code in HDFS-1580 is causing an issue with selectInputStreams in the
HA context. When the active is writing to the shared edits,
selectInputStreams is called on the standby. This ends up calling
{{journalSet.getInputStream}} but doesn't pass the {{inProgressOk=false}}
flag. So, {{getInputStream}} ends up reading and validating the in-progress
stream unnecessarily. Since the validation results are no longer properly
cached, {{findMaxTransaction}} also re-validates the in-progress stream, and
then breaks the corruption check in this code. The end result is a lot of
errors like:
2011-12-30 16:45:02,521 ERROR namenode.FileJournalManager
(FileJournalManager.java:getNumberOfTransactions(266)) - Gap in transactions,
max txnid is 579, 0 txns from 578
2011-12-30 16:45:02,521 INFO ha.EditLogTailer (EditLogTailer.java:run(163))
- Got error, will try again.
java.io.IOException: No non-corrupt logs for txid 578
at
org.apache.hadoop.hdfs.server.namenode.JournalSet.getInputStream(JournalSet.java:229)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1081)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:115)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$0(EditLogTailer.java:100)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:154)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2739) SecondaryNameNode doesn't start up
[
https://issues.apache.org/jira/browse/HDFS-2739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13184020#comment-13184020
]
Hudson commented on HDFS-2739:
--
Integrated in Hadoop-Hdfs-trunk #922 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/922/])
HDFS-2739. SecondaryNameNode doesn't start up.
jitendra :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1229877
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolServerSideTranslatorPB.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/NamenodeProtocolTranslatorPB.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/NamenodeProtocol.proto
SecondaryNameNode doesn't start up
--
Key: HDFS-2739
URL: https://issues.apache.org/jira/browse/HDFS-2739
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 0.24.0
Reporter: Sho Shimauchi
Assignee: Jitendra Nath Pandey
Priority: Critical
Attachments: HDFS-2739.trunk.patch
Built a 0.24-SNAPSHOT tar from today, used a general config, started NN/DN,
but SNN won't come up with following error:
{code}
11/12/31 12:13:14 ERROR namenode.SecondaryNameNode: Throwable Exception in
doCheckpoint
java.lang.RuntimeException: java.lang.NoSuchFieldException: versionID
at org.apache.hadoop.ipc.RPC.getProtocolVersion(RPC.java:154)
at
org.apache.hadoop.ipc.WritableRpcEngine$Invocation.init(WritableRpcEngine.java:112)
at
org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:226)
at $Proxy9.getTransationId(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.getTransactionID(NamenodeProtocolTranslatorPB.java:185)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.countUncheckpointedTxns(SecondaryNameNode.java:625)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.shouldCheckpointBasedOnCount(SecondaryNameNode.java:633)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:386)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:356)
at java.lang.Thread.run(Thread.java:680)
Caused by: java.lang.NoSuchFieldException: versionID
at java.lang.Class.getField(Class.java:1520)
at org.apache.hadoop.ipc.RPC.getProtocolVersion(RPC.java:150)
... 9 more
java.lang.RuntimeException: java.lang.NoSuchFieldException: versionID
at org.apache.hadoop.ipc.RPC.getProtocolVersion(RPC.java:154)
at
org.apache.hadoop.ipc.WritableRpcEngine$Invocation.init(WritableRpcEngine.java:112)
at
org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:226)
at $Proxy9.getTransationId(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.getTransactionID(NamenodeProtocolTranslatorPB.java:185)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.countUncheckpointedTxns(SecondaryNameNode.java:625)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.shouldCheckpointBasedOnCount(SecondaryNameNode.java:633)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doWork(SecondaryNameNode.java:386)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:356)
at java.lang.Thread.run(Thread.java:680)
Caused by: java.lang.NoSuchFieldException: versionID
at java.lang.Class.getField(Class.java:1520)
at org.apache.hadoop.ipc.RPC.getProtocolVersion(RPC.java:150)
... 9 more
11/12/31 12:13:14 INFO namenode.SecondaryNameNode: SHUTDOWN_MSG:
/
SHUTDOWN_MSG: Shutting down SecondaryNameNode at sho-mba.local/192.168.11.2
/
{code}
full error log: http://pastebin.com/mSaVbS34
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2499) Fix RPC client creation bug from HDFS-2459
[ https://issues.apache.org/jira/browse/HDFS-2499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13184021#comment-13184021 ] Hudson commented on HDFS-2499: -- Integrated in Hadoop-Hdfs-trunk #922 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/922/]) Add HDFS-2499 to CHANGES.txt. szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1229897 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Fix RPC client creation bug from HDFS-2459 -- Key: HDFS-2499 URL: https://issues.apache.org/jira/browse/HDFS-2499 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Affects Versions: 0.23.0, 0.24.0 Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 0.24.0 Attachments: HDFS-2499.txt, HDFS-2499.txt HDFS-2459 incorrectly implemented the RPC getProxy for the JournalProtocol client side. It sets retry policies and other policies that are not necessary -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2773) HA: reading edit logs from an earlier version leaves blocks in under-construction state
[
https://issues.apache.org/jira/browse/HDFS-2773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13184035#comment-13184035
]
Hudson commented on HDFS-2773:
--
Integrated in Hadoop-Hdfs-HAbranch-build #44 (See
[https://builds.apache.org/job/Hadoop-Hdfs-HAbranch-build/44/])
HDFS-2773. Reading edit logs from an earlier version should not leave
blocks in under-construction state. Contributed by Todd Lipcon.
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1229900
Files :
*
/hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/CHANGES.HDFS-1623.txt
*
/hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java
*
/hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeFileUnderConstruction.java
*
/hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestPersistBlocks.java
*
/hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/hadoop-1.0-multiblock-file.tgz
HA: reading edit logs from an earlier version leaves blocks in
under-construction state
---
Key: HDFS-2773
URL: https://issues.apache.org/jira/browse/HDFS-2773
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ha, name-node
Affects Versions: HA branch (HDFS-1623)
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Blocker
Fix For: HA branch (HDFS-1623)
Attachments: hadoop-1.0-multiblock-file.tgz, hdfs-2773.txt
In HDFS-2602, the code for applying OP_ADD and OP_CLOSE was changed a bit,
and the new code has the following problem: if an OP_CLOSE includes new
blocks (ie not previously seen in an OP_ADD) then those blocks will remain in
the under construction state rather than being marked complete. This is
because {{updateBlocks}} always creates {{BlockInfoUnderConstruction}}
regardless of the opcode. This bug only affects the upgrade path, since in
trunk we always persist blocks with OP_ADDs before we call OP_CLOSE.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2753) Standby namenode stuck in safemode during a failover
[ https://issues.apache.org/jira/browse/HDFS-2753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13184034#comment-13184034 ] Hudson commented on HDFS-2753: -- Integrated in Hadoop-Hdfs-HAbranch-build #44 (See [https://builds.apache.org/job/Hadoop-Hdfs-HAbranch-build/44/]) HDFS-2753. Fix standby getting stuck in safemode when blocks are written while SBN is down. Contributed by Hari Mankude and Todd Lipcon. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1229898 Files : * /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/CHANGES.HDFS-1623.txt * /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java * /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHASafeMode.java Standby namenode stuck in safemode during a failover Key: HDFS-2753 URL: https://issues.apache.org/jira/browse/HDFS-2753 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, name-node Affects Versions: HA branch (HDFS-1623) Reporter: Hari Mankude Assignee: Hari Mankude Fix For: HA branch (HDFS-1623) Attachments: HDFS-2753.patch, hdfs-2753.txt, hdfs-2753.txt Write traffic initiated from the client. Manual failover is done by killing NN and converting a different standby to active. NN is restarted as standby. The restarted standby stays in safemode forever. More information in the description. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2775) HA: TestStandbyCheckpoints.testBothNodesInStandbyState fails intermittently
[
https://issues.apache.org/jira/browse/HDFS-2775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13184036#comment-13184036
]
Hudson commented on HDFS-2775:
--
Integrated in Hadoop-Hdfs-HAbranch-build #44 (See
[https://builds.apache.org/job/Hadoop-Hdfs-HAbranch-build/44/])
HDFS-2775. Fix TestStandbyCheckpoints.testBothNodesInStandbyState failing
intermittently. Contributed by Todd Lipcon.
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1229901
Files :
*
/hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/CHANGES.HDFS-1623.txt
*
/hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
*
/hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestStandbyCheckpoints.java
HA: TestStandbyCheckpoints.testBothNodesInStandbyState fails intermittently
---
Key: HDFS-2775
URL: https://issues.apache.org/jira/browse/HDFS-2775
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ha, test
Affects Versions: HA branch (HDFS-1623)
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Fix For: HA branch (HDFS-1623)
Attachments: hdfs-2775.txt
This test is failing periodically on this assertion:
{code}
assertEquals(12, nn0.getNamesystem().getFSImage().getStorage()
.getMostRecentCheckpointTxId());
{code}
My guess is it's a test race. Investigating...
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2738) FSEditLog.selectinputStreams is reading through in-progress streams even when non-in-progress are requested
[
https://issues.apache.org/jira/browse/HDFS-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13184037#comment-13184037
]
Hudson commented on HDFS-2738:
--
Integrated in Hadoop-Hdfs-HAbranch-build #44 (See
[https://builds.apache.org/job/Hadoop-Hdfs-HAbranch-build/44/])
HDFS-2738. FSEditLog.selectinputStreams is reading through in-progress
streams even when non-in-progress are requested. Contributed by Aaron T. Myers
atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1229931
Files :
*
/hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/CHANGES.HDFS-1623.txt
*
/hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperJournalManager.java
*
/hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperJournalManager.java
*
/hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupJournalManager.java
*
/hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java
*
/hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
*
/hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileJournalManager.java
*
/hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalManager.java
*
/hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/JournalSet.java
*
/hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java
*
/hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileJournalManager.java
*
/hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestGenericJournalConf.java
FSEditLog.selectinputStreams is reading through in-progress streams even when
non-in-progress are requested
---
Key: HDFS-2738
URL: https://issues.apache.org/jira/browse/HDFS-2738
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ha, name-node
Affects Versions: HA branch (HDFS-1623)
Reporter: Todd Lipcon
Assignee: Aaron T. Myers
Priority: Blocker
Attachments: HDFS-2738-HDFS-1623.patch, HDFS-2738-HDFS-1623.patch,
HDFS-2738-HDFS-1623.patch
The new code in HDFS-1580 is causing an issue with selectInputStreams in the
HA context. When the active is writing to the shared edits,
selectInputStreams is called on the standby. This ends up calling
{{journalSet.getInputStream}} but doesn't pass the {{inProgressOk=false}}
flag. So, {{getInputStream}} ends up reading and validating the in-progress
stream unnecessarily. Since the validation results are no longer properly
cached, {{findMaxTransaction}} also re-validates the in-progress stream, and
then breaks the corruption check in this code. The end result is a lot of
errors like:
2011-12-30 16:45:02,521 ERROR namenode.FileJournalManager
(FileJournalManager.java:getNumberOfTransactions(266)) - Gap in transactions,
max txnid is 579, 0 txns from 578
2011-12-30 16:45:02,521 INFO ha.EditLogTailer (EditLogTailer.java:run(163))
- Got error, will try again.
java.io.IOException: No non-corrupt logs for txid 578
at
org.apache.hadoop.hdfs.server.namenode.JournalSet.getInputStream(JournalSet.java:229)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1081)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:115)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.access$0(EditLogTailer.java:100)
at
org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:154)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2499) Fix RPC client creation bug from HDFS-2459
[ https://issues.apache.org/jira/browse/HDFS-2499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13184062#comment-13184062 ] Hudson commented on HDFS-2499: -- Integrated in Hadoop-Mapreduce-trunk #955 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/955/]) Add HDFS-2499 to CHANGES.txt. szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1229897 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Fix RPC client creation bug from HDFS-2459 -- Key: HDFS-2499 URL: https://issues.apache.org/jira/browse/HDFS-2499 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Affects Versions: 0.23.0, 0.24.0 Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 0.24.0 Attachments: HDFS-2499.txt, HDFS-2499.txt HDFS-2459 incorrectly implemented the RPC getProxy for the JournalProtocol client side. It sets retry policies and other policies that are not necessary -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2778) HDFS Federation Documentation has incorrect configuration example
HDFS Federation Documentation has incorrect configuration example - Key: HDFS-2778 URL: https://issues.apache.org/jira/browse/HDFS-2778 Project: Hadoop HDFS Issue Type: Bug Components: documentation Affects Versions: 0.23.0 Reporter: praveen sripati Priority: Minor HDFS Federation documentation example (1) has the following property namedfs.namenode.rpc-address.ns1/name valuehdfs://nn-host1:rpc-port/value /property dfs.namenode.rpc-address.* should be set to hostname:port, hdfs:// should not be there. (1) - http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/Federation.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2183) [viewfs .next] viewfs not showing proper permission of top level directory in hdfs --config viewfs_conf dfs -ls /
[ https://issues.apache.org/jira/browse/HDFS-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13184194#comment-13184194 ] Dave Thompson commented on HDFS-2183: - This may have been fixed in trunk as of 1/10/12 [viewfs .next] viewfs not showing proper permission of top level directory in hdfs --config viewfs_conf dfs -ls / - Key: HDFS-2183 URL: https://issues.apache.org/jira/browse/HDFS-2183 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Sanjay Radia -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2767) HA: ConfiguredFailoverProxyProvider should support NameNodeProtocol
[ https://issues.apache.org/jira/browse/HDFS-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-2767: -- Attachment: HDFS-2767.patch HA: ConfiguredFailoverProxyProvider should support NameNodeProtocol --- Key: HDFS-2767 URL: https://issues.apache.org/jira/browse/HDFS-2767 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, hdfs client Affects Versions: HA branch (HDFS-1623) Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G Priority: Blocker Attachments: HDFS-2767.patch, HDFS-2767.patch, hdfs-2767-what-todd-had.txt Presentely ConfiguredFailoverProxyProvider supports ClinetProtocol. It should support NameNodeProtocol also, because Balancer uses NameNodeProtocol for getting blocks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2767) HA: ConfiguredFailoverProxyProvider should support NameNodeProtocol
[ https://issues.apache.org/jira/browse/HDFS-2767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13184229#comment-13184229 ] Uma Maheswara Rao G commented on HDFS-2767: --- Todd, updated the patch by addressing your comments. HA: ConfiguredFailoverProxyProvider should support NameNodeProtocol --- Key: HDFS-2767 URL: https://issues.apache.org/jira/browse/HDFS-2767 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, hdfs client Affects Versions: HA branch (HDFS-1623) Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G Priority: Blocker Attachments: HDFS-2767.patch, HDFS-2767.patch, hdfs-2767-what-todd-had.txt Presentely ConfiguredFailoverProxyProvider supports ClinetProtocol. It should support NameNodeProtocol also, because Balancer uses NameNodeProtocol for getting blocks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2592) HA: Balancer support for HA namenodes
[ https://issues.apache.org/jira/browse/HDFS-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-2592: -- Attachment: HDFS-2592.patch HA: Balancer support for HA namenodes - Key: HDFS-2592 URL: https://issues.apache.org/jira/browse/HDFS-2592 Project: Hadoop HDFS Issue Type: Sub-task Components: balancer, ha Affects Versions: HA branch (HDFS-1623) Reporter: Todd Lipcon Assignee: Uma Maheswara Rao G Attachments: HDFS-2592.patch, HDFS-2592.patch, HDFS-2592.patch The balancer currently interacts directly with namenode InetSocketAddresses and makes its own IPC proxies. We need to integrate it with HA so that it uses the same client failover infrastructure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2592) HA: Balancer support for HA namenodes
[ https://issues.apache.org/jira/browse/HDFS-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13184236#comment-13184236 ] Uma Maheswara Rao G commented on HDFS-2592: --- Updated the patch reflecting to HDFS-2767 HA: Balancer support for HA namenodes - Key: HDFS-2592 URL: https://issues.apache.org/jira/browse/HDFS-2592 Project: Hadoop HDFS Issue Type: Sub-task Components: balancer, ha Affects Versions: HA branch (HDFS-1623) Reporter: Todd Lipcon Assignee: Uma Maheswara Rao G Attachments: HDFS-2592.patch, HDFS-2592.patch, HDFS-2592.patch The balancer currently interacts directly with namenode InetSocketAddresses and makes its own IPC proxies. We need to integrate it with HA so that it uses the same client failover infrastructure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2778) HDFS Federation Documentation has incorrect configuration example
[ https://issues.apache.org/jira/browse/HDFS-2778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-2778: - Target Version/s: 0.24.0, 0.23.1 (was: 0.23.1, 0.24.0) Affects Version/s: 0.24.0 HDFS Federation Documentation has incorrect configuration example - Key: HDFS-2778 URL: https://issues.apache.org/jira/browse/HDFS-2778 Project: Hadoop HDFS Issue Type: Bug Components: documentation Affects Versions: 0.23.0, 0.24.0 Reporter: praveen sripati Assignee: Brandon Li Priority: Minor HDFS Federation documentation example (1) has the following property namedfs.namenode.rpc-address.ns1/name valuehdfs://nn-host1:rpc-port/value /property dfs.namenode.rpc-address.* should be set to hostname:port, hdfs:// should not be there. (1) - http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/Federation.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2776) Missing interface annotation on JournalSet
[ https://issues.apache.org/jira/browse/HDFS-2776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-2776: - Attachment: HDFS-2776.txt Add annotation on JournalSet Missing interface annotation on JournalSet -- Key: HDFS-2776 URL: https://issues.apache.org/jira/browse/HDFS-2776 Project: Hadoop HDFS Issue Type: Task Components: name-node Affects Versions: 0.24.0 Reporter: Todd Lipcon Assignee: Brandon Li Priority: Trivial Labels: newbie Attachments: HDFS-2776.txt This public class is missing an annotation that it is for private usage. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2776) Missing interface annotation on JournalSet
[ https://issues.apache.org/jira/browse/HDFS-2776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-2776: - Fix Version/s: 0.24.0 Labels: (was: newbie) Status: Patch Available (was: Open) Missing interface annotation on JournalSet -- Key: HDFS-2776 URL: https://issues.apache.org/jira/browse/HDFS-2776 Project: Hadoop HDFS Issue Type: Task Components: name-node Affects Versions: 0.24.0 Reporter: Todd Lipcon Assignee: Brandon Li Priority: Trivial Fix For: 0.24.0 Attachments: HDFS-2776.txt This public class is missing an annotation that it is for private usage. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2768) BackupNode stop can not close proxy connections because it is not a proxy instance.
[ https://issues.apache.org/jira/browse/HDFS-2768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13184283#comment-13184283 ] Uma Maheswara Rao G commented on HDFS-2768: --- Would someone please take a look on this patch.? BackupNode stop can not close proxy connections because it is not a proxy instance. --- Key: HDFS-2768 URL: https://issues.apache.org/jira/browse/HDFS-2768 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.24.0 Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G Attachments: HDFS-2768.patch Observe this from BackupNode tests: java.lang.IllegalArgumentException: not a proxy instance at java.lang.reflect.Proxy.getInvocationHandler(Unknown Source) at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:557) at org.apache.hadoop.hdfs.server.namenode.BackupNode.stop(BackupNode.java:194) at org.apache.hadoop.hdfs.server.namenode.TestBackupNode.testCheckpoint(TestBackupNode.java:355) at org.apache.hadoop.hdfs.server.namenode.TestBackupNode.testBackupNode(TestBackupNode.java:241) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2725) hdfs script usage information is missing the information about dfs command
[ https://issues.apache.org/jira/browse/HDFS-2725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13184295#comment-13184295 ] Koji Noguchi commented on HDFS-2725: Can we have this in 0.23 as well? hdfs script usage information is missing the information about dfs command Key: HDFS-2725 URL: https://issues.apache.org/jira/browse/HDFS-2725 Project: Hadoop HDFS Issue Type: Bug Components: hdfs client Affects Versions: 0.24.0 Reporter: Prashant Sharma Labels: hdfs Fix For: 0.24.0 Attachments: HDFS-2725.patch hdfs script does print the command dfs in the usage. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2776) Missing interface annotation on JournalSet
[ https://issues.apache.org/jira/browse/HDFS-2776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13184332#comment-13184332 ] Hadoop QA commented on HDFS-2776: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12510222/HDFS-2776.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated 21 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings). -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.server.datanode.TestMulitipleNNDataBlockScanner +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1772//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1772//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1772//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1772//console This message is automatically generated. Missing interface annotation on JournalSet -- Key: HDFS-2776 URL: https://issues.apache.org/jira/browse/HDFS-2776 Project: Hadoop HDFS Issue Type: Task Components: name-node Affects Versions: 0.24.0 Reporter: Todd Lipcon Assignee: Brandon Li Priority: Trivial Fix For: 0.24.0 Attachments: HDFS-2776.txt This public class is missing an annotation that it is for private usage. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2779) HA: Add lease recovery handling to HA
HA: Add lease recovery handling to HA - Key: HDFS-2779 URL: https://issues.apache.org/jira/browse/HDFS-2779 Project: Hadoop HDFS Issue Type: Sub-task Components: ha Affects Versions: HA branch (HDFS-1623) Reporter: Suresh Srinivas Fix For: HA branch (HDFS-1623) Lease recovery needs to be handled in HA setup -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-2779) HA: Add lease recovery handling to HA
[ https://issues.apache.org/jira/browse/HDFS-2779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas reassigned HDFS-2779: - Assignee: Suresh Srinivas HA: Add lease recovery handling to HA - Key: HDFS-2779 URL: https://issues.apache.org/jira/browse/HDFS-2779 Project: Hadoop HDFS Issue Type: Sub-task Components: ha Affects Versions: HA branch (HDFS-1623) Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: HA branch (HDFS-1623) Lease recovery needs to be handled in HA setup -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2776) Missing interface annotation on JournalSet
[ https://issues.apache.org/jira/browse/HDFS-2776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13184429#comment-13184429 ] Brandon Li commented on HDFS-2776: -- The warning messages generated by javadoc tool and release audit are unrelated with this patch. The test failure is also unrelated with the patch. Missing interface annotation on JournalSet -- Key: HDFS-2776 URL: https://issues.apache.org/jira/browse/HDFS-2776 Project: Hadoop HDFS Issue Type: Task Components: name-node Affects Versions: 0.24.0 Reporter: Todd Lipcon Assignee: Brandon Li Priority: Trivial Fix For: 0.24.0 Attachments: HDFS-2776.txt This public class is missing an annotation that it is for private usage. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2776) Missing interface annotation on JournalSet
[ https://issues.apache.org/jira/browse/HDFS-2776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13184430#comment-13184430 ] Jitendra Nath Pandey commented on HDFS-2776: +1 Missing interface annotation on JournalSet -- Key: HDFS-2776 URL: https://issues.apache.org/jira/browse/HDFS-2776 Project: Hadoop HDFS Issue Type: Task Components: name-node Affects Versions: 0.24.0 Reporter: Todd Lipcon Assignee: Brandon Li Priority: Trivial Fix For: 0.24.0 Attachments: HDFS-2776.txt This public class is missing an annotation that it is for private usage. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-69) Improve dfsadmin command line help
[ https://issues.apache.org/jira/browse/HDFS-69?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13184460#comment-13184460 ] Jakob Homan commented on HDFS-69: - +1 Improve dfsadmin command line help --- Key: HDFS-69 URL: https://issues.apache.org/jira/browse/HDFS-69 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 1.0.0 Reporter: Ravi Phulari Assignee: Harsh J Priority: Minor Attachments: HDFS-69.patch Enhance dfsadmin command line help informing A quota of one forces a directory to remain empty -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2780) All HAServiceProtocol methods have to throw IOException since they are invoked over the network
All HAServiceProtocol methods have to throw IOException since they are invoked over the network --- Key: HDFS-2780 URL: https://issues.apache.org/jira/browse/HDFS-2780 Project: Hadoop HDFS Issue Type: Bug Components: ha Affects Versions: HA branch (HDFS-1623) Reporter: Hari Mankude Assignee: Hari Mankude HAServiceProtocol methods have to throw ioexception in addition to other exceptions since they are RPCs which are invoked over the network. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2780) All HAServiceProtocol methods have to throw IOException since they are invoked over the network
[
https://issues.apache.org/jira/browse/HDFS-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13184473#comment-13184473
]
Aaron T. Myers commented on HDFS-2780:
--
+1, I bumped into this problem when working on HDFS-2772. Thanks a lot for
filing this, Hari.
Given that {{HAServiceProtocol}} is actually under the hadoop-common project,
perhaps this JIRA should be moved there as a sub-task of HADOOP-7454?
All HAServiceProtocol methods have to throw IOException since they are
invoked over the network
---
Key: HDFS-2780
URL: https://issues.apache.org/jira/browse/HDFS-2780
Project: Hadoop HDFS
Issue Type: Bug
Components: ha
Affects Versions: HA branch (HDFS-1623)
Reporter: Hari Mankude
Assignee: Hari Mankude
HAServiceProtocol methods have to throw ioexception in addition to other
exceptions since they are RPCs which are invoked over the network.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2757) Cannot read a local block that's being written to when using the local read short circuit
[
https://issues.apache.org/jira/browse/HDFS-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jean-Daniel Cryans updated HDFS-2757:
-
Summary: Cannot read a local block that's being written to when using the
local read short circuit (was: Cannot read a local file that's being written
to when using the local read short circuit)
Suresh, I changed the title to reflect what I think is really the issue, and
yes this is a bad bug.
Cannot read a local block that's being written to when using the local read
short circuit
-
Key: HDFS-2757
URL: https://issues.apache.org/jira/browse/HDFS-2757
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 1.0.0
Reporter: Jean-Daniel Cryans
Fix For: 1.1.0
When testing the tail'ing of a local file with the read short circuit on, I
get:
{noformat}
2012-01-06 00:17:31,598 WARN org.apache.hadoop.hdfs.DFSClient:
BlockReaderLocal requested with incorrect offset: Offset 0 and length
8230400 don't match block blk_-2842916025951313698_454072 ( blockLen 124 )
2012-01-06 00:17:31,598 WARN org.apache.hadoop.hdfs.DFSClient:
BlockReaderLocal: Removing blk_-2842916025951313698_454072 from cache because
local file
/export4/jdcryans/dfs/data/blocksBeingWritten/blk_-2842916025951313698 could
not be opened.
2012-01-06 00:17:31,599 INFO org.apache.hadoop.hdfs.DFSClient: Failed to read
block blk_-2842916025951313698_454072 on local machine java.io.IOException:
Offset 0 and length 8230400 don't match block blk_-2842916025951313698_454072
( blockLen 124 )
2012-01-06 00:17:31,599 INFO org.apache.hadoop.hdfs.DFSClient: Try reading
via the datanode on /10.4.13.38:51010
java.io.EOFException:
hdfs://sv4r11s38:9100/hbase-1/.logs/sv4r13s38,62023,1325808100311/sv4r13s38%2C62023%2C1325808100311.1325808100818,
entryStart=7190409, pos=8230400, end=8230400, edit=5
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2781) Add client protocol and DFSadmin for command to restore failed storage
Add client protocol and DFSadmin for command to restore failed storage -- Key: HDFS-2781 URL: https://issues.apache.org/jira/browse/HDFS-2781 Project: Hadoop HDFS Issue Type: Sub-task Components: ha Affects Versions: HA branch (HDFS-1623) Reporter: Eli Collins Assignee: Eli Collins Per HDFS-2769, it's important that an admin be able to ask the NN to try to restore failed storage since we may drop into SM until the shared edits dir is restored (w/o having to wait for the next checkpoint). There's currently an API (and usage in DFSAdmin) to flip the flag indicating whether the NN should try to restore failed storage but not that it should actually attempt to do so. This jira is to add one. This is useful outside HA but doing as an HDFS-1623 sub-task since it's motivated by HA. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2782) HA: Support multiple shared edits dirs
HA: Support multiple shared edits dirs -- Key: HDFS-2782 URL: https://issues.apache.org/jira/browse/HDFS-2782 Project: Hadoop HDFS Issue Type: Sub-task Components: ha Affects Versions: HA branch (HDFS-1623) Reporter: Eli Collins Assignee: Eli Collins Supporting multiple shared dirs will improve availability (eg see HDFS-2769). You may want to use multiple shared dirs on a single filer (eg for better fault isolation) or because you want to use multiple filers/mounts. Per HDFS-2752 (and HDFS-2735) we need to do things like use the JournalSet in EditLogTailer and add tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2769) HA: When HA is enabled with a shared edits dir, that dir should be marked required
[ https://issues.apache.org/jira/browse/HDFS-2769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13184523#comment-13184523 ] Eli Collins commented on HDFS-2769: --- ATM and I sync up offline, discussed a couple of options: #1 Allow the NN to stay available when a shared dir fails. We could do this by having the active extract a promise from the standby(s) that they won't become active until they hear from the active telling them it's OK to do so (eg when the shared dir is restored they could accept a failover). This improves (write) availability in the case where you have a single shared dir (or single filer). #2 Have the NN drop into SM when the shared dir fails, we have the option of failing over to the standby if it can reach the shared dir (ie the failure is the connection between the primary and the shared dir), or at least the primary can still service reads, and we can add an API (HDFS-2781) an admin can use to make the primary immediately exit SM as soon as the shared dir has been restored. We think #2 is simpler, and we think should be OK for v1 since we'll be adding support for multiple shared dirs (HDFS-2782), and people who care will deploy with redundant connections to a filer with redundant parts. HA: When HA is enabled with a shared edits dir, that dir should be marked required -- Key: HDFS-2769 URL: https://issues.apache.org/jira/browse/HDFS-2769 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, name-node Affects Versions: HA branch (HDFS-1623) Reporter: Aaron T. Myers Assignee: Aaron T. Myers HDFS-2430 introduced the concept of required edits/name dirs, which are not treated as redundant when considering NN storage failure. When a shared edits dir is used, this should automatically be marked as required. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2778) HDFS Federation Documentation has incorrect configuration example
[ https://issues.apache.org/jira/browse/HDFS-2778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-2778: - Attachment: HDFS-2778.txt remove hdfs:// from the value field for dfs.namenode.rpc-address.* The change in generated html file was verified by using mvn site under hadoop-mapreduce-project/hadoop-yarn/ HDFS Federation Documentation has incorrect configuration example - Key: HDFS-2778 URL: https://issues.apache.org/jira/browse/HDFS-2778 Project: Hadoop HDFS Issue Type: Bug Components: documentation Affects Versions: 0.23.0, 0.24.0 Reporter: praveen sripati Assignee: Brandon Li Priority: Minor Attachments: HDFS-2778.txt HDFS Federation documentation example (1) has the following property namedfs.namenode.rpc-address.ns1/name valuehdfs://nn-host1:rpc-port/value /property dfs.namenode.rpc-address.* should be set to hostname:port, hdfs:// should not be there. (1) - http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/Federation.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2783) HDFS jira for changes related to HADOOP-7970
HDFS jira for changes related to HADOOP-7970 Key: HDFS-2783 URL: https://issues.apache.org/jira/browse/HDFS-2783 Project: Hadoop HDFS Issue Type: Bug Components: ha Affects Versions: HA branch (HDFS-1623) Reporter: Hari Mankude Assignee: Hari Mankude Priority: Minor Create a separate hdfs patch for HDFS changes related to Hadoop-7970 to avoid test failures. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2783) HDFS jira for changes related to HADOOP-7970
[ https://issues.apache.org/jira/browse/HDFS-2783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Mankude updated HDFS-2783: --- Attachment: hdfs-2783.txt HDFS jira for changes related to HADOOP-7970 Key: HDFS-2783 URL: https://issues.apache.org/jira/browse/HDFS-2783 Project: Hadoop HDFS Issue Type: Bug Components: ha Affects Versions: HA branch (HDFS-1623) Reporter: Hari Mankude Assignee: Hari Mankude Priority: Minor Attachments: hdfs-2783.txt Create a separate hdfs patch for HDFS changes related to Hadoop-7970 to avoid test failures. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2783) HDFS jira for changes related to HADOOP-7970
[ https://issues.apache.org/jira/browse/HDFS-2783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Mankude updated HDFS-2783: --- Status: Patch Available (was: Open) HDFS jira for changes related to HADOOP-7970 Key: HDFS-2783 URL: https://issues.apache.org/jira/browse/HDFS-2783 Project: Hadoop HDFS Issue Type: Bug Components: ha Affects Versions: HA branch (HDFS-1623) Reporter: Hari Mankude Assignee: Hari Mankude Priority: Minor Attachments: hdfs-2783.txt Create a separate hdfs patch for HDFS changes related to Hadoop-7970 to avoid test failures. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2783) HDFS jira for changes related to HADOOP-7970
[ https://issues.apache.org/jira/browse/HDFS-2783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13184585#comment-13184585 ] Hadoop QA commented on HDFS-2783: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12510281/hdfs-2783.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1773//console This message is automatically generated. HDFS jira for changes related to HADOOP-7970 Key: HDFS-2783 URL: https://issues.apache.org/jira/browse/HDFS-2783 Project: Hadoop HDFS Issue Type: Bug Components: ha Affects Versions: HA branch (HDFS-1623) Reporter: Hari Mankude Assignee: Hari Mankude Priority: Minor Attachments: hdfs-2783.txt Create a separate hdfs patch for HDFS changes related to Hadoop-7970 to avoid test failures. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2778) HDFS Federation Documentation has incorrect configuration example
[ https://issues.apache.org/jira/browse/HDFS-2778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-2778: - Attachment: HDFS-2778.txt It's the same as the previous patch. Attached again just to grant license. HDFS Federation Documentation has incorrect configuration example - Key: HDFS-2778 URL: https://issues.apache.org/jira/browse/HDFS-2778 Project: Hadoop HDFS Issue Type: Bug Components: documentation Affects Versions: 0.23.0, 0.24.0 Reporter: praveen sripati Assignee: Brandon Li Priority: Minor Attachments: HDFS-2778.txt, HDFS-2778.txt HDFS Federation documentation example (1) has the following property namedfs.namenode.rpc-address.ns1/name valuehdfs://nn-host1:rpc-port/value /property dfs.namenode.rpc-address.* should be set to hostname:port, hdfs:// should not be there. (1) - http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/Federation.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2783) HDFS jira for changes related to HADOOP-7970
[ https://issues.apache.org/jira/browse/HDFS-2783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Mankude updated HDFS-2783: --- Attachment: hdfs-2783.txt Attach patch with right path HDFS jira for changes related to HADOOP-7970 Key: HDFS-2783 URL: https://issues.apache.org/jira/browse/HDFS-2783 Project: Hadoop HDFS Issue Type: Bug Components: ha Affects Versions: HA branch (HDFS-1623) Reporter: Hari Mankude Assignee: Hari Mankude Priority: Minor Attachments: hdfs-2783.txt, hdfs-2783.txt Create a separate hdfs patch for HDFS changes related to Hadoop-7970 to avoid test failures. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2783) HDFS jira for changes related to HADOOP-7970
[ https://issues.apache.org/jira/browse/HDFS-2783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13184622#comment-13184622 ] Hadoop QA commented on HDFS-2783: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12510285/hdfs-2783.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1774//console This message is automatically generated. HDFS jira for changes related to HADOOP-7970 Key: HDFS-2783 URL: https://issues.apache.org/jira/browse/HDFS-2783 Project: Hadoop HDFS Issue Type: Bug Components: ha Affects Versions: HA branch (HDFS-1623) Reporter: Hari Mankude Assignee: Hari Mankude Priority: Minor Attachments: hdfs-2783.txt, hdfs-2783.txt Create a separate hdfs patch for HDFS changes related to Hadoop-7970 to avoid test failures. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2783) HDFS jira for changes related to HADOOP-7970
[ https://issues.apache.org/jira/browse/HDFS-2783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Mankude updated HDFS-2783: --- Resolution: Invalid Status: Resolved (was: Patch Available) hadoop-7970 has the complete fix. HDFS jira for changes related to HADOOP-7970 Key: HDFS-2783 URL: https://issues.apache.org/jira/browse/HDFS-2783 Project: Hadoop HDFS Issue Type: Bug Components: ha Affects Versions: HA branch (HDFS-1623) Reporter: Hari Mankude Assignee: Hari Mankude Priority: Minor Attachments: hdfs-2783.txt, hdfs-2783.txt Create a separate hdfs patch for HDFS changes related to Hadoop-7970 to avoid test failures. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2776) Missing interface annotation on JournalSet
[ https://issues.apache.org/jira/browse/HDFS-2776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13184670#comment-13184670 ] Hadoop QA commented on HDFS-2776: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12510284/HDFS-2776.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated 21 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings). +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1775//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1775//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1775//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1775//console This message is automatically generated. Missing interface annotation on JournalSet -- Key: HDFS-2776 URL: https://issues.apache.org/jira/browse/HDFS-2776 Project: Hadoop HDFS Issue Type: Task Components: name-node Affects Versions: 0.24.0 Reporter: Todd Lipcon Assignee: Brandon Li Priority: Trivial Fix For: 0.24.0 Attachments: HDFS-2776.txt, HDFS-2776.txt This public class is missing an annotation that it is for private usage. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-69) Improve dfsadmin command line help
[ https://issues.apache.org/jira/browse/HDFS-69?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13184722#comment-13184722 ] Harsh J commented on HDFS-69: - Thanks for reviewing Jakob. Committing shortly. Improve dfsadmin command line help --- Key: HDFS-69 URL: https://issues.apache.org/jira/browse/HDFS-69 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 1.0.0 Reporter: Ravi Phulari Assignee: Harsh J Priority: Minor Attachments: HDFS-69.patch Enhance dfsadmin command line help informing A quota of one forces a directory to remain empty -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-69) Improve dfsadmin command line help
[ https://issues.apache.org/jira/browse/HDFS-69?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13184735#comment-13184735 ] Hudson commented on HDFS-69: Integrated in Hadoop-Hdfs-trunk-Commit #1608 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1608/]) HDFS-69. Improve the 'dfsadmin' commandline help. (harsh) harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1230398 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java Improve dfsadmin command line help --- Key: HDFS-69 URL: https://issues.apache.org/jira/browse/HDFS-69 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 1.0.0 Reporter: Ravi Phulari Assignee: Harsh J Priority: Minor Attachments: HDFS-69.patch Enhance dfsadmin command line help informing A quota of one forces a directory to remain empty -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2725) hdfs script usage information is missing the information about dfs command
[ https://issues.apache.org/jira/browse/HDFS-2725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated HDFS-2725: -- Description: hdfs script does not print the command dfs in the usage. (was: hdfs script does print the command dfs in the usage.) Target Version/s: 0.23.0, 0.24.0 (was: 0.24.0) Affects Version/s: 0.23.0 Fix Version/s: 0.23.0 hdfs script usage information is missing the information about dfs command Key: HDFS-2725 URL: https://issues.apache.org/jira/browse/HDFS-2725 Project: Hadoop HDFS Issue Type: Bug Components: hdfs client Affects Versions: 0.23.0, 0.24.0 Reporter: Prashant Sharma Labels: hdfs Fix For: 0.23.0, 0.24.0 Attachments: HDFS-2725.patch hdfs script does not print the command dfs in the usage. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2725) hdfs script usage information is missing the information about dfs command
[ https://issues.apache.org/jira/browse/HDFS-2725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated HDFS-2725: -- Target Version/s: 0.23.0, 0.24.0 (was: 0.24.0, 0.23.0) Status: Open (was: Patch Available) hdfs script usage information is missing the information about dfs command Key: HDFS-2725 URL: https://issues.apache.org/jira/browse/HDFS-2725 Project: Hadoop HDFS Issue Type: Bug Components: hdfs client Affects Versions: 0.23.0, 0.24.0 Reporter: Prashant Sharma Labels: hdfs Fix For: 0.24.0, 0.23.0 Attachments: HDFS-2725.patch hdfs script does not print the command dfs in the usage. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-69) Improve dfsadmin command line help
[ https://issues.apache.org/jira/browse/HDFS-69?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13184738#comment-13184738 ] Hudson commented on HDFS-69: Integrated in Hadoop-Common-trunk-Commit #1535 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1535/]) HDFS-69. Improve the 'dfsadmin' commandline help. (harsh) harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1230398 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java Improve dfsadmin command line help --- Key: HDFS-69 URL: https://issues.apache.org/jira/browse/HDFS-69 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 1.0.0 Reporter: Ravi Phulari Assignee: Harsh J Priority: Minor Attachments: HDFS-69.patch Enhance dfsadmin command line help informing A quota of one forces a directory to remain empty -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2725) hdfs script usage information is missing the information about dfs command
[ https://issues.apache.org/jira/browse/HDFS-2725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Sharma updated HDFS-2725: -- Target Version/s: 0.23.0, 0.24.0 (was: 0.24.0, 0.23.0) Status: Patch Available (was: Open) hdfs script usage information is missing the information about dfs command Key: HDFS-2725 URL: https://issues.apache.org/jira/browse/HDFS-2725 Project: Hadoop HDFS Issue Type: Bug Components: hdfs client Affects Versions: 0.23.0, 0.24.0 Reporter: Prashant Sharma Labels: hdfs Fix For: 0.24.0, 0.23.0 Attachments: HDFS-2725.patch hdfs script does not print the command dfs in the usage. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-69) Improve dfsadmin command line help
[ https://issues.apache.org/jira/browse/HDFS-69?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13184744#comment-13184744 ] Hudson commented on HDFS-69: Integrated in Hadoop-Mapreduce-trunk-Commit #1553 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1553/]) HDFS-69. Improve the 'dfsadmin' commandline help. (harsh) harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1230398 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java Improve dfsadmin command line help --- Key: HDFS-69 URL: https://issues.apache.org/jira/browse/HDFS-69 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 1.0.0 Reporter: Ravi Phulari Assignee: Harsh J Priority: Minor Attachments: HDFS-69.patch Enhance dfsadmin command line help informing A quota of one forces a directory to remain empty -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2776) Missing interface annotation on JournalSet
[ https://issues.apache.org/jira/browse/HDFS-2776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDFS-2776: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed. Thanks to Brandon! Missing interface annotation on JournalSet -- Key: HDFS-2776 URL: https://issues.apache.org/jira/browse/HDFS-2776 Project: Hadoop HDFS Issue Type: Task Components: name-node Affects Versions: 0.24.0 Reporter: Todd Lipcon Assignee: Brandon Li Priority: Trivial Fix For: 0.24.0 Attachments: HDFS-2776.txt, HDFS-2776.txt This public class is missing an annotation that it is for private usage. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2778) HDFS Federation Documentation has incorrect configuration example
[ https://issues.apache.org/jira/browse/HDFS-2778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13184753#comment-13184753 ] Jitendra Nath Pandey commented on HDFS-2778: +1. HDFS Federation Documentation has incorrect configuration example - Key: HDFS-2778 URL: https://issues.apache.org/jira/browse/HDFS-2778 Project: Hadoop HDFS Issue Type: Bug Components: documentation Affects Versions: 0.23.0, 0.24.0 Reporter: praveen sripati Assignee: Brandon Li Priority: Minor Attachments: HDFS-2778.txt, HDFS-2778.txt HDFS Federation documentation example (1) has the following property namedfs.namenode.rpc-address.ns1/name valuehdfs://nn-host1:rpc-port/value /property dfs.namenode.rpc-address.* should be set to hostname:port, hdfs:// should not be there. (1) - http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/Federation.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-69) Improve dfsadmin command line help
[ https://issues.apache.org/jira/browse/HDFS-69?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated HDFS-69: Resolution: Fixed Fix Version/s: 0.23.1 Target Version/s: 0.24.0, 0.23.1 (was: 0.23.1, 0.24.0) Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to branch-0.23 and trunk. Thanks Ravi and Jakob! Improve dfsadmin command line help --- Key: HDFS-69 URL: https://issues.apache.org/jira/browse/HDFS-69 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 1.0.0 Reporter: Ravi Phulari Assignee: Harsh J Priority: Minor Fix For: 0.23.1 Attachments: HDFS-69.patch Enhance dfsadmin command line help informing A quota of one forces a directory to remain empty -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-69) Improve dfsadmin command line help
[ https://issues.apache.org/jira/browse/HDFS-69?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13184768#comment-13184768 ] Hudson commented on HDFS-69: Integrated in Hadoop-Hdfs-0.23-Commit #362 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/362/]) merge HDFS-69 (harsh) harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1230414 Files : * /hadoop/common/branches/branch-0.23 * /hadoop/common/branches/branch-0.23/hadoop-common-project * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/.gitignore * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-examples * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++ * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/FileBench.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/TestSequenceFileMergeProgress.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/security/authorize/TestServiceLevelAuthorization.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/test/MapredTestDriver.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job * /hadoop/common/branches/branch-0.23/hadoop-project * /hadoop/common/branches/branch-0.23/hadoop-project/src/site Improve dfsadmin command line help --- Key: HDFS-69 URL: https://issues.apache.org/jira/browse/HDFS-69 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 1.0.0 Reporter: Ravi Phulari
[jira] [Commented] (HDFS-69) Improve dfsadmin command line help
[ https://issues.apache.org/jira/browse/HDFS-69?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13184770#comment-13184770 ] Hudson commented on HDFS-69: Integrated in Hadoop-Common-0.23-Commit #372 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/372/]) merge HDFS-69 (harsh) harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1230414 Files : * /hadoop/common/branches/branch-0.23 * /hadoop/common/branches/branch-0.23/hadoop-common-project * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/.gitignore * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-examples * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++ * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/FileBench.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/TestSequenceFileMergeProgress.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/security/authorize/TestServiceLevelAuthorization.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/test/MapredTestDriver.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job * /hadoop/common/branches/branch-0.23/hadoop-project * /hadoop/common/branches/branch-0.23/hadoop-project/src/site Improve dfsadmin command line help --- Key: HDFS-69 URL: https://issues.apache.org/jira/browse/HDFS-69 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 1.0.0 Reporter: Ravi
[jira] [Commented] (HDFS-69) Improve dfsadmin command line help
[ https://issues.apache.org/jira/browse/HDFS-69?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13184776#comment-13184776 ] Hudson commented on HDFS-69: Integrated in Hadoop-Mapreduce-0.23-Commit #384 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/384/]) merge HDFS-69 (harsh) harsh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1230414 Files : * /hadoop/common/branches/branch-0.23 * /hadoop/common/branches/branch-0.23/hadoop-common-project * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-auth * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/docs * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/core * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/native * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/.gitignore * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/bin * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/conf * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-examples * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/c++ * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/block_forensics * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build-contrib.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/build.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/data_join * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/eclipse-plugin * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/index * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/vaidya * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/examples * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/fs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/hdfs * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/FileBench.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/io/TestSequenceFileMergeProgress.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/ipc * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/security/authorize/TestServiceLevelAuthorization.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/test/MapredTestDriver.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job * /hadoop/common/branches/branch-0.23/hadoop-project * /hadoop/common/branches/branch-0.23/hadoop-project/src/site Improve dfsadmin command line help --- Key: HDFS-69 URL: https://issues.apache.org/jira/browse/HDFS-69 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 1.0.0 Reporter:
[jira] [Commented] (HDFS-2601) Proposal to store edits and checkpoints inside HDFS itself for namenode HA
[ https://issues.apache.org/jira/browse/HDFS-2601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13184778#comment-13184778 ] Sanjay Radia commented on HDFS-2601: Makes a lot of sense; HA is just one of its benefits (as Todd states); i suggest we take HA out of the JIRA title but record it as one of the benfits. Several folks have talked about this in the past - the main issue was write latency and keeping the metadata in an independent storage system. Proposal to store edits and checkpoints inside HDFS itself for namenode HA -- Key: HDFS-2601 URL: https://issues.apache.org/jira/browse/HDFS-2601 Project: Hadoop HDFS Issue Type: New Feature Components: name-node Reporter: Karthik Ranganathan Would have liked to make this a brainstorming JIRA but couldn't find the option for some reason. I have talked to a quite a few people about this proposal at Facebook internally (HDFS folks like Hairong and Dhruba, as well as HBase folks interested in this feature), and wanted to broaden the audience. At the core of the HA feature, we need 2 things: A. the secondary NN (or avatar stand-by or whatever we call it) needs to read all the fsedits and fsimage data written by the primary NN B. Once the stand-by has taken over, the old NN should not be allowed to make any edits The basic idea is as follows (there are some variants, we can hone in on the details if we like the general approach): 1. The write path for fsedits and fsimage: 1.1 The NN uses a dfs client to write fsedits and fsimage. These will be regular hdfs files written using the write pipeline. 1.2 Let us say the fsimage and fsedits files are written to a well-known location in the local HDFS itself (say /.META or some such location) 1.3 The create files and add blocks to files in this path are not written to fsimage or fsedits. The location of the blocks for the files in this location are known to all namenodes - primary and standby - somehow (some possibilities here - write these block ids to zk or use reserved block ids or write some meta-data into the blocks itself and store the blocks in a well known location on all the datanodes) 1.4 If the replication factor on the write pipeline decreases, we close the block immediately and allow NN to re-replicate to bring up the replication factor. We continue writing to a new block 2. The read path on a NN failure 2.1 Since the new NN knows the location of the blocks for the fsedits and fsimage (again the same possibilities as mentioned above), there is nothing to do to determine this 2.2 It can read the files it needs using the HDFS client itself 3. Fencing - if a NN is unresponsive, a new NN takes over, old NN should not be allowed to perform any action 3.1 Use HDFS lease recovery for the fsedits and fsimage files - the new NN will close all these files baing written to by the old NN (and hence all the blocks) 3.2 The new NN (avatar NN) will write its address into ZK to let everyone know its the master 3.3 The new NN now gets the lease for these files and starts writing into the fsedits and fsimage 3.4 The old NN cannot write into the file as the block it was writing to was closed and it does not have the lease. If it needs to re-open these files, it needs to check zk to see it is indeed the current master, if not it should exit. 4. Misc considerations: 4.1 If needed, we can specify favored nodes to place the blocks for this data in specific set of nodes (say we want to use a different set of RAIDed nodes, etc). 4.2 Since we wont record the entries for /.META in fsedits and fsimage, a hadoop dfs -ls / command wont show the files. This is probably ok, and can be fixed if not. 4.3 If we have 256MB block sizes, then 20GB fsimage file would need 80 block ids - the NN would need only these 80 block ids to read all the fsedits data. The fsimage data is even lesser. This is very tractable using a variety of the techniques (the possibilities mentioned above). The advantage is that we are re-using the existing HDFS client (with some enhancements of course), and making the solution self-sufficient on the existing HDFS. Also, the operational complexity is greatly reduced. Thoughts? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2601) Proposal to store edits and checkpoints inside HDFS itself for namenode HA
[ https://issues.apache.org/jira/browse/HDFS-2601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13184780#comment-13184780 ] Sanjay Radia commented on HDFS-2601: Putting the Image in HDFS should be a lot simpler and does not require addressing the latency issue. The checkpointer could write the image directly to HDFS; this is feasible due to the image/edits cleanup (HDFS-1073). There is a bootstrapping issue on NN startup but that should be solvable. Proposal to store edits and checkpoints inside HDFS itself for namenode HA -- Key: HDFS-2601 URL: https://issues.apache.org/jira/browse/HDFS-2601 Project: Hadoop HDFS Issue Type: New Feature Components: name-node Reporter: Karthik Ranganathan Would have liked to make this a brainstorming JIRA but couldn't find the option for some reason. I have talked to a quite a few people about this proposal at Facebook internally (HDFS folks like Hairong and Dhruba, as well as HBase folks interested in this feature), and wanted to broaden the audience. At the core of the HA feature, we need 2 things: A. the secondary NN (or avatar stand-by or whatever we call it) needs to read all the fsedits and fsimage data written by the primary NN B. Once the stand-by has taken over, the old NN should not be allowed to make any edits The basic idea is as follows (there are some variants, we can hone in on the details if we like the general approach): 1. The write path for fsedits and fsimage: 1.1 The NN uses a dfs client to write fsedits and fsimage. These will be regular hdfs files written using the write pipeline. 1.2 Let us say the fsimage and fsedits files are written to a well-known location in the local HDFS itself (say /.META or some such location) 1.3 The create files and add blocks to files in this path are not written to fsimage or fsedits. The location of the blocks for the files in this location are known to all namenodes - primary and standby - somehow (some possibilities here - write these block ids to zk or use reserved block ids or write some meta-data into the blocks itself and store the blocks in a well known location on all the datanodes) 1.4 If the replication factor on the write pipeline decreases, we close the block immediately and allow NN to re-replicate to bring up the replication factor. We continue writing to a new block 2. The read path on a NN failure 2.1 Since the new NN knows the location of the blocks for the fsedits and fsimage (again the same possibilities as mentioned above), there is nothing to do to determine this 2.2 It can read the files it needs using the HDFS client itself 3. Fencing - if a NN is unresponsive, a new NN takes over, old NN should not be allowed to perform any action 3.1 Use HDFS lease recovery for the fsedits and fsimage files - the new NN will close all these files baing written to by the old NN (and hence all the blocks) 3.2 The new NN (avatar NN) will write its address into ZK to let everyone know its the master 3.3 The new NN now gets the lease for these files and starts writing into the fsedits and fsimage 3.4 The old NN cannot write into the file as the block it was writing to was closed and it does not have the lease. If it needs to re-open these files, it needs to check zk to see it is indeed the current master, if not it should exit. 4. Misc considerations: 4.1 If needed, we can specify favored nodes to place the blocks for this data in specific set of nodes (say we want to use a different set of RAIDed nodes, etc). 4.2 Since we wont record the entries for /.META in fsedits and fsimage, a hadoop dfs -ls / command wont show the files. This is probably ok, and can be fixed if not. 4.3 If we have 256MB block sizes, then 20GB fsimage file would need 80 block ids - the NN would need only these 80 block ids to read all the fsedits data. The fsimage data is even lesser. This is very tractable using a variety of the techniques (the possibilities mentioned above). The advantage is that we are re-using the existing HDFS client (with some enhancements of course), and making the solution self-sufficient on the existing HDFS. Also, the operational complexity is greatly reduced. Thoughts? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
