[jira] [Commented] (HDFS-3077) Quorum-based protocol for reading and writing edit logs
[
https://issues.apache.org/jira/browse/HDFS-3077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13229657#comment-13229657
]
Ivan Kelly commented on HDFS-3077:
--
{quote}
We have a single writer, except for when we don't. During a failover, without a
STONITH capability,
{quote}
Without some sort of fencing, you're going to have to run agreement on every
update. If this is acceptable, you could have just made the namenode a thin RPC
layer on top of zookeeper, and you get fault tolerance for free.
Quorum-based protocol for reading and writing edit logs
---
Key: HDFS-3077
URL: https://issues.apache.org/jira/browse/HDFS-3077
Project: Hadoop HDFS
Issue Type: New Feature
Components: ha, name-node
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Currently, one of the weak points of the HA design is that it relies on
shared storage such as an NFS filer for the shared edit log. One alternative
that has been proposed is to depend on BookKeeper, a ZooKeeper subproject
which provides a highly available replicated edit log on commodity hardware.
This JIRA is to implement another alternative, based on a quorum commit
protocol, integrated more tightly in HDFS and with the requirements driven
only by HDFS's needs rather than more generic use cases. More details to
follow.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2743) Streamline usage of bookkeeper journal manager
[ https://issues.apache.org/jira/browse/HDFS-2743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13182636#comment-13182636 ] Ivan Kelly commented on HDFS-2743: -- As before, warnings are unrelated. Streamline usage of bookkeeper journal manager -- Key: HDFS-2743 URL: https://issues.apache.org/jira/browse/HDFS-2743 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Assignee: Ivan Kelly Fix For: 0.24.0 Attachments: HDFS-2743.diff, HDFS-2743.diff The current method of installing bkjournal manager involves generating a tarball, and extracting it with special flags over the hdfs distribution. This is cumbersome and prone to being broken by other changes (see https://svn.apache.org/repos/asf/hadoop/common/trunk@1220940). I think a cleaner way to doing this is to generate a single jar that can be placed in the lib dir of hdfs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2743) Streamline usage of bookkeeper journal manager
[
https://issues.apache.org/jira/browse/HDFS-2743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13178808#comment-13178808
]
Ivan Kelly commented on HDFS-2743:
--
{qoute}-1 tests included. The patch doesn't appear to include any new or
modified tests.{quote}
Change is to packaging, no test required.
{quote}-1 javadoc. The javadoc tool appears to have generated 20 warning
messages.
-1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9)
warnings.
{quote}
This are not within the bkjournal contrib module.
{quote}
-1 release audit. The applied patch generated 1 release audit warnings (more
than the trunk's current 0 warnings).
{quote}
This relates to: br/
!?
/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/dev-support/findbugsExcludeFile.xml
Streamline usage of bookkeeper journal manager
--
Key: HDFS-2743
URL: https://issues.apache.org/jira/browse/HDFS-2743
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Ivan Kelly
Assignee: Ivan Kelly
Fix For: 0.24.0
Attachments: HDFS-2743.diff
The current method of installing bkjournal manager involves generating a
tarball, and extracting it with special flags over the hdfs distribution.
This is cumbersome and prone to being broken by other changes (see
https://svn.apache.org/repos/asf/hadoop/common/trunk@1220940). I think a
cleaner way to doing this is to generate a single jar that can be placed in
the lib dir of hdfs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-234) Integration with BookKeeper logging system
[ https://issues.apache.org/jira/browse/HDFS-234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13169212#comment-13169212 ] Ivan Kelly commented on HDFS-234: - Awesome :D Thanks for your help Jitendra. Integration with BookKeeper logging system -- Key: HDFS-234 URL: https://issues.apache.org/jira/browse/HDFS-234 Project: Hadoop HDFS Issue Type: New Feature Reporter: Luca Telloli Assignee: Ivan Kelly Fix For: HA branch (HDFS-1623), 0.24.0 Attachments: HADOOP-5189-trunk-preview.patch, HADOOP-5189-trunk-preview.patch, HADOOP-5189-trunk-preview.patch, HADOOP-5189-v.19.patch, HADOOP-5189.patch, HDFS-234.diff, HDFS-234.diff, HDFS-234.diff, HDFS-234.diff, HDFS-234.diff, HDFS-234.patch, create.png, hdfs_tpt_lat.pdf, zookeeper-dev-bookkeeper.jar, zookeeper-dev.jar BookKeeper is a system to reliably log streams of records (https://issues.apache.org/jira/browse/ZOOKEEPER-276). The NameNode is a natural target for such a system for being the metadata repository of the entire file system for HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-234) Integration with BookKeeper logging system
[ https://issues.apache.org/jira/browse/HDFS-234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13167586#comment-13167586 ] Ivan Kelly commented on HDFS-234: - I think we should commit it to hdfs contrib. Please move the bkjournal package to contrib. In principle I have no problem with this. However, contrib doesn't seem to be set up to run with maven at the moment. This needs to be compiled and tested as part of the normal build cycle. Otherwise, changes to the NN interfaces will silently break this. Integration with BookKeeper logging system -- Key: HDFS-234 URL: https://issues.apache.org/jira/browse/HDFS-234 Project: Hadoop HDFS Issue Type: New Feature Reporter: Luca Telloli Assignee: Ivan Kelly Fix For: HA branch (HDFS-1623), 0.24.0 Attachments: HADOOP-5189-trunk-preview.patch, HADOOP-5189-trunk-preview.patch, HADOOP-5189-trunk-preview.patch, HADOOP-5189-v.19.patch, HADOOP-5189.patch, HDFS-234.diff, HDFS-234.diff, HDFS-234.diff, HDFS-234.diff, HDFS-234.patch, create.png, hdfs_tpt_lat.pdf, zookeeper-dev-bookkeeper.jar, zookeeper-dev.jar BookKeeper is a system to reliably log streams of records (https://issues.apache.org/jira/browse/ZOOKEEPER-276). The NameNode is a natural target for such a system for being the metadata repository of the entire file system for HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[
https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13161234#comment-13161234
]
Ivan Kelly commented on HDFS-1580:
--
I couldn't find any, but multiple reruns of it and the surrounding tests didnt
have any fails :/
{code}
Running org.apache.hadoop.hdfs.TestMissingBlocksAlert
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.422 sec
Running org.apache.hadoop.hdfs.TestDecommission
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 172.954 sec
Running org.apache.hadoop.hdfs.TestAppendDifferentChecksum
Tests run: 3, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 7.52 sec
Running org.apache.hadoop.hdfs.TestBlocksScheduledCounter
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.017 sec
Running org.apache.hadoop.hdfs.server.namenode.TestGenericJournalConf
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.41 sec
{code}
Im going to reupload the patch to kick off tests again.
Add interface for generic Write Ahead Logging mechanisms
Key: HDFS-1580
URL: https://issues.apache.org/jira/browse/HDFS-1580
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Ivan Kelly
Assignee: Jitendra Nath Pandey
Fix For: HA branch (HDFS-1623), 0.24.0
Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf,
EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff,
HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff,
HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff,
HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf,
generic_wal_iface.pdf, generic_wal_iface.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13143541#comment-13143541 ] Ivan Kelly commented on HDFS-1580: -- Local reruns of TestAbandonBlock passes cleanly. Seems flakey in general, also failed on Build #1527, Build #1527, Build #1527 Build #1511. I didn't look back any further. Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Assignee: Jitendra Nath Pandey Fix For: HA branch (HDFS-1623), 0.24.0 Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, EditlogInterface.3.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1975) HA: Support for sharing the namenode state from active to standby.
[ https://issues.apache.org/jira/browse/HDFS-1975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13130463#comment-13130463 ] Ivan Kelly commented on HDFS-1975: -- General direction looks good. I've a few comments. There's a comma in configuration key. In FSEditLog, check the states before transitioning them. Im not sure the tailer will work as is. What happens if you open an inprogress input stream with this? As I understand it, you'll end up with lastTxnId in the middle of the segment. In stopReadingEditLogs(), instead of doing the start stop, to ensure up to dateness, you could have a call on EditLogTailer#applyLatestUpdates(). Then EditLogTailerThread could call this in the loop also. HA: Support for sharing the namenode state from active to standby. -- Key: HDFS-1975 URL: https://issues.apache.org/jira/browse/HDFS-1975 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Suresh Srinivas Assignee: Jitendra Nath Pandey Attachments: HDFS-1975-HA.patch, hdfs-1975.txt, hdfs-1975.txt To enable hot standby namenode, the standby node must have current information for - namenode state (image + edits) and block location information. This jira addresses keeping the namenode state current in the standby node. To do this, the proposed solution in this jira is to use a shared storage to store the namenode state. Note one could also build an alternative solution by augmenting the backup node. A seperate jira could explore this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2188) HDFS-1580: Make FSEditLog create its journals from a list of URIs rather than NNStorage
[ https://issues.apache.org/jira/browse/HDFS-2188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13118088#comment-13118088 ] Ivan Kelly commented on HDFS-2188: -- Uploaded new patch, rebased on trunk (over HDFS-2158). I've left in the old constructor now, it's @VisibleForTesting. I didn't changes the FileJournalManager constructor. This would be better done later i think, once HDFS-2174 is done. We shouldn't move attemptRestoreRemovedStorage into FileJournalManager at this point, as it would mean that it could get called multiple times if there are multiple FileJournalManagers. HDFS-1580: Make FSEditLog create its journals from a list of URIs rather than NNStorage --- Key: HDFS-2188 URL: https://issues.apache.org/jira/browse/HDFS-2188 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Assignee: Ivan Kelly Fix For: 0.23.0 Attachments: HDFS-2188.diff, HDFS-2188.diff, HDFS-2188.diff, HDFS-2188.diff, HDFS-2188.diff Currently, FSEditLog retrieves the list of Journals to create from NNStorage. Obviously this is file specific. This JIRA aims to remove this restriction to make it possible to create journals of custom types. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
