[jira] [Commented] (HDFS-2316) [umbrella] webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP
[ https://issues.apache.org/jira/browse/HDFS-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158253#comment-13158253 ] Hudson commented on HDFS-2316: -- Integrated in Hadoop-Mapreduce-trunk-Commit #1353 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1353/]) HDFS-2316. Record completion of umbrella jira. Contributed by Tsz Wo (Nicholas), Sze. mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1206990 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > [umbrella] webhdfs: a complete FileSystem implementation for accessing HDFS > over HTTP > - > > Key: HDFS-2316 > URL: https://issues.apache.org/jira/browse/HDFS-2316 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Labels: critical-0.22.0 > Fix For: 0.23.1, 1.0.0 > > Attachments: WebHdfsAPI20111020.pdf, WebHdfsAPI2003.pdf, > WebHdfsAPI2011.pdf, test-webhdfs, test-webhdfs-0.20s > > > We current have hftp for accessing HDFS over HTTP. However, hftp is a > read-only FileSystem and does not provide "write" accesses. > In HDFS-2284, we propose to have webhdfs for providing a complete FileSystem > implementation for accessing HDFS over HTTP. The is the umbrella JIRA for > the tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2316) [umbrella] webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP
[ https://issues.apache.org/jira/browse/HDFS-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158233#comment-13158233 ] Hudson commented on HDFS-2316: -- Integrated in Hadoop-Common-trunk-Commit #1329 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1329/]) HDFS-2316. Record completion of umbrella jira. Contributed by Tsz Wo (Nicholas), Sze. mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1206990 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > [umbrella] webhdfs: a complete FileSystem implementation for accessing HDFS > over HTTP > - > > Key: HDFS-2316 > URL: https://issues.apache.org/jira/browse/HDFS-2316 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Labels: critical-0.22.0 > Fix For: 0.23.1, 1.0.0 > > Attachments: WebHdfsAPI20111020.pdf, WebHdfsAPI2003.pdf, > WebHdfsAPI2011.pdf, test-webhdfs, test-webhdfs-0.20s > > > We current have hftp for accessing HDFS over HTTP. However, hftp is a > read-only FileSystem and does not provide "write" accesses. > In HDFS-2284, we propose to have webhdfs for providing a complete FileSystem > implementation for accessing HDFS over HTTP. The is the umbrella JIRA for > the tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2316) [umbrella] webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP
[ https://issues.apache.org/jira/browse/HDFS-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158231#comment-13158231 ] Hudson commented on HDFS-2316: -- Integrated in Hadoop-Hdfs-trunk-Commit #1403 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1403/]) HDFS-2316. Record completion of umbrella jira. Contributed by Tsz Wo (Nicholas), Sze. mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1206990 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > [umbrella] webhdfs: a complete FileSystem implementation for accessing HDFS > over HTTP > - > > Key: HDFS-2316 > URL: https://issues.apache.org/jira/browse/HDFS-2316 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Labels: critical-0.22.0 > Fix For: 0.23.1, 1.0.0 > > Attachments: WebHdfsAPI20111020.pdf, WebHdfsAPI2003.pdf, > WebHdfsAPI2011.pdf, test-webhdfs, test-webhdfs-0.20s > > > We current have hftp for accessing HDFS over HTTP. However, hftp is a > read-only FileSystem and does not provide "write" accesses. > In HDFS-2284, we propose to have webhdfs for providing a complete FileSystem > implementation for accessing HDFS over HTTP. The is the umbrella JIRA for > the tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2316) [umbrella] webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP
[ https://issues.apache.org/jira/browse/HDFS-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158232#comment-13158232 ] Hudson commented on HDFS-2316: -- Integrated in Hadoop-Mapreduce-0.23-Commit #231 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/231/]) HDFS-2316. Record completion of umbrella jira. Contributed by Tsz Wo (Nicholas), Sze. mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1206989 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > [umbrella] webhdfs: a complete FileSystem implementation for accessing HDFS > over HTTP > - > > Key: HDFS-2316 > URL: https://issues.apache.org/jira/browse/HDFS-2316 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Labels: critical-0.22.0 > Fix For: 0.23.1, 1.0.0 > > Attachments: WebHdfsAPI20111020.pdf, WebHdfsAPI2003.pdf, > WebHdfsAPI2011.pdf, test-webhdfs, test-webhdfs-0.20s > > > We current have hftp for accessing HDFS over HTTP. However, hftp is a > read-only FileSystem and does not provide "write" accesses. > In HDFS-2284, we propose to have webhdfs for providing a complete FileSystem > implementation for accessing HDFS over HTTP. The is the umbrella JIRA for > the tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2316) [umbrella] webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP
[ https://issues.apache.org/jira/browse/HDFS-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158222#comment-13158222 ] Hudson commented on HDFS-2316: -- Integrated in Hadoop-Common-0.23-Commit #220 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/220/]) HDFS-2316. Record completion of umbrella jira. Contributed by Tsz Wo (Nicholas), Sze. mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1206989 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > [umbrella] webhdfs: a complete FileSystem implementation for accessing HDFS > over HTTP > - > > Key: HDFS-2316 > URL: https://issues.apache.org/jira/browse/HDFS-2316 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Labels: critical-0.22.0 > Fix For: 0.23.1, 1.0.0 > > Attachments: WebHdfsAPI20111020.pdf, WebHdfsAPI2003.pdf, > WebHdfsAPI2011.pdf, test-webhdfs, test-webhdfs-0.20s > > > We current have hftp for accessing HDFS over HTTP. However, hftp is a > read-only FileSystem and does not provide "write" accesses. > In HDFS-2284, we propose to have webhdfs for providing a complete FileSystem > implementation for accessing HDFS over HTTP. The is the umbrella JIRA for > the tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2316) [umbrella] webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP
[ https://issues.apache.org/jira/browse/HDFS-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158219#comment-13158219 ] Hudson commented on HDFS-2316: -- Integrated in Hadoop-Hdfs-0.23-Commit #217 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/217/]) HDFS-2316. Record completion of umbrella jira. Contributed by Tsz Wo (Nicholas), Sze. mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1206989 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > [umbrella] webhdfs: a complete FileSystem implementation for accessing HDFS > over HTTP > - > > Key: HDFS-2316 > URL: https://issues.apache.org/jira/browse/HDFS-2316 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Labels: critical-0.22.0 > Fix For: 0.23.1, 1.0.0 > > Attachments: WebHdfsAPI20111020.pdf, WebHdfsAPI2003.pdf, > WebHdfsAPI2011.pdf, test-webhdfs, test-webhdfs-0.20s > > > We current have hftp for accessing HDFS over HTTP. However, hftp is a > read-only FileSystem and does not provide "write" accesses. > In HDFS-2284, we propose to have webhdfs for providing a complete FileSystem > implementation for accessing HDFS over HTTP. The is the umbrella JIRA for > the tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2596) TestDirectoryScanner doesn't test parallel scans
[
https://issues.apache.org/jira/browse/HDFS-2596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins updated HDFS-2596:
--
Attachment: hdfs-2596-1.patch
Patch attached.
> TestDirectoryScanner doesn't test parallel scans
>
>
> Key: HDFS-2596
> URL: https://issues.apache.org/jira/browse/HDFS-2596
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: data-node, test
>Affects Versions: 0.22.0, 0.23.0
>Reporter: Eli Collins
>Assignee: Eli Collins
> Attachments: hdfs-2596-1.patch
>
>
> The code from HDFS-854 below doesn't run the test with parallel scanning.
> They probably intended "parallelism < 3".
> {code}
> + public void testDirectoryScanner() throws Exception {
> +// Run the test with and without parallel scanning
> +for (int parallelism = 1; parallelism < 2; parallelism++) {
> + runTest(parallelism);
> +}
> + }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2596) TestDirectoryScanner doesn't test parallel scans
TestDirectoryScanner doesn't test parallel scans
Key: HDFS-2596
URL: https://issues.apache.org/jira/browse/HDFS-2596
Project: Hadoop HDFS
Issue Type: Bug
Components: data-node, test
Affects Versions: 0.23.0, 0.22.0
Reporter: Eli Collins
Assignee: Eli Collins
Attachments: hdfs-2596-1.patch
The code from HDFS-854 below doesn't run the test with parallel scanning. They
probably intended "parallelism < 3".
{code}
+ public void testDirectoryScanner() throws Exception {
+// Run the test with and without parallel scanning
+for (int parallelism = 1; parallelism < 2; parallelism++) {
+ runTest(parallelism);
+}
+ }
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2316) [umbrella] webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP
[ https://issues.apache.org/jira/browse/HDFS-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HDFS-2316: - Target Version/s: 0.23.0, 0.22.0, 1.0.0 (was: 1.0.0, 0.22.0, 0.23.0) Summary: [umbrella] webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP (was: webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP) > [umbrella] webhdfs: a complete FileSystem implementation for accessing HDFS > over HTTP > - > > Key: HDFS-2316 > URL: https://issues.apache.org/jira/browse/HDFS-2316 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Labels: critical-0.22.0 > Fix For: 0.23.1, 1.0.0 > > Attachments: WebHdfsAPI20111020.pdf, WebHdfsAPI2003.pdf, > WebHdfsAPI2011.pdf, test-webhdfs, test-webhdfs-0.20s > > > We current have hftp for accessing HDFS over HTTP. However, hftp is a > read-only FileSystem and does not provide "write" accesses. > In HDFS-2284, we propose to have webhdfs for providing a complete FileSystem > implementation for accessing HDFS over HTTP. The is the umbrella JIRA for > the tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2316) webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP
[ https://issues.apache.org/jira/browse/HDFS-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HDFS-2316: - Target Version/s: 0.23.0, 0.22.0, 1.0.0 (was: 0.22.0, 1.0.0, 0.23.0) Release Note: Provide webhdfs as a complete FileSystem implementation for accessing HDFS over HTTP. Previous hftp feature was a read-only FileSystem and does not provide "write" accesses. > webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP > -- > > Key: HDFS-2316 > URL: https://issues.apache.org/jira/browse/HDFS-2316 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Labels: critical-0.22.0 > Fix For: 0.23.1, 1.0.0 > > Attachments: WebHdfsAPI20111020.pdf, WebHdfsAPI2003.pdf, > WebHdfsAPI2011.pdf, test-webhdfs, test-webhdfs-0.20s > > > We current have hftp for accessing HDFS over HTTP. However, hftp is a > read-only FileSystem and does not provide "write" accesses. > In HDFS-2284, we propose to have webhdfs for providing a complete FileSystem > implementation for accessing HDFS over HTTP. The is the umbrella JIRA for > the tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-2316) webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP
[ https://issues.apache.org/jira/browse/HDFS-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley reopened HDFS-2316: -- re-opening to accomodate request to fix also in 0.22.0. > webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP > -- > > Key: HDFS-2316 > URL: https://issues.apache.org/jira/browse/HDFS-2316 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Labels: critical-0.22.0 > Fix For: 0.23.1, 1.0.0 > > Attachments: WebHdfsAPI20111020.pdf, WebHdfsAPI2003.pdf, > WebHdfsAPI2011.pdf, test-webhdfs, test-webhdfs-0.20s > > > We current have hftp for accessing HDFS over HTTP. However, hftp is a > read-only FileSystem and does not provide "write" accesses. > In HDFS-2284, we propose to have webhdfs for providing a complete FileSystem > implementation for accessing HDFS over HTTP. The is the umbrella JIRA for > the tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2588) hdfs jsp pages missing DOCTYPE [post-split branches]
[ https://issues.apache.org/jira/browse/HDFS-2588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HDFS-2588: - Target Version/s: 0.23.1 Fix Version/s: (was: 1.0.0) Summary: hdfs jsp pages missing DOCTYPE [post-split branches] (was: hdfs jsp pages missing DOCTYPE) Corrected this bug to only relate to post-split branches/trunk. The parent bug only will be used for pre-split 1.0. > hdfs jsp pages missing DOCTYPE [post-split branches] > > > Key: HDFS-2588 > URL: https://issues.apache.org/jira/browse/HDFS-2588 > Project: Hadoop HDFS > Issue Type: Bug > Components: scripts >Reporter: Dave Vronay >Assignee: Dave Vronay >Priority: Trivial > Fix For: 0.23.1 > > > Some jsp pages in the UI are missing a DOCTYPE declaration. This causes the > pages to render incorrectly on some browsers, such as IE9. Please see parent > bug HADOOP-7827 for details and patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2246) Shortcut a local client reads to a Datanodes files directly
[ https://issues.apache.org/jira/browse/HDFS-2246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HDFS-2246: - Target Version/s: 0.23.1, 1.0.0 (was: 0.23.1, 0.24.0, 1.0.0) Fix Version/s: (was: 0.24.0) 1.0.0 0.23.1 corrected Fix Versions field > Shortcut a local client reads to a Datanodes files directly > --- > > Key: HDFS-2246 > URL: https://issues.apache.org/jira/browse/HDFS-2246 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Sanjay Radia >Assignee: Jitendra Nath Pandey > Fix For: 0.23.1, 1.0.0 > > Attachments: 0001-HDFS-347.-Local-reads.patch, > HDFS-2246-branch-0.20-security-205.1.patch, > HDFS-2246-branch-0.20-security-205.2.patch, > HDFS-2246-branch-0.20-security-205.patch, > HDFS-2246-branch-0.20-security-205.patch, > HDFS-2246-branch-0.20-security-205.patch, > HDFS-2246-branch-0.20-security.3.patch, > HDFS-2246-branch-0.20-security.no-softref.patch, > HDFS-2246-branch-0.20-security.patch, HDFS-2246-branch-0.20-security.patch, > HDFS-2246-branch-0.20-security.patch, HDFS-2246-branch-0.20-security.patch, > HDFS-2246-trunk.patch, HDFS-2246-trunk.patch, HDFS-2246-trunk.patch, > HDFS-2246-trunk.patch, HDFS-2246-trunk.patch, HDFS-2246-trunk.patch, > HDFS-2246.20s.1.patch, HDFS-2246.20s.2.txt, HDFS-2246.20s.3.txt, > HDFS-2246.20s.4.txt, HDFS-2246.20s.patch, TestShortCircuitLocalRead.java, > localReadShortcut20-security.2patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1975) HA: Support for sharing the namenode state from active to standby.
[ https://issues.apache.org/jira/browse/HDFS-1975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158168#comment-13158168 ] Todd Lipcon commented on HDFS-1975: --- Another issue that we have to tackle before this can provide hot standby is this: When we close a file, or add another block to a file, we write OP_CLOSE or OP_ADD in the txn log. FSEditLogLoader, when it sees these types of transactions, creates new BlockInfo objects for all of the blocks listed in the transaction. These new BlockInfos have no block locations associated. So, when we close a file, the SBNN loses its block locations info for that file and is no longer "hot". I have an ugly hack which copies over the old BlockInfos from the existing INode, but I'm not convinced it's the right way. It might be cleaner to add new opcode types like OP_ADD_ADDITIONAL_BLOCK, and actually treat OP_CLOSE as just a finalization of INodeFileUnderConstruction to INodeFile, rather than replacing block info at all. Thoughts? > HA: Support for sharing the namenode state from active to standby. > -- > > Key: HDFS-1975 > URL: https://issues.apache.org/jira/browse/HDFS-1975 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Reporter: Suresh Srinivas >Assignee: Jitendra Nath Pandey > Attachments: HDFS-1975-HA.2.patch, HDFS-1975-HA.patch, > HDFS-1975-HDFS-1623.patch, HDFS-1975-HDFS-1623.patch, > HDFS-1975-HDFS-1623.patch, hdfs-1975.txt, hdfs-1975.txt > > > To enable hot standby namenode, the standby node must have current > information for - namenode state (image + edits) and block location > information. This jira addresses keeping the namenode state current in the > standby node. To do this, the proposed solution in this jira is to use a > shared storage to store the namenode state. > Note one could also build an alternative solution by augmenting the backup > node. A seperate jira could explore this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1108) Log newly allocated blocks
[ https://issues.apache.org/jira/browse/HDFS-1108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158161#comment-13158161 ] Todd Lipcon commented on HDFS-1108: --- oh, duh... you're right... I think we should add back the configuration which allows it to persist blocks even if HA is not enabled. As the tests demonstrates, it's necessary for "fast restart" as a form of HA for tiny clusters. It will also allow this test to work without setting up the entire HA infrastructure. > Log newly allocated blocks > -- > > Key: HDFS-1108 > URL: https://issues.apache.org/jira/browse/HDFS-1108 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Reporter: dhruba borthakur >Assignee: Todd Lipcon > Fix For: HA branch (HDFS-1623) > > Attachments: HDFS-1108.patch, hdfs-1108-habranch.txt, > hdfs-1108-habranch.txt, hdfs-1108-habranch.txt, hdfs-1108.txt > > > The current HDFS design says that newly allocated blocks for a file are not > persisted in the NN transaction log when the block is allocated. Instead, a > hflush() or a close() on the file persists the blocks into the transaction > log. It would be nice if we can immediately persist newly allocated blocks > (as soon as they are allocated) for specific files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1108) Log newly allocated blocks
[ https://issues.apache.org/jira/browse/HDFS-1108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158156#comment-13158156 ] Eli Collins commented on HDFS-1108: --- Doesn't the test need to enable HA and shared edits dir or is it only covering the syncs triggered by supportAppends? > Log newly allocated blocks > -- > > Key: HDFS-1108 > URL: https://issues.apache.org/jira/browse/HDFS-1108 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Reporter: dhruba borthakur >Assignee: Todd Lipcon > Fix For: HA branch (HDFS-1623) > > Attachments: HDFS-1108.patch, hdfs-1108-habranch.txt, > hdfs-1108-habranch.txt, hdfs-1108-habranch.txt, hdfs-1108.txt > > > The current HDFS design says that newly allocated blocks for a file are not > persisted in the NN transaction log when the block is allocated. Instead, a > hflush() or a close() on the file persists the blocks into the transaction > log. It would be nice if we can immediately persist newly allocated blocks > (as soon as they are allocated) for specific files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1108) Log newly allocated blocks
[ https://issues.apache.org/jira/browse/HDFS-1108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158154#comment-13158154 ] Hadoop QA commented on HDFS-1108: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12505289/hdfs-1108-habranch.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1610//console This message is automatically generated. > Log newly allocated blocks > -- > > Key: HDFS-1108 > URL: https://issues.apache.org/jira/browse/HDFS-1108 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Reporter: dhruba borthakur >Assignee: Todd Lipcon > Fix For: HA branch (HDFS-1623) > > Attachments: HDFS-1108.patch, hdfs-1108-habranch.txt, > hdfs-1108-habranch.txt, hdfs-1108-habranch.txt, hdfs-1108.txt > > > The current HDFS design says that newly allocated blocks for a file are not > persisted in the NN transaction log when the block is allocated. Instead, a > hflush() or a close() on the file persists the blocks into the transaction > log. It would be nice if we can immediately persist newly allocated blocks > (as soon as they are allocated) for specific files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1975) HA: Support for sharing the namenode state from active to standby.
[ https://issues.apache.org/jira/browse/HDFS-1975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158153#comment-13158153 ] Todd Lipcon commented on HDFS-1975: --- another possible answer is that we need to modify FSNamesystem.isPopulatingReplQueues to return false on the standby, and then when it switches from standby to active, initialize the replication queues only after reading the latest edits... I think that will solve the SET_REPLICATION issue, but not certain if it will solve all the issues in this general class. > HA: Support for sharing the namenode state from active to standby. > -- > > Key: HDFS-1975 > URL: https://issues.apache.org/jira/browse/HDFS-1975 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Reporter: Suresh Srinivas >Assignee: Jitendra Nath Pandey > Attachments: HDFS-1975-HA.2.patch, HDFS-1975-HA.patch, > HDFS-1975-HDFS-1623.patch, HDFS-1975-HDFS-1623.patch, > HDFS-1975-HDFS-1623.patch, hdfs-1975.txt, hdfs-1975.txt > > > To enable hot standby namenode, the standby node must have current > information for - namenode state (image + edits) and block location > information. This jira addresses keeping the namenode state current in the > standby node. To do this, the proposed solution in this jira is to use a > shared storage to store the namenode state. > Note one could also build an alternative solution by augmenting the backup > node. A seperate jira could explore this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1108) Log newly allocated blocks
[ https://issues.apache.org/jira/browse/HDFS-1108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-1108: -- Attachment: hdfs-1108-habranch.txt Updated patch on top of HDFS-2582 - same patch except for trivial change that we need to pass the namespace id to HAUtil.isHaEnabled now. After HDFS-2582 goes in, I'll commit this based on the earlier +1s from folks > Log newly allocated blocks > -- > > Key: HDFS-1108 > URL: https://issues.apache.org/jira/browse/HDFS-1108 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Reporter: dhruba borthakur >Assignee: Todd Lipcon > Fix For: HA branch (HDFS-1623) > > Attachments: HDFS-1108.patch, hdfs-1108-habranch.txt, > hdfs-1108-habranch.txt, hdfs-1108-habranch.txt, hdfs-1108.txt > > > The current HDFS design says that newly allocated blocks for a file are not > persisted in the NN transaction log when the block is allocated. Instead, a > hflush() or a close() on the file persists the blocks into the transaction > log. It would be nice if we can immediately persist newly allocated blocks > (as soon as they are allocated) for specific files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1975) HA: Support for sharing the namenode state from active to standby.
[ https://issues.apache.org/jira/browse/HDFS-1975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-1975: -- Attachment: HDFS-1975-HDFS-1623.patch Attached patch rebased on top of latest HDFS-2582, HDFS-2591, HDFS-1971, in that order. > HA: Support for sharing the namenode state from active to standby. > -- > > Key: HDFS-1975 > URL: https://issues.apache.org/jira/browse/HDFS-1975 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Reporter: Suresh Srinivas >Assignee: Jitendra Nath Pandey > Attachments: HDFS-1975-HA.2.patch, HDFS-1975-HA.patch, > HDFS-1975-HDFS-1623.patch, HDFS-1975-HDFS-1623.patch, > HDFS-1975-HDFS-1623.patch, hdfs-1975.txt, hdfs-1975.txt > > > To enable hot standby namenode, the standby node must have current > information for - namenode state (image + edits) and block location > information. This jira addresses keeping the namenode state current in the > standby node. To do this, the proposed solution in this jira is to use a > shared storage to store the namenode state. > Note one could also build an alternative solution by augmenting the backup > node. A seperate jira could explore this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1971) HA: Send block report from datanode to both active and standby namenodes
[ https://issues.apache.org/jira/browse/HDFS-1971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-1971: -- Attachment: dualbr5.txt This patch just has a few small cleanups from dualbr4.patch. It still leaves the refreshNamenodes bit TODO, and adds a few more TODOs where we have to sort out what to do in terms of HA. I think it's worth committing this now, even with the TODOs, so we can address the TODOs in parallel. It will unblock other work and allow us to start figuring out more subtle issues. I can file follow-up JIRAs for each of the TODOs when this is committed. I've been testing this patch on top of HDFS-2582 and HDFS-2591 in that order, though I think it only strictly depends on HDFS-2582 > HA: Send block report from datanode to both active and standby namenodes > > > Key: HDFS-1971 > URL: https://issues.apache.org/jira/browse/HDFS-1971 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: data-node, name-node >Reporter: Suresh Srinivas >Assignee: Sanjay Radia > Attachments: DualBlockReports.pdf, daulBr1.patch, dualBr2.patch, > dualBr3.patch, dualbr4.txt, dualbr5.txt > > > To enable hot standby namenode, the standby node must have current > information for - namenode state (image + edits) and block location > information. This jira addresses keeping the block location information > current in the standby node. To do this, the proposed solution is to send > block reports from the datanodes to both the active and the standby namenode. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2591) HA: MiniDFSCluster support to mix and match federation with HA
[ https://issues.apache.org/jira/browse/HDFS-2591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2591: -- Attachment: hdfs-2591-v2.txt v2 patch fixes a number of issues discovered trying to build HA test cases on top of this. In particular, it has code to set up shared edits directories (to be enabled by HDFS-1971), as well as initialize the SBNN by copying one of the formatted namedirs from the active NN. (otherwise the two NNs would have different nsids/bpids and not be able to failover). > HA: MiniDFSCluster support to mix and match federation with HA > -- > > Key: HDFS-2591 > URL: https://issues.apache.org/jira/browse/HDFS-2591 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, test >Affects Versions: HA branch (HDFS-1623) >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: hdfs-2591-v2.txt, hdfs-2591.txt > > > Right now the MiniDFS builder is somewhat inflexible - it just takes a > "numNameNodes" parameter, which is used to specify federated nameservices. In > order to add HA support, we need to be able to be more specific when > configuring the NNs -- eg to test the case where there is one nameservice > that is HA and another which is not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1975) HA: Support for sharing the namenode state from active to standby.
[
https://issues.apache.org/jira/browse/HDFS-1975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158069#comment-13158069
]
Todd Lipcon commented on HDFS-1975:
---
I got this patch in conjunction with HDFS-1108 and HDFS-1971 to properly
replicate the creation of a new file, but then moved on to working on
{{setReplication}} and ran into issues there. The issue I'm seeing is this:
1) Active NN receives setReplication to drop some file's replication from 3 to 1
2) It writes OP_SET_REPLICATION to its log, invalidates two replicas, and
returns
3) The DNs report BLOCK_INVALIDATED back to both the ActiveNN and SBNN.
4) The SBNN hasn't received the OP_SET_REPLICATION yet, so it marks the block
as under-replicated.
In the case of raising replication (eg from 1 to 3) we get the opposite
problem: the SBNN marks the block as over-replicated and adds two of the
replicas to its invalidation list.
Generation stamps don't help here, because changing replication level of a
block doesn't change its gen-stamp (and it shouldn't).
The solution I'm thinking of is that we have to track the transaction ID when
we send comments to DNs. So, if a setReplication command at txid=123 causes
invalidation of two blocks, we'd send the INVALIDATE command with "txid=123".
Then, when the DN does delete these blocks, it would ack back with that txid to
both NNs. The SBNN wouldn't process this message until it had loaded that txid.
A bit of a simplification from this would be that any command being processed
from an NN will include the NN's txid, which the DN records in BPOfferService
as "latestCommandTxId". Then, any calls to the NN would include this txid. This
is a bit more conservative than tracking it with each block command, but
probably less prone to bugs.
I plan to take a pass at implementing this latter approach.
> HA: Support for sharing the namenode state from active to standby.
> --
>
> Key: HDFS-1975
> URL: https://issues.apache.org/jira/browse/HDFS-1975
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: name-node
>Reporter: Suresh Srinivas
>Assignee: Jitendra Nath Pandey
> Attachments: HDFS-1975-HA.2.patch, HDFS-1975-HA.patch,
> HDFS-1975-HDFS-1623.patch, HDFS-1975-HDFS-1623.patch, hdfs-1975.txt,
> hdfs-1975.txt
>
>
> To enable hot standby namenode, the standby node must have current
> information for - namenode state (image + edits) and block location
> information. This jira addresses keeping the namenode state current in the
> standby node. To do this, the proposed solution in this jira is to use a
> shared storage to store the namenode state.
> Note one could also build an alternative solution by augmenting the backup
> node. A seperate jira could explore this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1975) HA: Support for sharing the namenode state from active to standby.
[
https://issues.apache.org/jira/browse/HDFS-1975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158067#comment-13158067
]
Todd Lipcon commented on HDFS-1975:
---
bq. How about update the generation stamp after logging the new block
The issue is that we need the new block to have the new genstamp, so we have to
call {{nextGenerationStamp}} before we create the new block. Alternatively, we
could split up {{nextGenerationStamp}} into two parts -- one which increments,
and another which logs. But then we may have an issue if there is a crash so
the edit log includes the block allocation but not the SET_GENERATION_STAMP.
I've temporarily worked around this by changing the FSEditLogLoader code to
only call {{notifyGenStampUpdate}} in {{OP_ADD}}, not in {{OP_SET_GENSTAMP}}.
So, the new block is added to the block manager before it's notified as OK to
handle the DN messages.
*BUT* -- there's another issue with using genstamps as our "gating" mechanism
for the DN messages - comment to follow.
> HA: Support for sharing the namenode state from active to standby.
> --
>
> Key: HDFS-1975
> URL: https://issues.apache.org/jira/browse/HDFS-1975
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: name-node
>Reporter: Suresh Srinivas
>Assignee: Jitendra Nath Pandey
> Attachments: HDFS-1975-HA.2.patch, HDFS-1975-HA.patch,
> HDFS-1975-HDFS-1623.patch, HDFS-1975-HDFS-1623.patch, hdfs-1975.txt,
> hdfs-1975.txt
>
>
> To enable hot standby namenode, the standby node must have current
> information for - namenode state (image + edits) and block location
> information. This jira addresses keeping the namenode state current in the
> standby node. To do this, the proposed solution in this jira is to use a
> shared storage to store the namenode state.
> Note one could also build an alternative solution by augmenting the backup
> node. A seperate jira could explore this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2582) Scope dfs.ha.namenodes config by nameservice
[
https://issues.apache.org/jira/browse/HDFS-2582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158061#comment-13158061
]
Eli Collins commented on HDFS-2582:
---
+1 to v4, the fix in the delta lgtm
> Scope dfs.ha.namenodes config by nameservice
>
>
> Key: HDFS-2582
> URL: https://issues.apache.org/jira/browse/HDFS-2582
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node, name-node
>Affects Versions: HA branch (HDFS-1623)
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
> Fix For: HA branch (HDFS-1623)
>
> Attachments: hdfs-2582-v2.txt, hdfs-2582-v3.txt, hdfs-2582-v4.txt,
> hdfs-2582.txt
>
>
> HDFS-2231 started the process of adding configuration for HA, but one piece
> is missing. The current state of the configuration is, I believe:
> {{dfs.ha.namenodes}} - a list of identifiers for HA namenodes
> {{dfs.federation.nameservices}} - a list of federated nameservices
> {{dfs.namenode.rpc-address[.nameservice-id][.namenode-id]}} - some specific
> config for the given namenode. If HA or federation is disabled, the extra
> components can be elided for backwards compatibility.
> The issue here is that there is no easy way to discern which NN is paired
> with which other NN. Additionally, adding a new federated nameservice to a
> config will require changes to {{dfs.ha.namenodes}} which makes templating
> harder. It would be simpler to change {{dfs.ha.namenodes}} to be
> nameservice-scoped: {{dfs.ha.namenodes.}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2291) HA: Checkpointing in an HA setup
[ https://issues.apache.org/jira/browse/HDFS-2291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158060#comment-13158060 ] Eli Collins commented on HDFS-2291: --- Agree that the SBN should be able to do checkpoints - someone running a typical 20x configuration with two hosts (NN and 2NN) should be able to keep the same hardware config (NN and SBN). > HA: Checkpointing in an HA setup > > > Key: HDFS-2291 > URL: https://issues.apache.org/jira/browse/HDFS-2291 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: HA branch (HDFS-1623) >Reporter: Aaron T. Myers >Assignee: Todd Lipcon > Fix For: HA branch (HDFS-1623) > > > We obviously need to create checkpoints when HA is enabled. One thought is to > use a third, dedicated checkpointing node in addition to the active and > standby nodes. Another option would be to make the standby capable of also > performing the function of checkpointing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2582) Scope dfs.ha.namenodes config by nameservice
[
https://issues.apache.org/jira/browse/HDFS-2582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated HDFS-2582:
--
Attachment: hdfs-2582-v4.txt
Here's v4, which I built a few more patches on top. One more slight fix to the
nnId detection stuff, nothing major changed.
I agree the map> is kind of clunky, but I'd like to punt to a follow-on
to clean this up, since I have some more patches built on top.
I'll commit this soon based on Eli's prior +1 unless there are any other
comments.
> Scope dfs.ha.namenodes config by nameservice
>
>
> Key: HDFS-2582
> URL: https://issues.apache.org/jira/browse/HDFS-2582
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node, name-node
>Affects Versions: HA branch (HDFS-1623)
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
> Fix For: HA branch (HDFS-1623)
>
> Attachments: hdfs-2582-v2.txt, hdfs-2582-v3.txt, hdfs-2582-v4.txt,
> hdfs-2582.txt
>
>
> HDFS-2231 started the process of adding configuration for HA, but one piece
> is missing. The current state of the configuration is, I believe:
> {{dfs.ha.namenodes}} - a list of identifiers for HA namenodes
> {{dfs.federation.nameservices}} - a list of federated nameservices
> {{dfs.namenode.rpc-address[.nameservice-id][.namenode-id]}} - some specific
> config for the given namenode. If HA or federation is disabled, the extra
> components can be elided for backwards compatibility.
> The issue here is that there is no easy way to discern which NN is paired
> with which other NN. Additionally, adding a new federated nameservice to a
> config will require changes to {{dfs.ha.namenodes}} which makes templating
> harder. It would be simpler to change {{dfs.ha.namenodes}} to be
> nameservice-scoped: {{dfs.ha.namenodes.}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2591) HA: MiniDFSCluster support to mix and match federation with HA
[ https://issues.apache.org/jira/browse/HDFS-2591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158024#comment-13158024 ] Eli Collins commented on HDFS-2591: --- What you have so far looks good. If we end up creating a class for the map in HDFS-2582 it could probably share the code here since it's mostly the same map essentially in class form. > HA: MiniDFSCluster support to mix and match federation with HA > -- > > Key: HDFS-2591 > URL: https://issues.apache.org/jira/browse/HDFS-2591 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, test >Affects Versions: HA branch (HDFS-1623) >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: hdfs-2591.txt > > > Right now the MiniDFS builder is somewhat inflexible - it just takes a > "numNameNodes" parameter, which is used to specify federated nameservices. In > order to add HA support, we need to be able to be more specific when > configuring the NNs -- eg to test the case where there is one nameservice > that is HA and another which is not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2582) Scope dfs.ha.namenodes config by nameservice
[
https://issues.apache.org/jira/browse/HDFS-2582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158019#comment-13158019
]
Eli Collins commented on HDFS-2582:
---
lgtm, +1 with the follow-on jiras.
The code that uses the new map (Map>) is a little
unwieldy since we don't have types for the ns and nn ids. Given that this is a
tiny map the readability might be worth the overhead of a class (eg in the same
vain of ConfiguredNNAddress). Don't feel strongly so feel free to punt if you
disagree.
> Scope dfs.ha.namenodes config by nameservice
>
>
> Key: HDFS-2582
> URL: https://issues.apache.org/jira/browse/HDFS-2582
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: data-node, name-node
>Affects Versions: HA branch (HDFS-1623)
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
> Fix For: HA branch (HDFS-1623)
>
> Attachments: hdfs-2582-v2.txt, hdfs-2582-v3.txt, hdfs-2582.txt
>
>
> HDFS-2231 started the process of adding configuration for HA, but one piece
> is missing. The current state of the configuration is, I believe:
> {{dfs.ha.namenodes}} - a list of identifiers for HA namenodes
> {{dfs.federation.nameservices}} - a list of federated nameservices
> {{dfs.namenode.rpc-address[.nameservice-id][.namenode-id]}} - some specific
> config for the given namenode. If HA or federation is disabled, the extra
> components can be elided for backwards compatibility.
> The issue here is that there is no easy way to discern which NN is paired
> with which other NN. Additionally, adding a new federated nameservice to a
> config will require changes to {{dfs.ha.namenodes}} which makes templating
> harder. It would be simpler to change {{dfs.ha.namenodes}} to be
> nameservice-scoped: {{dfs.ha.namenodes.}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2595) misleading message when fs.default.name not set and running sbin/start-dfs.sh
[
https://issues.apache.org/jira/browse/HDFS-2595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13157903#comment-13157903
]
Joe Crobak commented on HDFS-2595:
--
One possible solution is to change the text of the exception to include a
message about fs.default.name in DfsUtils#getNNServiceRpcAddresses(). The
catch clause for the IAE when computing default address could log a warning
about fs.default.name not being set correctly (similar to previous versions of
hadoop).
> misleading message when fs.default.name not set and running sbin/start-dfs.sh
> -
>
> Key: HDFS-2595
> URL: https://issues.apache.org/jira/browse/HDFS-2595
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
>Affects Versions: 0.23.0
>Reporter: Joe Crobak
>Priority: Minor
>
> sbin/start-dfs.sh calls {{bin/hdfs getconf -namenodes}} which reports the
> following error if the default value of fs.default.name from core-default.xml
> is used: {noformat}Incorrect configuration: namenode address
> dfs.namenode.servicerpc-address or dfs.namenode.rpc-address is not
> configured.{noformat}
> In previous versions of hadoop, I would see a slightly more helpful message:
> {noformat}org.apache.hadoop.hdfs.server.namenode.NameNode:
> java.lang.IllegalArgumentException: Does not contain a valid host:port
> authority: file:///{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2595) misleading message when fs.default.name not set and running sbin/start-dfs.sh
misleading message when fs.default.name not set and running sbin/start-dfs.sh
-
Key: HDFS-2595
URL: https://issues.apache.org/jira/browse/HDFS-2595
Project: Hadoop HDFS
Issue Type: Improvement
Components: name-node
Affects Versions: 0.23.0
Reporter: Joe Crobak
Priority: Minor
sbin/start-dfs.sh calls {{bin/hdfs getconf -namenodes}} which reports the
following error if the default value of fs.default.name from core-default.xml
is used: {noformat}Incorrect configuration: namenode address
dfs.namenode.servicerpc-address or dfs.namenode.rpc-address is not
configured.{noformat}
In previous versions of hadoop, I would see a slightly more helpful message:
{noformat}org.apache.hadoop.hdfs.server.namenode.NameNode:
java.lang.IllegalArgumentException: Does not contain a valid host:port
authority: file:///{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
