[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14252063#comment-14252063 ] Ming Ma commented on HDFS-5535: --- Opened https://issues.apache.org/jira/browse/HDFS-7541 to explore ideas for more efficient DN rolling upgrades. > Umbrella jira for improved HDFS rolling upgrades > > > Key: HDFS-5535 > URL: https://issues.apache.org/jira/browse/HDFS-5535 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, ha, hdfs-client, namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Nathan Roberts >Assignee: Tsz Wo Nicholas Sze > Fix For: 2.4.0 > > Attachments: HDFSRollingUpgradesHighLevelDesign.pdf, > HDFSRollingUpgradesHighLevelDesign.v2.pdf, > HDFSRollingUpgradesHighLevelDesign.v3.pdf, h5535_20140219.patch, > h5535_20140220-1554.patch, h5535_20140220b.patch, h5535_20140221-2031.patch, > h5535_20140224-1931.patch, h5535_20140225-1225.patch, > h5535_20140226-1328.patch, h5535_20140226-1911.patch, > h5535_20140227-1239.patch, h5535_20140228-1714.patch, > h5535_20140304-1138.patch, h5535_20140304-branch-2.patch, > h5535_20140310-branch-2.patch, hdfs-5535-test-plan.pdf > > > In order to roll a new HDFS release through a large cluster quickly and > safely, a few enhancements are needed in HDFS. An initial High level design > document will be attached to this jira, and sub-jiras will itemize the > individual tasks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14167403#comment-14167403 ] Tsz Wo Nicholas Sze commented on HDFS-5535: --- FYI: just have filed HDFS-7230 for discussing how to support rolling downgrade. > Umbrella jira for improved HDFS rolling upgrades > > > Key: HDFS-5535 > URL: https://issues.apache.org/jira/browse/HDFS-5535 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, ha, hdfs-client, namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Nathan Roberts >Assignee: Tsz Wo Nicholas Sze > Fix For: 2.4.0 > > Attachments: HDFSRollingUpgradesHighLevelDesign.pdf, > HDFSRollingUpgradesHighLevelDesign.v2.pdf, > HDFSRollingUpgradesHighLevelDesign.v3.pdf, h5535_20140219.patch, > h5535_20140220-1554.patch, h5535_20140220b.patch, h5535_20140221-2031.patch, > h5535_20140224-1931.patch, h5535_20140225-1225.patch, > h5535_20140226-1328.patch, h5535_20140226-1911.patch, > h5535_20140227-1239.patch, h5535_20140228-1714.patch, > h5535_20140304-1138.patch, h5535_20140304-branch-2.patch, > h5535_20140310-branch-2.patch, hdfs-5535-test-plan.pdf > > > In order to roll a new HDFS release through a large cluster quickly and > safely, a few enhancements are needed in HDFS. An initial High level design > document will be attached to this jira, and sub-jiras will itemize the > individual tasks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14076114#comment-14076114 ] Patrick Morton commented on HDFS-5535: -- Dausey has used people to evaluate obsessive time shootings and systems between superior bean costs and their music neurons. http://www.surveyanalytics.com//userimages/sub-2/2007589/3153260/29851518/7787457-29851518-stopadd43.html Bands, from new york city, then practiced a wealth fact addition, only burnt on historical gatekeeper imprisonment. > Umbrella jira for improved HDFS rolling upgrades > > > Key: HDFS-5535 > URL: https://issues.apache.org/jira/browse/HDFS-5535 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, ha, hdfs-client, namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Nathan Roberts >Assignee: Tsz Wo Nicholas Sze > Fix For: 2.4.0 > > Attachments: HDFSRollingUpgradesHighLevelDesign.pdf, > HDFSRollingUpgradesHighLevelDesign.v2.pdf, > HDFSRollingUpgradesHighLevelDesign.v3.pdf, h5535_20140219.patch, > h5535_20140220-1554.patch, h5535_20140220b.patch, h5535_20140221-2031.patch, > h5535_20140224-1931.patch, h5535_20140225-1225.patch, > h5535_20140226-1328.patch, h5535_20140226-1911.patch, > h5535_20140227-1239.patch, h5535_20140228-1714.patch, > h5535_20140304-1138.patch, h5535_20140304-branch-2.patch, > h5535_20140310-branch-2.patch, hdfs-5535-test-plan.pdf > > > In order to roll a new HDFS release through a large cluster quickly and > safely, a few enhancements are needed in HDFS. An initial High level design > document will be attached to this jira, and sub-jiras will itemize the > individual tasks. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13930419#comment-13930419 ] Hudson commented on HDFS-5535: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1723 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1723/]) Move HDFS-5535 to Release 2.4.0 in CHANGES.txt. (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1576148) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > Umbrella jira for improved HDFS rolling upgrades > > > Key: HDFS-5535 > URL: https://issues.apache.org/jira/browse/HDFS-5535 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, ha, hdfs-client, namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Nathan Roberts >Assignee: Tsz Wo Nicholas Sze > Fix For: 2.4.0 > > Attachments: HDFSRollingUpgradesHighLevelDesign.pdf, > HDFSRollingUpgradesHighLevelDesign.v2.pdf, > HDFSRollingUpgradesHighLevelDesign.v3.pdf, h5535_20140219.patch, > h5535_20140220-1554.patch, h5535_20140220b.patch, h5535_20140221-2031.patch, > h5535_20140224-1931.patch, h5535_20140225-1225.patch, > h5535_20140226-1328.patch, h5535_20140226-1911.patch, > h5535_20140227-1239.patch, h5535_20140228-1714.patch, > h5535_20140304-1138.patch, h5535_20140304-branch-2.patch, > h5535_20140310-branch-2.patch, hdfs-5535-test-plan.pdf > > > In order to roll a new HDFS release through a large cluster quickly and > safely, a few enhancements are needed in HDFS. An initial High level design > document will be attached to this jira, and sub-jiras will itemize the > individual tasks. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13930348#comment-13930348 ] Hudson commented on HDFS-5535: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1698 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1698/]) Move HDFS-5535 to Release 2.4.0 in CHANGES.txt. (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1576148) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > Umbrella jira for improved HDFS rolling upgrades > > > Key: HDFS-5535 > URL: https://issues.apache.org/jira/browse/HDFS-5535 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, ha, hdfs-client, namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Nathan Roberts >Assignee: Tsz Wo Nicholas Sze > Fix For: 2.4.0 > > Attachments: HDFSRollingUpgradesHighLevelDesign.pdf, > HDFSRollingUpgradesHighLevelDesign.v2.pdf, > HDFSRollingUpgradesHighLevelDesign.v3.pdf, h5535_20140219.patch, > h5535_20140220-1554.patch, h5535_20140220b.patch, h5535_20140221-2031.patch, > h5535_20140224-1931.patch, h5535_20140225-1225.patch, > h5535_20140226-1328.patch, h5535_20140226-1911.patch, > h5535_20140227-1239.patch, h5535_20140228-1714.patch, > h5535_20140304-1138.patch, h5535_20140304-branch-2.patch, > h5535_20140310-branch-2.patch, hdfs-5535-test-plan.pdf > > > In order to roll a new HDFS release through a large cluster quickly and > safely, a few enhancements are needed in HDFS. An initial High level design > document will be attached to this jira, and sub-jiras will itemize the > individual tasks. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13930225#comment-13930225 ] Hudson commented on HDFS-5535: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #506 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/506/]) Move HDFS-5535 to Release 2.4.0 in CHANGES.txt. (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1576148) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > Umbrella jira for improved HDFS rolling upgrades > > > Key: HDFS-5535 > URL: https://issues.apache.org/jira/browse/HDFS-5535 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, ha, hdfs-client, namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Nathan Roberts >Assignee: Tsz Wo Nicholas Sze > Fix For: 2.4.0 > > Attachments: HDFSRollingUpgradesHighLevelDesign.pdf, > HDFSRollingUpgradesHighLevelDesign.v2.pdf, > HDFSRollingUpgradesHighLevelDesign.v3.pdf, h5535_20140219.patch, > h5535_20140220-1554.patch, h5535_20140220b.patch, h5535_20140221-2031.patch, > h5535_20140224-1931.patch, h5535_20140225-1225.patch, > h5535_20140226-1328.patch, h5535_20140226-1911.patch, > h5535_20140227-1239.patch, h5535_20140228-1714.patch, > h5535_20140304-1138.patch, h5535_20140304-branch-2.patch, > h5535_20140310-branch-2.patch, hdfs-5535-test-plan.pdf > > > In order to roll a new HDFS release through a large cluster quickly and > safely, a few enhancements are needed in HDFS. An initial High level design > document will be attached to this jira, and sub-jiras will itemize the > individual tasks. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13929854#comment-13929854 ] Hudson commented on HDFS-5535: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5302 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5302/]) Move HDFS-5535 to Release 2.4.0 in CHANGES.txt. (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1576148) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > Umbrella jira for improved HDFS rolling upgrades > > > Key: HDFS-5535 > URL: https://issues.apache.org/jira/browse/HDFS-5535 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, ha, hdfs-client, namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Nathan Roberts >Assignee: Tsz Wo Nicholas Sze > Fix For: 2.4.0 > > Attachments: HDFSRollingUpgradesHighLevelDesign.pdf, > HDFSRollingUpgradesHighLevelDesign.v2.pdf, > HDFSRollingUpgradesHighLevelDesign.v3.pdf, h5535_20140219.patch, > h5535_20140220-1554.patch, h5535_20140220b.patch, h5535_20140221-2031.patch, > h5535_20140224-1931.patch, h5535_20140225-1225.patch, > h5535_20140226-1328.patch, h5535_20140226-1911.patch, > h5535_20140227-1239.patch, h5535_20140228-1714.patch, > h5535_20140304-1138.patch, h5535_20140304-branch-2.patch, > h5535_20140310-branch-2.patch, hdfs-5535-test-plan.pdf > > > In order to roll a new HDFS release through a large cluster quickly and > safely, a few enhancements are needed in HDFS. An initial High level design > document will be attached to this jira, and sub-jiras will itemize the > individual tasks. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13926297#comment-13926297 ] Tsz Wo Nicholas Sze commented on HDFS-5535: --- > ... I assume CHANGES.txt will be manually merged and CHANGES_HDFS-5535.txt > be removed. Yes, I will update CHANGES.txt as well as editsStored manually. Will merge it shortly. > Umbrella jira for improved HDFS rolling upgrades > > > Key: HDFS-5535 > URL: https://issues.apache.org/jira/browse/HDFS-5535 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, ha, hdfs-client, namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Nathan Roberts > Attachments: HDFSRollingUpgradesHighLevelDesign.pdf, > HDFSRollingUpgradesHighLevelDesign.v2.pdf, > HDFSRollingUpgradesHighLevelDesign.v3.pdf, h5535_20140219.patch, > h5535_20140220-1554.patch, h5535_20140220b.patch, h5535_20140221-2031.patch, > h5535_20140224-1931.patch, h5535_20140225-1225.patch, > h5535_20140226-1328.patch, h5535_20140226-1911.patch, > h5535_20140227-1239.patch, h5535_20140228-1714.patch, > h5535_20140304-1138.patch, h5535_20140304-branch-2.patch, > h5535_20140310-branch-2.patch, hdfs-5535-test-plan.pdf > > > In order to roll a new HDFS release through a large cluster quickly and > safely, a few enhancements are needed in HDFS. An initial High level design > document will be attached to this jira, and sub-jiras will itemize the > individual tasks. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13926234#comment-13926234 ] Kihwal Lee commented on HDFS-5535: -- The new patch looks good. I compare it against the trunk version and also ran several related test cases. +1 for using it for branch-2 merging. I assume CHANGES.txt will be manually merged and CHANGES_HDFS-5535.txt be removed. > Umbrella jira for improved HDFS rolling upgrades > > > Key: HDFS-5535 > URL: https://issues.apache.org/jira/browse/HDFS-5535 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, ha, hdfs-client, namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Nathan Roberts > Attachments: HDFSRollingUpgradesHighLevelDesign.pdf, > HDFSRollingUpgradesHighLevelDesign.v2.pdf, > HDFSRollingUpgradesHighLevelDesign.v3.pdf, h5535_20140219.patch, > h5535_20140220-1554.patch, h5535_20140220b.patch, h5535_20140221-2031.patch, > h5535_20140224-1931.patch, h5535_20140225-1225.patch, > h5535_20140226-1328.patch, h5535_20140226-1911.patch, > h5535_20140227-1239.patch, h5535_20140228-1714.patch, > h5535_20140304-1138.patch, h5535_20140304-branch-2.patch, > h5535_20140310-branch-2.patch, hdfs-5535-test-plan.pdf > > > In order to roll a new HDFS release through a large cluster quickly and > safely, a few enhancements are needed in HDFS. An initial High level design > document will be attached to this jira, and sub-jiras will itemize the > individual tasks. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[
https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13925750#comment-13925750
]
Kihwal Lee commented on HDFS-5535:
--
HDFS-5064 got merged to branch-2 and the patch won't compile. In
{{testReadsAllowedDuringCheckpoint()}}, the call to {{saveNamespace()}} is
missing one argument. Otherwise the patch looks good.
> Umbrella jira for improved HDFS rolling upgrades
>
>
> Key: HDFS-5535
> URL: https://issues.apache.org/jira/browse/HDFS-5535
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, ha, hdfs-client, namenode
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Nathan Roberts
> Attachments: HDFSRollingUpgradesHighLevelDesign.pdf,
> HDFSRollingUpgradesHighLevelDesign.v2.pdf,
> HDFSRollingUpgradesHighLevelDesign.v3.pdf, h5535_20140219.patch,
> h5535_20140220-1554.patch, h5535_20140220b.patch, h5535_20140221-2031.patch,
> h5535_20140224-1931.patch, h5535_20140225-1225.patch,
> h5535_20140226-1328.patch, h5535_20140226-1911.patch,
> h5535_20140227-1239.patch, h5535_20140228-1714.patch,
> h5535_20140304-1138.patch, h5535_20140304-branch-2.patch,
> hdfs-5535-test-plan.pdf
>
>
> In order to roll a new HDFS release through a large cluster quickly and
> safely, a few enhancements are needed in HDFS. An initial High level design
> document will be attached to this jira, and sub-jiras will itemize the
> individual tasks.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[
https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13924696#comment-13924696
]
Hadoop QA commented on HDFS-5535:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12633522/h5535_20140304-branch-2.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6354//console
This message is automatically generated.
> Umbrella jira for improved HDFS rolling upgrades
>
>
> Key: HDFS-5535
> URL: https://issues.apache.org/jira/browse/HDFS-5535
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, ha, hdfs-client, namenode
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Nathan Roberts
> Attachments: HDFSRollingUpgradesHighLevelDesign.pdf,
> HDFSRollingUpgradesHighLevelDesign.v2.pdf,
> HDFSRollingUpgradesHighLevelDesign.v3.pdf, h5535_20140219.patch,
> h5535_20140220-1554.patch, h5535_20140220b.patch, h5535_20140221-2031.patch,
> h5535_20140224-1931.patch, h5535_20140225-1225.patch,
> h5535_20140226-1328.patch, h5535_20140226-1911.patch,
> h5535_20140227-1239.patch, h5535_20140228-1714.patch,
> h5535_20140304-1138.patch, h5535_20140304-branch-2.patch,
> hdfs-5535-test-plan.pdf
>
>
> In order to roll a new HDFS release through a large cluster quickly and
> safely, a few enhancements are needed in HDFS. An initial High level design
> document will be attached to this jira, and sub-jiras will itemize the
> individual tasks.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13924625#comment-13924625 ] Tsz Wo Nicholas Sze commented on HDFS-5535: --- We don't really need to wait for HDFS-5138 and HDFS-5840. I will post a merge patch. > Umbrella jira for improved HDFS rolling upgrades > > > Key: HDFS-5535 > URL: https://issues.apache.org/jira/browse/HDFS-5535 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, ha, hdfs-client, namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Nathan Roberts > Attachments: HDFSRollingUpgradesHighLevelDesign.pdf, > HDFSRollingUpgradesHighLevelDesign.v2.pdf, > HDFSRollingUpgradesHighLevelDesign.v3.pdf, h5535_20140219.patch, > h5535_20140220-1554.patch, h5535_20140220b.patch, h5535_20140221-2031.patch, > h5535_20140224-1931.patch, h5535_20140225-1225.patch, > h5535_20140226-1328.patch, h5535_20140226-1911.patch, > h5535_20140227-1239.patch, h5535_20140228-1714.patch, > h5535_20140304-1138.patch, hdfs-5535-test-plan.pdf > > > In order to roll a new HDFS release through a large cluster quickly and > safely, a few enhancements are needed in HDFS. An initial High level design > document will be attached to this jira, and sub-jiras will itemize the > individual tasks. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13922630#comment-13922630 ] Kihwal Lee commented on HDFS-5535: -- Are we waiting for HDFS-5131 and HDFS-5840 before merging to branch-2 and branch-2.4? > Umbrella jira for improved HDFS rolling upgrades > > > Key: HDFS-5535 > URL: https://issues.apache.org/jira/browse/HDFS-5535 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, ha, hdfs-client, namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Nathan Roberts > Attachments: HDFSRollingUpgradesHighLevelDesign.pdf, > HDFSRollingUpgradesHighLevelDesign.v2.pdf, > HDFSRollingUpgradesHighLevelDesign.v3.pdf, h5535_20140219.patch, > h5535_20140220-1554.patch, h5535_20140220b.patch, h5535_20140221-2031.patch, > h5535_20140224-1931.patch, h5535_20140225-1225.patch, > h5535_20140226-1328.patch, h5535_20140226-1911.patch, > h5535_20140227-1239.patch, h5535_20140228-1714.patch, > h5535_20140304-1138.patch, hdfs-5535-test-plan.pdf > > > In order to roll a new HDFS release through a large cluster quickly and > safely, a few enhancements are needed in HDFS. An initial High level design > document will be attached to this jira, and sub-jiras will itemize the > individual tasks. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[
https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13920990#comment-13920990
]
Hadoop QA commented on HDFS-5535:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12632843/HDFSRollingUpgradesHighLevelDesign.v3.pdf
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6307//console
This message is automatically generated.
> Umbrella jira for improved HDFS rolling upgrades
>
>
> Key: HDFS-5535
> URL: https://issues.apache.org/jira/browse/HDFS-5535
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, ha, hdfs-client, namenode
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Nathan Roberts
> Attachments: HDFSRollingUpgradesHighLevelDesign.pdf,
> HDFSRollingUpgradesHighLevelDesign.v2.pdf,
> HDFSRollingUpgradesHighLevelDesign.v3.pdf, h5535_20140219.patch,
> h5535_20140220-1554.patch, h5535_20140220b.patch, h5535_20140221-2031.patch,
> h5535_20140224-1931.patch, h5535_20140225-1225.patch,
> h5535_20140226-1328.patch, h5535_20140226-1911.patch,
> h5535_20140227-1239.patch, h5535_20140228-1714.patch,
> h5535_20140304-1138.patch, hdfs-5535-test-plan.pdf
>
>
> In order to roll a new HDFS release through a large cluster quickly and
> safely, a few enhancements are needed in HDFS. An initial High level design
> document will be attached to this jira, and sub-jiras will itemize the
> individual tasks.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13920911#comment-13920911 ] Hudson commented on HDFS-5535: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1717 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1717/]) svn merge --reintegrate https://svn.apache.org/repos/asf/hadoop/common/branches/HDFS-5535 back to trunk. (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1574259) * /hadoop/common/trunk * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/docs * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightGSet.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/core * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperJournalManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperEditLogStreams.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeLocalInfo.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LayoutVersion.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/RollingUpgradeException.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/RollingUpgradeInfo.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/RollingUpgradeStatus.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/PipelineAck.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/AsyncLogger.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/AsyncLoggerSet.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/IPCLoggerChannel.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/protocol/QJournalProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/protocolPB/QJournalProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjou
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13920838#comment-13920838 ] Hudson commented on HDFS-5535: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1692 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1692/]) svn merge --reintegrate https://svn.apache.org/repos/asf/hadoop/common/branches/HDFS-5535 back to trunk. (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1574259) * /hadoop/common/trunk * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/docs * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightGSet.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/core * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperJournalManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperEditLogStreams.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeLocalInfo.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LayoutVersion.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/RollingUpgradeException.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/RollingUpgradeInfo.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/RollingUpgradeStatus.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/PipelineAck.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/AsyncLogger.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/AsyncLoggerSet.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/IPCLoggerChannel.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/protocol/QJournalProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/protocolPB/QJournalProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/proto
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13920745#comment-13920745 ] Hudson commented on HDFS-5535: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #500 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/500/]) svn merge --reintegrate https://svn.apache.org/repos/asf/hadoop/common/branches/HDFS-5535 back to trunk. (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1574259) * /hadoop/common/trunk * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/docs * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightGSet.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/core * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperJournalManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperEditLogStreams.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeLocalInfo.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LayoutVersion.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/RollingUpgradeException.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/RollingUpgradeInfo.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/RollingUpgradeStatus.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/PipelineAck.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/AsyncLogger.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/AsyncLoggerSet.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/IPCLoggerChannel.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/protocol/QJournalProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/protocolPB/QJournalProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/protoco
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13920303#comment-13920303 ] Hudson commented on HDFS-5535: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5263 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5263/]) svn merge --reintegrate https://svn.apache.org/repos/asf/hadoop/common/branches/HDFS-5535 back to trunk. (szetszwo: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1574259) * /hadoop/common/trunk * /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/docs * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/LightWeightGSet.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/core * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperJournalManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperEditLogStreams.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/DatanodeLocalInfo.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/HdfsConstants.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/LayoutVersion.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/RollingUpgradeException.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/RollingUpgradeInfo.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/RollingUpgradeStatus.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/PipelineAck.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/AsyncLogger.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/AsyncLoggerSet.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/IPCLoggerChannel.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/protocol/QJournalProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/protocolPB/QJournalProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/p
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13920252#comment-13920252 ] Tsz Wo (Nicholas), SZE commented on HDFS-5535: -- I have merged HDFS-5535 branch back to trunk. Thanks everyone who has worked on this! > Umbrella jira for improved HDFS rolling upgrades > > > Key: HDFS-5535 > URL: https://issues.apache.org/jira/browse/HDFS-5535 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, ha, hdfs-client, namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Nathan Roberts > Attachments: HDFSRollingUpgradesHighLevelDesign.pdf, > HDFSRollingUpgradesHighLevelDesign.v2.pdf, h5535_20140219.patch, > h5535_20140220-1554.patch, h5535_20140220b.patch, h5535_20140221-2031.patch, > h5535_20140224-1931.patch, h5535_20140225-1225.patch, > h5535_20140226-1328.patch, h5535_20140226-1911.patch, > h5535_20140227-1239.patch, h5535_20140228-1714.patch, > h5535_20140304-1138.patch, hdfs-5535-test-plan.pdf > > > In order to roll a new HDFS release through a large cluster quickly and > safely, a few enhancements are needed in HDFS. An initial High level design > document will be attached to this jira, and sub-jiras will itemize the > individual tasks. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13920230#comment-13920230 ] Tsz Wo (Nicholas), SZE commented on HDFS-5535: -- TestOfflineEditsViewer need the new editsStored file. The other failed tests does not seem related to the patch. I tried them in my machine and they all passed. > Umbrella jira for improved HDFS rolling upgrades > > > Key: HDFS-5535 > URL: https://issues.apache.org/jira/browse/HDFS-5535 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, ha, hdfs-client, namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Nathan Roberts > Attachments: HDFSRollingUpgradesHighLevelDesign.pdf, > HDFSRollingUpgradesHighLevelDesign.v2.pdf, h5535_20140219.patch, > h5535_20140220-1554.patch, h5535_20140220b.patch, h5535_20140221-2031.patch, > h5535_20140224-1931.patch, h5535_20140225-1225.patch, > h5535_20140226-1328.patch, h5535_20140226-1911.patch, > h5535_20140227-1239.patch, h5535_20140228-1714.patch, > h5535_20140304-1138.patch, hdfs-5535-test-plan.pdf > > > In order to roll a new HDFS release through a large cluster quickly and > safely, a few enhancements are needed in HDFS. An initial High level design > document will be attached to this jira, and sub-jiras will itemize the > individual tasks. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[
https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13920188#comment-13920188
]
Hadoop QA commented on HDFS-5535:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12632582/h5535_20140304-1138.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 42 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:
org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl
org.apache.hadoop.hdfs.TestPread
org.apache.hadoop.hdfs.server.namenode.ha.TestHASafeMode
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
org.apache.hadoop.cli.TestAclCLI
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6300//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/6300//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6300//console
This message is automatically generated.
> Umbrella jira for improved HDFS rolling upgrades
>
>
> Key: HDFS-5535
> URL: https://issues.apache.org/jira/browse/HDFS-5535
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, ha, hdfs-client, namenode
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Nathan Roberts
> Attachments: HDFSRollingUpgradesHighLevelDesign.pdf,
> HDFSRollingUpgradesHighLevelDesign.v2.pdf, h5535_20140219.patch,
> h5535_20140220-1554.patch, h5535_20140220b.patch, h5535_20140221-2031.patch,
> h5535_20140224-1931.patch, h5535_20140225-1225.patch,
> h5535_20140226-1328.patch, h5535_20140226-1911.patch,
> h5535_20140227-1239.patch, h5535_20140228-1714.patch,
> h5535_20140304-1138.patch, hdfs-5535-test-plan.pdf
>
>
> In order to roll a new HDFS release through a large cluster quickly and
> safely, a few enhancements are needed in HDFS. An initial High level design
> document will be attached to this jira, and sub-jiras will itemize the
> individual tasks.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[
https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918543#comment-13918543
]
Hadoop QA commented on HDFS-5535:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12632351/HDFSRollingUpgradesHighLevelDesign.v2.pdf
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6280//console
This message is automatically generated.
> Umbrella jira for improved HDFS rolling upgrades
>
>
> Key: HDFS-5535
> URL: https://issues.apache.org/jira/browse/HDFS-5535
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, ha, hdfs-client, namenode
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Nathan Roberts
> Attachments: HDFSRollingUpgradesHighLevelDesign.pdf,
> HDFSRollingUpgradesHighLevelDesign.v2.pdf, h5535_20140219.patch,
> h5535_20140220-1554.patch, h5535_20140220b.patch, h5535_20140221-2031.patch,
> h5535_20140224-1931.patch, h5535_20140225-1225.patch,
> h5535_20140226-1328.patch, h5535_20140226-1911.patch,
> h5535_20140227-1239.patch, h5535_20140228-1714.patch, hdfs-5535-test-plan.pdf
>
>
> In order to roll a new HDFS release through a large cluster quickly and
> safely, a few enhancements are needed in HDFS. An initial High level design
> document will be attached to this jira, and sub-jiras will itemize the
> individual tasks.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[
https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13916783#comment-13916783
]
Hadoop QA commented on HDFS-5535:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12631886/h5535_20140228-1714.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 42 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6271//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/6271//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6271//console
This message is automatically generated.
> Umbrella jira for improved HDFS rolling upgrades
>
>
> Key: HDFS-5535
> URL: https://issues.apache.org/jira/browse/HDFS-5535
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, ha, hdfs-client, namenode
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Nathan Roberts
> Attachments: HDFSRollingUpgradesHighLevelDesign.pdf,
> h5535_20140219.patch, h5535_20140220-1554.patch, h5535_20140220b.patch,
> h5535_20140221-2031.patch, h5535_20140224-1931.patch,
> h5535_20140225-1225.patch, h5535_20140226-1328.patch,
> h5535_20140226-1911.patch, h5535_20140227-1239.patch,
> h5535_20140228-1714.patch, hdfs-5535-test-plan.pdf
>
>
> In order to roll a new HDFS release through a large cluster quickly and
> safely, a few enhancements are needed in HDFS. An initial High level design
> document will be attached to this jira, and sub-jiras will itemize the
> individual tasks.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[
https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13915224#comment-13915224
]
Hadoop QA commented on HDFS-5535:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12631598/h5535_20140227-1239.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 42 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6258//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/6258//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6258//console
This message is automatically generated.
> Umbrella jira for improved HDFS rolling upgrades
>
>
> Key: HDFS-5535
> URL: https://issues.apache.org/jira/browse/HDFS-5535
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, ha, hdfs-client, namenode
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Nathan Roberts
> Attachments: HDFSRollingUpgradesHighLevelDesign.pdf,
> h5535_20140219.patch, h5535_20140220-1554.patch, h5535_20140220b.patch,
> h5535_20140221-2031.patch, h5535_20140224-1931.patch,
> h5535_20140225-1225.patch, h5535_20140226-1328.patch,
> h5535_20140226-1911.patch, h5535_20140227-1239.patch, hdfs-5535-test-plan.pdf
>
>
> In order to roll a new HDFS release through a large cluster quickly and
> safely, a few enhancements are needed in HDFS. An initial High level design
> document will be attached to this jira, and sub-jiras will itemize the
> individual tasks.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[
https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13914136#comment-13914136
]
Hadoop QA commented on HDFS-5535:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12631445/h5535_20140226-1911.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 41 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6251//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/6251//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/6251//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6251//console
This message is automatically generated.
> Umbrella jira for improved HDFS rolling upgrades
>
>
> Key: HDFS-5535
> URL: https://issues.apache.org/jira/browse/HDFS-5535
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, ha, hdfs-client, namenode
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Nathan Roberts
> Attachments: HDFSRollingUpgradesHighLevelDesign.pdf,
> h5535_20140219.patch, h5535_20140220-1554.patch, h5535_20140220b.patch,
> h5535_20140221-2031.patch, h5535_20140224-1931.patch,
> h5535_20140225-1225.patch, h5535_20140226-1328.patch,
> h5535_20140226-1911.patch, hdfs-5535-test-plan.pdf
>
>
> In order to roll a new HDFS release through a large cluster quickly and
> safely, a few enhancements are needed in HDFS. An initial High level design
> document will be attached to this jira, and sub-jiras will itemize the
> individual tasks.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13913959#comment-13913959 ] Jing Zhao commented on HDFS-5535: - Filed HDFS-6026 to fix TestDataNodeRollingUpgrade and TestDFSUpgrade. TestSafeMode and TestPread seem to be some flaky unit tests and should be unrelated. TestOfflineEditsViewer needs the binary update. > Umbrella jira for improved HDFS rolling upgrades > > > Key: HDFS-5535 > URL: https://issues.apache.org/jira/browse/HDFS-5535 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, ha, hdfs-client, namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Nathan Roberts > Attachments: HDFSRollingUpgradesHighLevelDesign.pdf, > h5535_20140219.patch, h5535_20140220-1554.patch, h5535_20140220b.patch, > h5535_20140221-2031.patch, h5535_20140224-1931.patch, > h5535_20140225-1225.patch, h5535_20140226-1328.patch, hdfs-5535-test-plan.pdf > > > In order to roll a new HDFS release through a large cluster quickly and > safely, a few enhancements are needed in HDFS. An initial High level design > document will be attached to this jira, and sub-jiras will itemize the > individual tasks. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13913831#comment-13913831 ] Fengdong Yu commented on HDFS-5535: --- Thanks for the test plan, I'll start test on our dev cluster. > Umbrella jira for improved HDFS rolling upgrades > > > Key: HDFS-5535 > URL: https://issues.apache.org/jira/browse/HDFS-5535 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, ha, hdfs-client, namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Nathan Roberts > Attachments: HDFSRollingUpgradesHighLevelDesign.pdf, > h5535_20140219.patch, h5535_20140220-1554.patch, h5535_20140220b.patch, > h5535_20140221-2031.patch, h5535_20140224-1931.patch, > h5535_20140225-1225.patch, h5535_20140226-1328.patch, hdfs-5535-test-plan.pdf > > > In order to roll a new HDFS release through a large cluster quickly and > safely, a few enhancements are needed in HDFS. An initial High level design > document will be attached to this jira, and sub-jiras will itemize the > individual tasks. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[
https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13913801#comment-13913801
]
Hadoop QA commented on HDFS-5535:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12631351/h5535_20140226-1328.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 41 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:
org.apache.hadoop.hdfs.TestSafeMode
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
org.apache.hadoop.hdfs.TestPread
org.apache.hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade
org.apache.hadoop.hdfs.TestDFSUpgrade
The test build failed in
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6248//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/6248//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/6248//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6248//console
This message is automatically generated.
> Umbrella jira for improved HDFS rolling upgrades
>
>
> Key: HDFS-5535
> URL: https://issues.apache.org/jira/browse/HDFS-5535
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, ha, hdfs-client, namenode
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Nathan Roberts
> Attachments: HDFSRollingUpgradesHighLevelDesign.pdf,
> h5535_20140219.patch, h5535_20140220-1554.patch, h5535_20140220b.patch,
> h5535_20140221-2031.patch, h5535_20140224-1931.patch,
> h5535_20140225-1225.patch, h5535_20140226-1328.patch, hdfs-5535-test-plan.pdf
>
>
> In order to roll a new HDFS release through a large cluster quickly and
> safely, a few enhancements are needed in HDFS. An initial High level design
> document will be attached to this jira, and sub-jiras will itemize the
> individual tasks.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[
https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13913517#comment-13913517
]
Hadoop QA commented on HDFS-5535:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12631348/hdfs-5535-test-plan.pdf
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6247//console
This message is automatically generated.
> Umbrella jira for improved HDFS rolling upgrades
>
>
> Key: HDFS-5535
> URL: https://issues.apache.org/jira/browse/HDFS-5535
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, ha, hdfs-client, namenode
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Nathan Roberts
> Attachments: HDFSRollingUpgradesHighLevelDesign.pdf,
> h5535_20140219.patch, h5535_20140220-1554.patch, h5535_20140220b.patch,
> h5535_20140221-2031.patch, h5535_20140224-1931.patch,
> h5535_20140225-1225.patch, h5535_20140226-1256.patch, hdfs-5535-test-plan.pdf
>
>
> In order to roll a new HDFS release through a large cluster quickly and
> safely, a few enhancements are needed in HDFS. An initial High level design
> document will be attached to this jira, and sub-jiras will itemize the
> individual tasks.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[
https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13913361#comment-13913361
]
Colin Patrick McCabe commented on HDFS-5535:
bq. Stack said: Chatting w/ Colin too, it sound like SSR, if it fails a local
read, it will then retry the local read again after some number of minutes have
elapsed.
Yeah. The {{DomainSocketFactory}} has a blacklist of domain socket paths, but
they expire after 10 minutes.
bq. Kihwal said: If the local DN was added to deadNodes in a DFSInputStream
because it was restarted, we may be able to (asynchronously?) probe and remove
it from deadNodes.
Why not just have a time limit on the blacklist, like 15 minutes? This would
also help with the case where a DN is temporarily overloaded, and gets added to
the blacklist on a long-open file, and never removed. We should do the simple
things first, and then perhaps move on to more complex schemes.
If you really want to get fancy, you could have a separate daemon running on
the DN which would stay up during the duration of the upgrade, and tell clients
who asked what the status of the upgrade was. But that seems like a big
project, when we haven't even done the simple things, like sharing information
about deadNodes between different DFSInputStreams in the same client.
> Umbrella jira for improved HDFS rolling upgrades
>
>
> Key: HDFS-5535
> URL: https://issues.apache.org/jira/browse/HDFS-5535
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, ha, hdfs-client, namenode
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Nathan Roberts
> Attachments: HDFSRollingUpgradesHighLevelDesign.pdf,
> h5535_20140219.patch, h5535_20140220-1554.patch, h5535_20140220b.patch,
> h5535_20140221-2031.patch, h5535_20140224-1931.patch,
> h5535_20140225-1225.patch
>
>
> In order to roll a new HDFS release through a large cluster quickly and
> safely, a few enhancements are needed in HDFS. An initial High level design
> document will be attached to this jira, and sub-jiras will itemize the
> individual tasks.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13912575#comment-13912575 ] stack commented on HDFS-5535: - [~kihwal] bq. we may be able to (asynchronously?) probe and remove it from deadNodes. I believe that is what our mighty [~cmccabe] is suggesting too in his comment above (i.e. HDFS-4246 added something like this on the write path and then there is apparently an issue to do similar at read time). Chatting w/ Colin too, it sound like SSR, if it fails a local read, it will then retry the local read again after some number of minutes have elapsed. This could be enough to get reads over the DN restart blip. Thanks. > Umbrella jira for improved HDFS rolling upgrades > > > Key: HDFS-5535 > URL: https://issues.apache.org/jira/browse/HDFS-5535 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, ha, hdfs-client, namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Nathan Roberts > Attachments: HDFSRollingUpgradesHighLevelDesign.pdf, > h5535_20140219.patch, h5535_20140220-1554.patch, h5535_20140220b.patch, > h5535_20140221-2031.patch, h5535_20140224-1931.patch, > h5535_20140225-1225.patch > > > In order to roll a new HDFS release through a large cluster quickly and > safely, a few enhancements are needed in HDFS. An initial High level design > document will be attached to this jira, and sub-jiras will itemize the > individual tasks. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[
https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13912525#comment-13912525
]
Kihwal Lee commented on HDFS-5535:
--
[~stack]: If the local DN was added to {{deadNodes}} in a {{DFSInputStream}}
because it was restarted, we may be able to (asynchronously?) probe and remove
it from {{deadNodes}}. Then when a block boundary is crossed, the local node
will get used again. If explicit restart notifications are complex or
unreliable (due to timing & network, etc), we could do the recovery solely
based on the failure mode. E.g. abrupt connection breakage by the remote end,
connection refusal/reset, etc. Background ping thread can reintroduce it
independent of on-going reads. I haven't checked how easy it is to track the
cause of failures, so I can't say it is feasible just yet.
> Umbrella jira for improved HDFS rolling upgrades
>
>
> Key: HDFS-5535
> URL: https://issues.apache.org/jira/browse/HDFS-5535
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, ha, hdfs-client, namenode
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Nathan Roberts
> Attachments: HDFSRollingUpgradesHighLevelDesign.pdf,
> h5535_20140219.patch, h5535_20140220-1554.patch, h5535_20140220b.patch,
> h5535_20140221-2031.patch, h5535_20140224-1931.patch,
> h5535_20140225-1225.patch
>
>
> In order to roll a new HDFS release through a large cluster quickly and
> safely, a few enhancements are needed in HDFS. An initial High level design
> document will be attached to this jira, and sub-jiras will itemize the
> individual tasks.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13912513#comment-13912513 ] Kihwal Lee commented on HDFS-5535: -- I will take care of the new findbugs warnings in HDFS-6020. > Umbrella jira for improved HDFS rolling upgrades > > > Key: HDFS-5535 > URL: https://issues.apache.org/jira/browse/HDFS-5535 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, ha, hdfs-client, namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Nathan Roberts > Attachments: HDFSRollingUpgradesHighLevelDesign.pdf, > h5535_20140219.patch, h5535_20140220-1554.patch, h5535_20140220b.patch, > h5535_20140221-2031.patch, h5535_20140224-1931.patch, > h5535_20140225-1225.patch > > > In order to roll a new HDFS release through a large cluster quickly and > safely, a few enhancements are needed in HDFS. An initial High level design > document will be attached to this jira, and sub-jiras will itemize the > individual tasks. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[
https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13912319#comment-13912319
]
Hadoop QA commented on HDFS-5535:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12631032/h5535_20140225-1225.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 41 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 5 new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6235//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/6235//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/6235//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6235//console
This message is automatically generated.
> Umbrella jira for improved HDFS rolling upgrades
>
>
> Key: HDFS-5535
> URL: https://issues.apache.org/jira/browse/HDFS-5535
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, ha, hdfs-client, namenode
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Nathan Roberts
> Attachments: HDFSRollingUpgradesHighLevelDesign.pdf,
> h5535_20140219.patch, h5535_20140220-1554.patch, h5535_20140220b.patch,
> h5535_20140221-2031.patch, h5535_20140224-1931.patch,
> h5535_20140225-1225.patch
>
>
> In order to roll a new HDFS release through a large cluster quickly and
> safely, a few enhancements are needed in HDFS. An initial High level design
> document will be attached to this jira, and sub-jiras will itemize the
> individual tasks.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13912309#comment-13912309 ] Colin Patrick McCabe commented on HDFS-5535: [~stack]: I think we need to do HDFS-4246, but for the DFSInputStrean (HDFS-4246 was for the output stream). > Umbrella jira for improved HDFS rolling upgrades > > > Key: HDFS-5535 > URL: https://issues.apache.org/jira/browse/HDFS-5535 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, ha, hdfs-client, namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Nathan Roberts > Attachments: HDFSRollingUpgradesHighLevelDesign.pdf, > h5535_20140219.patch, h5535_20140220-1554.patch, h5535_20140220b.patch, > h5535_20140221-2031.patch, h5535_20140224-1931.patch, > h5535_20140225-1225.patch > > > In order to roll a new HDFS release through a large cluster quickly and > safely, a few enhancements are needed in HDFS. An initial High level design > document will be attached to this jira, and sub-jiras will itemize the > individual tasks. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13912245#comment-13912245 ] stack commented on HDFS-5535: - [~kihwal] bq. I will be more than happy to further improve the DN restart latency, if it becomes major hurdle for HBase. Is there anything that can be done about the following? DN is rolling restarted under a RegionServer. It takes > timeout for the DN to come back. DN gets added to deadnode list in DFSInputStream (right?) because we failed the read and had to go to a remote replica instead. Hereafter all reads take longer because they are no longer local (unless we restart regionserver). Thanks. > Umbrella jira for improved HDFS rolling upgrades > > > Key: HDFS-5535 > URL: https://issues.apache.org/jira/browse/HDFS-5535 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, ha, hdfs-client, namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Nathan Roberts > Attachments: HDFSRollingUpgradesHighLevelDesign.pdf, > h5535_20140219.patch, h5535_20140220-1554.patch, h5535_20140220b.patch, > h5535_20140221-2031.patch, h5535_20140224-1931.patch, > h5535_20140225-1225.patch > > > In order to roll a new HDFS release through a large cluster quickly and > safely, a few enhancements are needed in HDFS. An initial High level design > document will be attached to this jira, and sub-jiras will itemize the > individual tasks. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13912212#comment-13912212 ] Colin Patrick McCabe commented on HDFS-5535: I took a look at the design doc and some of the subtasks. It's clever to use NameNode HA to provide a zero-downtime upgrade. Splitting the NN and DN layout versions is sensible and will avoid headaches in the future. I always thought the DN registration version check was a hack. It's good to see it go away, replaced by a simple check of the LayoutVersion and protocol version. HDFS-5496 is also a good idea. It is too bad that the newly introduced "downgrade" functionality is only available between dot releases, but I think we can live with that (just like we're living with it now). It's not a regression. I agree that downgrade between dot releases should be fairly straightforward. HDFS-5498, caching the DU result, seems reasonable. Similarly, parallelizing the block scanner seems like an obvious improvement. One area where I see some complications is in the out-of-band notification sent to DFSClient instances when a Datanode is about to go down. This is certainly something that HBase (among others) could use, but it seems like a big new change to a rarely used codepath. In the interest of getting this into 2.4, might there be some benefit to splitting out this part? We still have a lot of unaddressed issues on the DN write pipeline like HDFS-4504, and I'm nervous about adding too many new features until those bugs are addressed. I think some small changes like HDFS-6016 would be sufficient to dramatically improve DN rolling upgrade. > Umbrella jira for improved HDFS rolling upgrades > > > Key: HDFS-5535 > URL: https://issues.apache.org/jira/browse/HDFS-5535 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, ha, hdfs-client, namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Nathan Roberts > Attachments: HDFSRollingUpgradesHighLevelDesign.pdf, > h5535_20140219.patch, h5535_20140220-1554.patch, h5535_20140220b.patch, > h5535_20140221-2031.patch, h5535_20140224-1931.patch, > h5535_20140225-1225.patch > > > In order to roll a new HDFS release through a large cluster quickly and > safely, a few enhancements are needed in HDFS. An initial High level design > document will be attached to this jira, and sub-jiras will itemize the > individual tasks. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13912160#comment-13912160 ] Andrew Wang commented on HDFS-5535: --- Hi Suresh, thanks for commenting back: bq. This is something Ming Ma asked about above regarding TOR switch upgrades, and Kihwal said it might be considered with lite-decom. Basically, what to do when the restart will take longer than the client timeout. Doing a full decom in this case sucks since we know the DN will be back soon, but just not soon enough to wait it out. I was wondering if this was going to be handled, now that you're getting close with the rest. bq. If we get feature flags finished (HDFS-5223), we can keep edit/fsimage compatibility between minor releases until the operator enables a new feature. Even after it's enabled, like you said, we'll sometimes be able to just drop the new information on downgrade, e.g. ignore ACLs on an inode. I agree that sometimes we'll have to break compat, but rolling upgrade + feature flags + PB means our upgrade/downgrade story is better for operators who are primarily concerned with bug fixes rather than new features. > Umbrella jira for improved HDFS rolling upgrades > > > Key: HDFS-5535 > URL: https://issues.apache.org/jira/browse/HDFS-5535 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, ha, hdfs-client, namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Nathan Roberts > Attachments: HDFSRollingUpgradesHighLevelDesign.pdf, > h5535_20140219.patch, h5535_20140220-1554.patch, h5535_20140220b.patch, > h5535_20140221-2031.patch, h5535_20140224-1931.patch, > h5535_20140225-1225.patch > > > In order to roll a new HDFS release through a large cluster quickly and > safely, a few enhancements are needed in HDFS. An initial High level design > document will be attached to this jira, and sub-jiras will itemize the > individual tasks. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[
https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13912153#comment-13912153
]
Kihwal Lee commented on HDFS-5535:
--
{quote}
[~andrew.wang] : Can you comment on how riding out DN restarts interacts with
the HBase MTTR work? I know they've done a lot of work to reduce timeouts
throughout the stack, and riding out restarts sounds like we need to keep the
timeouts up. It might help to specify your target restart time, for example
with a DN with 500k blocks.
{quote}
There are multiple factors that affect the DN upgrade latency. First, shutdown
can take up to about 2 seconds. The time is bounded so that slow
clients/network cannot delay the shutdown. That is, the shutdown notification
(via the OOB acks) is advisory and may not reach all clients. HDFS does not
provide a integrated tool for software/config upgrade & restart. The process
will be different depending on how hadoop and its config is deployed. Whatever
the method is, it will add a bit of delay to the whole process.
The actual DN startup time varies depending on the number of blocks and the
number of disks (typically corresponds to # of volumes). Before HDFS-5498, DN
would run "du" on each volume serially and also list and stat block and meta
files serially. Now "du" doesn't have to run if restarted within 5 minutes of
shutdown. Also block/meta scans are done concurrently if multiple volumes are
present. So, the startup time will be shorter if blocks are spread across more
disks. In a test carried out in the beginning of HDFS-5498, the restart time of
a node shrunk from over a minute to about 12 seconds. We could make restart
even shorter by saving more state on shutdown.
The final step before being able to serve client is the registration. If the
block token is used, DNs won't be able to validate them until registration with
NN is complete. This usually happens immediately after scanning and adding
blocks. We could save the shared secret before shutdown, but obviously it can
be a security risk.
I will be more than happy to further improve the DN restart latency, if it
becomes major hurdle for HBase.
{quote}
[~stack] : + "3. Clients" It says "For shutdown, clients may choose to copy
partially written replica to another node..." The DFSClient would do this
internally (or 'may' do this)?
{quote}
This is done internally. This is what happens to clients with # replicas >= 3
today when node failures leave only (#replica / 2) nodes in the pipeline. If
the restarting node is not local or more than only nodes are in the pipeline,
the client will simply exclude the restarting node and , if necessary, add more
nodes to the pipeline and copy partially written data over to the new node. If
the restarting node is the local node or is the only node in the pipeline,
client will wait for the node to be restarted. There is a configurable timeout
for the wait.
If the restarting node was the only node in the pipeline, the write will fail
after the restart timeout. This is very unlikely to blocks with 3 or more
replicas due to the DN replacement policy. For blocks being written with
replication factor of 2 can, however, suffer, if the writer has already lost
one node due to a failure and loses another because of restart failure. To
address this issue, HDFS-6016 has been filed. However, this is a non-blocking
issue for the rolling upgrade feature.
Regarding use of shortened read/write socket timeout, I can see the use cases
favoring failing over to a slower node than waiting longer for the same
fail-over or recovery, if lucky. Reads will fail over to a different source
during a DN restart. For writes, the restart timeout is independent of the
socket write timeout, so the client may block for more than what the user wants
to. If that is undesirable, the configuration should be changed so that
DFSOutputStream can timeout on restart much quicker. The default is currently
30 seconds.
> Umbrella jira for improved HDFS rolling upgrades
>
>
> Key: HDFS-5535
> URL: https://issues.apache.org/jira/browse/HDFS-5535
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, ha, hdfs-client, namenode
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Nathan Roberts
> Attachments: HDFSRollingUpgradesHighLevelDesign.pdf,
> h5535_20140219.patch, h5535_20140220-1554.patch, h5535_20140220b.patch,
> h5535_20140221-2031.patch, h5535_20140224-1931.patch,
> h5535_20140225-1225.patch
>
>
> In order to roll a new HDFS release through a large cluster quickly and
> safely, a few enhancements are needed in HDFS. An initial High level design
> document will be attached to this jira, and sub-jiras will itemize the
> individual tasks.
--
This message was sent by Atlassi
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13912136#comment-13912136 ] Tsz Wo (Nicholas), SZE commented on HDFS-5535: -- > It looks like this feature is getting close, nice work! Can we get a rev of > the design doc at a less-high level as we approach merge? It seems like > details surrounding e.g. user API and implementation have been ironed out, so > should be included. ... Thanks. We are going to revise the design doc. However, there is not much user API to add since this feature, Rolling Upgrade, does not really have meaningful user API. There are new CLI and startup options. The usage of them and other user guide can be found in the user documentation committed to the branch. > ..., sorry in advance if these were already answered in the comments: No problem. Perhaps, you should take your time and read the comments first. No hurry. > Can you expand on NN/DN consistency with the rollback marker and heartbeat > notifications? I'm not familiar with append or lease recovery, so it'd be > nice to get more explanation on those in particular. This part is similar to current upgrade/rollback mechanism. Do you familiar with it? For append and lease recovery, you may want to read the append design doc in HDFS-265. > Umbrella jira for improved HDFS rolling upgrades > > > Key: HDFS-5535 > URL: https://issues.apache.org/jira/browse/HDFS-5535 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, ha, hdfs-client, namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Nathan Roberts > Attachments: HDFSRollingUpgradesHighLevelDesign.pdf, > h5535_20140219.patch, h5535_20140220-1554.patch, h5535_20140220b.patch, > h5535_20140221-2031.patch, h5535_20140224-1931.patch, > h5535_20140225-1225.patch > > > In order to roll a new HDFS release through a large cluster quickly and > safely, a few enhancements are needed in HDFS. An initial High level design > document will be attached to this jira, and sub-jiras will itemize the > individual tasks. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13912079#comment-13912079 ] stack commented on HDFS-5535: - bq. Downgrade sounds like it will be a load of work I wrote the above and then on review didn't intend to include it. Agree it looks straight-forward in this case. > Umbrella jira for improved HDFS rolling upgrades > > > Key: HDFS-5535 > URL: https://issues.apache.org/jira/browse/HDFS-5535 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, ha, hdfs-client, namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Nathan Roberts > Attachments: HDFSRollingUpgradesHighLevelDesign.pdf, > h5535_20140219.patch, h5535_20140220-1554.patch, h5535_20140220b.patch, > h5535_20140221-2031.patch, h5535_20140224-1931.patch, > h5535_20140225-1225.patch > > > In order to roll a new HDFS release through a large cluster quickly and > safely, a few enhancements are needed in HDFS. An initial High level design > document will be attached to this jira, and sub-jiras will itemize the > individual tasks. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[
https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13911339#comment-13911339
]
Suresh Srinivas commented on HDFS-5535:
---
Thanks for the comments [~andrew.wang] and [~stack]. The design needs to be
updated. We will do that in a day or two. The comments related to unifying the
sections will be taken care of then.
Responses:
{quote}
Could you comment on your experiences regarding the interval between an upgrade
and finalize? My impression was that right now, cluster operators might wait a
long time before finalizing to be safe (e.g. a week or two). Since
checkpointing would be paused with the rollback marker, a lot of edits would
accumulate, and NN startup time would suffer.
{quote}
With the latest changes from HDFS-6000, during rolling upgrades, checkpointing
will continue to happen. We only retain a special fsimage for rolling back. BTW
not finalizing for a week or two comes at significant storage cost, since no
blocks are deleted. I generally recommend ~ 3days to finalize, depending on the
storage pressure in the cluster.
bq. We should also call this out on the Hadoop compatibility wiki page when
this JIRA is merged goes in.
I will add some information about this. I do not think we can leave layout
version as is, as new features are added. While the old version of namenode can
perhaps handle new editlog, getting rid of data saved by newly added feature
may not be straightforward in all the cases.
bq. It might help to specify your target restart time, for example with a DN
with 500k blocks.
bq. What you folks thinking here? I saw 60 seconds earlier up in the doc. Some
HBase deploys have this ratcheted down to a few seconds or so...
[~kihwal] or [~brandonli] can comment on the final timeout chosen here.
bq. Are longer restarts (e.g. OS or hardware upgrade) part of the scope?
What specifically are you referring to? Can you add more details?
bq. Downgrade sounds like it will be a load of work
[~stack], this is fairly straight forward. When namenode layout has not
changed, the older release of namenode can handle both fsimage and edits
written by the new namenode. In the current upgrade mechanism, we lose lots of
newly written data on rollback. With downgrade, the newly created data can be
retained. This mainly would work in dot releases and most likely not in minor
release upgrade, where rollback may have to be used.
> Umbrella jira for improved HDFS rolling upgrades
>
>
> Key: HDFS-5535
> URL: https://issues.apache.org/jira/browse/HDFS-5535
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, ha, hdfs-client, namenode
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Nathan Roberts
> Attachments: HDFSRollingUpgradesHighLevelDesign.pdf,
> h5535_20140219.patch, h5535_20140220-1554.patch, h5535_20140220b.patch,
> h5535_20140221-2031.patch, h5535_20140224-1931.patch
>
>
> In order to roll a new HDFS release through a large cluster quickly and
> safely, a few enhancements are needed in HDFS. An initial High level design
> document will be attached to this jira, and sub-jiras will itemize the
> individual tasks.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[
https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13911300#comment-13911300
]
Hadoop QA commented on HDFS-5535:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12630877/h5535_20140224-1931.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 41 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:
org.apache.hadoop.hdfs.server.datanode.TestBlockRecovery
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6225//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/6225//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/6225//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6225//console
This message is automatically generated.
> Umbrella jira for improved HDFS rolling upgrades
>
>
> Key: HDFS-5535
> URL: https://issues.apache.org/jira/browse/HDFS-5535
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, ha, hdfs-client, namenode
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Nathan Roberts
> Attachments: HDFSRollingUpgradesHighLevelDesign.pdf,
> h5535_20140219.patch, h5535_20140220-1554.patch, h5535_20140220b.patch,
> h5535_20140221-2031.patch, h5535_20140224-1931.patch
>
>
> In order to roll a new HDFS release through a large cluster quickly and
> safely, a few enhancements are needed in HDFS. An initial High level design
> document will be attached to this jira, and sub-jiras will itemize the
> individual tasks.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13911274#comment-13911274 ] stack commented on HDFS-5535: - Few late comments on the design doc: + "It has an added benefit of not losing data locality, which is critical for applications such as HBase." Thanks for the consideration of clients that keep files open while they are up. +1 on suggestion that we stall the pipeline rather than have the restarted replica cause a break in existing pipelines. + "For example, MTTR should be less than the default client socket timeout for successful restarts." What you folks thinking here? I saw 60 seconds earlier up in the doc. Some HBase deploys have this ratcheted down to a few seconds or so (For example, see http://goo.gl/Ue3FPl for where pinterest talk about 3 seconds socket timeout on read, 5 seconds for write socket timeout and 1 second on ipc w/ retries set to two). It'd be coolio if we didn't have to rolling restart hbase on top of an hdfs rolling restart, if they could be done independent of each other without incurring loss of locality. + "3. Clients" It says "For shutdown, clients may choose to copy partially written replica to another node..." The DFSClient would do this internally (or 'may' do this)? Good stuff. + "Rollback and downgrade requires cluster downtime and is not done in a rolling fashion." + Downgrade sounds like it will be a load of work (I used to work at a place where eng. spent 30-40% of its time making sure migrations from one version to another worked going forward and backwards because someone thought it was a good idea not realizing I"m sure the cost involved). You for sure want to support that? > Umbrella jira for improved HDFS rolling upgrades > > > Key: HDFS-5535 > URL: https://issues.apache.org/jira/browse/HDFS-5535 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, ha, hdfs-client, namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Nathan Roberts > Attachments: HDFSRollingUpgradesHighLevelDesign.pdf, > h5535_20140219.patch, h5535_20140220-1554.patch, h5535_20140220b.patch, > h5535_20140221-2031.patch, h5535_20140224-1931.patch > > > In order to roll a new HDFS release through a large cluster quickly and > safely, a few enhancements are needed in HDFS. An initial High level design > document will be attached to this jira, and sub-jiras will itemize the > individual tasks. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13910982#comment-13910982 ] Andrew Wang commented on HDFS-5535: --- Hi all, It looks like this feature is getting close, nice work! Can we get a rev of the design doc at a less-high level as we approach merge? It seems like details surrounding e.g. user API and implementation have been ironed out, so should be included. There are also (I believe deprecated) mentions of lite-decom. It'd also be nice if someone could unify the section title formatting, since there are a number of different parts (checkpoint/rollback, NN failover, DN restart), and they each use their own formatting schemes. Namely, it'd be very helpful to consistently number the section titles (most word processing apps can do this for you). I also had a few questions after reading the doc, sorry in advance if these were already answered in the comments: * Can you expand on NN/DN consistency with the rollback marker and heartbeat notifications? I'm not familiar with append or lease recovery, so it'd be nice to get more explanation on those in particular. * Could you comment on your experiences regarding the interval between an upgrade and finalize? My impression was that right now, cluster operators might wait a long time before finalizing to be safe (e.g. a week or two). Since checkpointing would be paused with the rollback marker, a lot of edits would accumulate, and NN startup time would suffer. * Big +1 to not changing the layout version any further in the 2.x line after this. With PB'd metadata and feature flags (whenever they arrive), this makes NN upgrade a lot more pleasant. We should also call this out on the Hadoop compatibility wiki page when this JIRA is merged goes in. * Can you comment on how riding out DN restarts interacts with the HBase MTTR work? I know they've done a lot of work to reduce timeouts throughout the stack, and riding out restarts sounds like we need to keep the timeouts up. It might help to specify your target restart time, for example with a DN with 500k blocks. * Are longer restarts (e.g. OS or hardware upgrade) part of the scope? Obviously, 1-repl blocks would become an issue, and a super long timeout is not a good solution. Maybe this is just the normal decom process needing love, but it'd be nice to address these longer maintenance restarts too. > Umbrella jira for improved HDFS rolling upgrades > > > Key: HDFS-5535 > URL: https://issues.apache.org/jira/browse/HDFS-5535 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, ha, hdfs-client, namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Nathan Roberts > Attachments: HDFSRollingUpgradesHighLevelDesign.pdf, > h5535_20140219.patch, h5535_20140220-1554.patch, h5535_20140220b.patch, > h5535_20140221-2031.patch > > > In order to roll a new HDFS release through a large cluster quickly and > safely, a few enhancements are needed in HDFS. An initial High level design > document will be attached to this jira, and sub-jiras will itemize the > individual tasks. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13909668#comment-13909668 ] Fengdong Yu commented on HDFS-5535: --- Thanks [~szetszwo], I'll do testing next week. and post here. > Umbrella jira for improved HDFS rolling upgrades > > > Key: HDFS-5535 > URL: https://issues.apache.org/jira/browse/HDFS-5535 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, ha, hdfs-client, namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Nathan Roberts > Attachments: HDFSRollingUpgradesHighLevelDesign.pdf, > h5535_20140219.patch, h5535_20140220-1554.patch, h5535_20140220b.patch, > h5535_20140221-2031.patch > > > In order to roll a new HDFS release through a large cluster quickly and > safely, a few enhancements are needed in HDFS. An initial High level design > document will be attached to this jira, and sub-jiras will itemize the > individual tasks. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13909564#comment-13909564 ] Tsz Wo (Nicholas), SZE commented on HDFS-5535: -- - The release audit is about CHANGES_HDFS-5535.txt which will be removed after merge. - TestOfflineEditsViewer is already fixed in the branch. It needs the new [editsStored|https://issues.apache.org/jira/secure/attachment/12630243/editsStored] file posted in HDFS-5992. > Umbrella jira for improved HDFS rolling upgrades > > > Key: HDFS-5535 > URL: https://issues.apache.org/jira/browse/HDFS-5535 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, ha, hdfs-client, namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Nathan Roberts > Attachments: HDFSRollingUpgradesHighLevelDesign.pdf, > h5535_20140219.patch, h5535_20140220-1554.patch, h5535_20140220b.patch, > h5535_20140221-2031.patch > > > In order to roll a new HDFS release through a large cluster quickly and > safely, a few enhancements are needed in HDFS. An initial High level design > document will be attached to this jira, and sub-jiras will itemize the > individual tasks. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13909553#comment-13909553 ] Tsz Wo (Nicholas), SZE commented on HDFS-5535: -- > ... RollingUpgrade is a little big change, so does that enough only run unit > tests to test it? or you'd already did some manual testing on the dev cluster? Yes, we will post a test plan which includes manual tests. > Does this patch based on trunk? I can do some verification on my testing > cluster. The patch does apply to trunk. It would be great if you could help testing it. Thank you in advance! > Umbrella jira for improved HDFS rolling upgrades > > > Key: HDFS-5535 > URL: https://issues.apache.org/jira/browse/HDFS-5535 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, ha, hdfs-client, namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Nathan Roberts > Attachments: HDFSRollingUpgradesHighLevelDesign.pdf, > h5535_20140219.patch, h5535_20140220-1554.patch, h5535_20140220b.patch, > h5535_20140221-2031.patch > > > In order to roll a new HDFS release through a large cluster quickly and > safely, a few enhancements are needed in HDFS. An initial High level design > document will be attached to this jira, and sub-jiras will itemize the > individual tasks. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[
https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13909290#comment-13909290
]
Hadoop QA commented on HDFS-5535:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12630456/h5535_20140221-2031.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 40 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6214//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/6214//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6214//console
This message is automatically generated.
> Umbrella jira for improved HDFS rolling upgrades
>
>
> Key: HDFS-5535
> URL: https://issues.apache.org/jira/browse/HDFS-5535
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, ha, hdfs-client, namenode
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Nathan Roberts
> Attachments: HDFSRollingUpgradesHighLevelDesign.pdf,
> h5535_20140219.patch, h5535_20140220-1554.patch, h5535_20140220b.patch,
> h5535_20140221-2031.patch
>
>
> In order to roll a new HDFS release through a large cluster quickly and
> safely, a few enhancements are needed in HDFS. An initial High level design
> document will be attached to this jira, and sub-jiras will itemize the
> individual tasks.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13909255#comment-13909255 ] Fengdong Yu commented on HDFS-5535: --- [~szetszwo] , RollingUpgrade is a little big change, so does that enough only run unit tests to test it? or you'd already did some manual testing on the dev cluster? Does this patch based on trunk? I can do some verification on my testing cluster. > Umbrella jira for improved HDFS rolling upgrades > > > Key: HDFS-5535 > URL: https://issues.apache.org/jira/browse/HDFS-5535 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, ha, hdfs-client, namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Nathan Roberts > Attachments: HDFSRollingUpgradesHighLevelDesign.pdf, > h5535_20140219.patch, h5535_20140220-1554.patch, h5535_20140220b.patch, > h5535_20140221-2031.patch > > > In order to roll a new HDFS release through a large cluster quickly and > safely, a few enhancements are needed in HDFS. An initial High level design > document will be attached to this jira, and sub-jiras will itemize the > individual tasks. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[
https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13908082#comment-13908082
]
Hadoop QA commented on HDFS-5535:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12630246/h5535_20140220b.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 42 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade
org.apache.hadoop.fs.loadGenerator.TestLoadGenerator
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6203//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/6203//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6203//console
This message is automatically generated.
> Umbrella jira for improved HDFS rolling upgrades
>
>
> Key: HDFS-5535
> URL: https://issues.apache.org/jira/browse/HDFS-5535
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, ha, hdfs-client, namenode
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Nathan Roberts
> Attachments: HDFSRollingUpgradesHighLevelDesign.pdf,
> h5535_20140219.patch, h5535_20140220-1554.patch, h5535_20140220b.patch
>
>
> In order to roll a new HDFS release through a large cluster quickly and
> safely, a few enhancements are needed in HDFS. An initial High level design
> document will be attached to this jira, and sub-jiras will itemize the
> individual tasks.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[
https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13907921#comment-13907921
]
Hadoop QA commented on HDFS-5535:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12630196/h5535_20140220-1554.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 39 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:
org.apache.hadoop.hdfs.server.namenode.TestStorageRestore
org.apache.hadoop.hdfs.server.namenode.TestSecondaryNameNodeUpgrade
org.apache.hadoop.hdfs.server.namenode.snapshot.TestCheckpointsWithSnapshots
org.apache.hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits
org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives
org.apache.hadoop.hdfs.qjournal.client.TestQJMWithFaults
org.apache.hadoop.fs.loadGenerator.TestLoadGenerator
org.apache.hadoop.hdfs.TestRollingUpgrade
org.apache.hadoop.hdfs.TestRollingUpgradeRollback
org.apache.hadoop.hdfs.qjournal.server.TestJournalNode
org.apache.hadoop.hdfs.util.TestMD5FileUtils
org.apache.hadoop.hdfs.qjournal.TestNNWithQJM
org.apache.hadoop.hdfs.server.namenode.TestNameEditsConfigs
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
org.apache.hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade
org.apache.hadoop.hdfs.qjournal.client.TestQuorumJournalManager
org.apache.hadoop.hdfs.server.namenode.TestStartup
org.apache.hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA
The following test timeouts occurred in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:
org.apache.hadoop.hdfs.server.namenode.TestBackupNode
org.apache.hadoop.hdfs.server.namenode.TestCheckpoint
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6199//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/6199//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6199//console
This message is automatically generated.
> Umbrella jira for improved HDFS rolling upgrades
>
>
> Key: HDFS-5535
> URL: https://issues.apache.org/jira/browse/HDFS-5535
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, ha, hdfs-client, namenode
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Nathan Roberts
> Attachments: HDFSRollingUpgradesHighLevelDesign.pdf,
> h5535_20140219.patch, h5535_20140220-1554.patch
>
>
> In order to roll a new HDFS release through a large cluster quickly and
> safely, a few enhancements are needed in HDFS. An initial High level design
> document will be attached to this jira, and sub-jiras will itemize the
> individual tasks.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13906995#comment-13906995 ] Kihwal Lee commented on HDFS-5535: -- [~azuryy] I believe remaining parts are mostly independent, so we can start testing and fixing problems now. > Umbrella jira for improved HDFS rolling upgrades > > > Key: HDFS-5535 > URL: https://issues.apache.org/jira/browse/HDFS-5535 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, ha, hdfs-client, namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Nathan Roberts > Attachments: HDFSRollingUpgradesHighLevelDesign.pdf, > h5535_20140219.patch > > > In order to roll a new HDFS release through a large cluster quickly and > safely, a few enhancements are needed in HDFS. An initial High level design > document will be attached to this jira, and sub-jiras will itemize the > individual tasks. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[
https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13906717#comment-13906717
]
Hadoop QA commented on HDFS-5535:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12629993/h5535_20140219.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 39 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 5 new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:
org.apache.hadoop.hdfs.TestInjectionForSimulatedStorage
org.apache.hadoop.hdfs.TestPread
org.apache.hadoop.hdfs.TestReplication
org.apache.hadoop.hdfs.TestSmallBlock
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
org.apache.hadoop.hdfs.server.datanode.TestDataNodeMetrics
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer
org.apache.hadoop.hdfs.TestFileCreation
org.apache.hadoop.hdfs.server.datanode.TestReadOnlySharedStorage
org.apache.hadoop.hdfs.server.namenode.TestEditLog
org.apache.hadoop.hdfs.TestSetrepIncreasing
org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
org.apache.hadoop.hdfs.server.namenode.TestFileLimit
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
org.apache.hadoop.hdfs.server.balancer.TestBalancer
org.apache.hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/6189//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/6189//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/6189//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6189//console
This message is automatically generated.
> Umbrella jira for improved HDFS rolling upgrades
>
>
> Key: HDFS-5535
> URL: https://issues.apache.org/jira/browse/HDFS-5535
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, ha, hdfs-client, namenode
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Nathan Roberts
> Attachments: HDFSRollingUpgradesHighLevelDesign.pdf,
> h5535_20140219.patch
>
>
> In order to roll a new HDFS release through a large cluster quickly and
> safely, a few enhancements are needed in HDFS. An initial High level design
> document will be attached to this jira, and sub-jiras will itemize the
> individual tasks.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13906661#comment-13906661 ] Fengdong Yu commented on HDFS-5535: --- There are still some sub tasks is not closed, so does this patch is ready for unit tests? > Umbrella jira for improved HDFS rolling upgrades > > > Key: HDFS-5535 > URL: https://issues.apache.org/jira/browse/HDFS-5535 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, ha, hdfs-client, namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Nathan Roberts > Attachments: HDFSRollingUpgradesHighLevelDesign.pdf, > h5535_20140219.patch > > > In order to roll a new HDFS release through a large cluster quickly and > safely, a few enhancements are needed in HDFS. An initial High level design > document will be attached to this jira, and sub-jiras will itemize the > individual tasks. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[
https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872303#comment-13872303
]
Kihwal Lee commented on HDFS-5535:
--
{quote}
bq. This will allow an 8600 node cluster to complete in 24 hours.
I dint understand this calculation.
{quote}
I believe this number came out from #nodes * 10 seconds. Currently a big data
node can take minutes to be fully up after restart and it gets worse with
-upgrade. Perhaps we should refine this requirement since the timing is
affected by many factors in both software and hardware.
- # of blocks
- # of volumes (disks)
- performance of disks and controller
- amount of memory available for caching file system objects (mainly directory
entries and inodes).
- type of underlying file system and operating system
> Umbrella jira for improved HDFS rolling upgrades
>
>
> Key: HDFS-5535
> URL: https://issues.apache.org/jira/browse/HDFS-5535
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, ha, hdfs-client, namenode
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Nathan Roberts
> Attachments: HDFSRollingUpgradesHighLevelDesign.pdf
>
>
> In order to roll a new HDFS release through a large cluster quickly and
> safely, a few enhancements are needed in HDFS. An initial High level design
> document will be attached to this jira, and sub-jiras will itemize the
> individual tasks.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872269#comment-13872269 ] Kihwal Lee commented on HDFS-5535: -- bq. The total time required to upgrade a cluster MUST not exceed #Nodes_in_cluster * 10 seconds. This is about how fast the upgrade process can go while minimally impacting service and data availability. Please note that this is a requirement for the upgrade feature. It does not dictate what users should do. This requirement exists mainly to help users estimate how soon a cluster can be upgraded and also force us to guarantee estimates stay valid in the future. bq. Probably meant to say that old software should be able to support whatever state of the file system left after the upgrade experiment was terminated? I know you didn't intended it to be, but this sounds like the requirement is reduced to maintaining file system integrity. It could simply be "Data durability must not be compromised by upgrades or downgrades". bq. May be it needs to roll edits in some special way to indicate the start of the rolling upgrade? I believe this came up during discussions, but do not remember the conclusion. We will clarify this. bq. What is MTTR? Mean time to recovery. bq. Looks like Lite-Decom and “Optimizing DN Restart time” are competing proposals Yes, indeed. We will do the latter, which will be more in-line with existing tool-driven approaches. Lite-Decom may be considered in later development phases for other use cases(e.g. the case Ming Ma mentioned above), but regular DN rolling upgrade won't depend on it. > Umbrella jira for improved HDFS rolling upgrades > > > Key: HDFS-5535 > URL: https://issues.apache.org/jira/browse/HDFS-5535 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, ha, hdfs-client, namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Nathan Roberts > Attachments: HDFSRollingUpgradesHighLevelDesign.pdf > > > In order to roll a new HDFS release through a large cluster quickly and > safely, a few enhancements are needed in HDFS. An initial High level design > document will be attached to this jira, and sub-jiras will itemize the > individual tasks. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13870341#comment-13870341 ] Konstantin Shvachko commented on HDFS-5535: --- Thanks for the design doc, guys. My few questions. (Quotations from the document are in italic) # ??The total time required to upgrade a cluster MUST not exceed #Nodes_in_cluster * 10 seconds.?? Not sure I understood how you measure the time to upgrade. Administrators should be able to spend as much time as they need. On the other hand I can write a script that calls upgrade commands in sequence, then push a button and the upgrade is done for me. Just trying to understand the meaning of the requirement. # ??During upgrade or downgrade, no data loss MUST occur.?? Not clear what this means in case a bug in new software led to a loss of data. Probably meant to say that old software should be able to support whatever state of the file system left after the upgrade experiment was terminated? # Does finalize require a checkpoint in the design? # ??For rollback, NN read editlog in startup as usual. It stops at the marker position, writes the fsimage back to disk and then discards the editlog.?? What happens if the edits is corrupted by the new software and the marker is not recognizable? May be it needs to roll edits in some special way to indicate the start of the rolling upgrade? # ??Software version is the version of the running software. In the current rolling upgrade mechanism?? What is the current rolling upgrade mechanism? It would make more sense to me if word "current" is removed from the above phrase. # What is MTTR? # Looks like Lite-Decom and “Optimizing DN Restart time” are competing proposals. Which one do you actually propose? Sounds like both are still being designed? The last question is because this seems to be the most intricate part of the issue. Conceptually rolling upgrades are possible with a simple patch, which eliminates the Software Version verification, plus very careful cluster administration, of course. And the trick indeed is to avoid client failures so that HBase and other apps could run during the upgrade. > Umbrella jira for improved HDFS rolling upgrades > > > Key: HDFS-5535 > URL: https://issues.apache.org/jira/browse/HDFS-5535 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, ha, hdfs-client, namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Nathan Roberts > Attachments: HDFSRollingUpgradesHighLevelDesign.pdf > > > In order to roll a new HDFS release through a large cluster quickly and > safely, a few enhancements are needed in HDFS. An initial High level design > document will be attached to this jira, and sub-jiras will itemize the > individual tasks. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[
https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13866405#comment-13866405
]
Arpit Agarwal commented on HDFS-5535:
-
{quote}
This means even changing the internal protocols also breaks the compatibility.
In that case have to wait for major releases..? or add overloaded apis..?
{quote}
It is often possible to change protocols incrementally without breaking
compatibility. Usually this means *a.* not adding or removing any required
fields (required fields are considered bad for this reason); *b.* not adding or
changing the meaning of any optional fields in such a way as to break a peer
running an older version and *c.* some protocol changes may have to be
introduced in an incremental manner across two minor releases, i.e. release n+1
adds support for handling protocol changes to be introduced in release n+2.
Note that the doc mentions rolling upgrades must be possible between
consecutive minor releases only.
The last is per my understanding, I don't want to speak for the design authors.
> Umbrella jira for improved HDFS rolling upgrades
>
>
> Key: HDFS-5535
> URL: https://issues.apache.org/jira/browse/HDFS-5535
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode, ha, hdfs-client, namenode
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Nathan Roberts
> Attachments: HDFSRollingUpgradesHighLevelDesign.pdf
>
>
> In order to roll a new HDFS release through a large cluster quickly and
> safely, a few enhancements are needed in HDFS. An initial High level design
> document will be attached to this jira, and sub-jiras will itemize the
> individual tasks.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13866350#comment-13866350 ] Vinay commented on HDFS-5535: - Very good work. >From the doc what I can understand is, Rolling upgrades are only possible from the versions which contains this feature, to future versions, *but not from the already released versions.* Some doubts: bq. The minor releases are for introducing features. Both these MUST NOT introduce incompatible changes. This means even changing the internal protocols also breaks the compatibility. In that case have to wait for major releases..? or add overloaded apis..? bq. This will allow an 8600 node cluster to complete in 24 hours. I dint understand this calculation. :-( > Umbrella jira for improved HDFS rolling upgrades > > > Key: HDFS-5535 > URL: https://issues.apache.org/jira/browse/HDFS-5535 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, ha, hdfs-client, namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Nathan Roberts > Attachments: HDFSRollingUpgradesHighLevelDesign.pdf > > > In order to roll a new HDFS release through a large cluster quickly and > safely, a few enhancements are needed in HDFS. An initial High level design > document will be attached to this jira, and sub-jiras will itemize the > individual tasks. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13865141#comment-13865141 ] Ming Ma commented on HDFS-5535: --- Nice work. Some comments: 1. HDFS Configuration update is another scenario; it could be different from code upgrade in terms of the design. For example, this requirement could mean if we can support DN dynamic config reload to handle certain config changes, no DN restart is required. 2. The write pipeline pause and resume approach is interesting as NN isn’t involved. One scenario similar to DN rolling upgrade is top-of-rack switch upgrade for 30 minutes. During the 30 minutes, we don’t want NN to consider DNs dead and trigger replication. For this specific scenario, write pipeline pause and resume approach might not be enough. > Umbrella jira for improved HDFS rolling upgrades > > > Key: HDFS-5535 > URL: https://issues.apache.org/jira/browse/HDFS-5535 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, ha, hdfs-client, namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Nathan Roberts > Attachments: HDFSRollingUpgradesHighLevelDesign.pdf > > > In order to roll a new HDFS release through a large cluster quickly and > safely, a few enhancements are needed in HDFS. An initial High level design > document will be attached to this jira, and sub-jiras will itemize the > individual tasks. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13861252#comment-13861252 ] Fengdong Yu commented on HDFS-5535: --- This feature targeting 2.4, That's great. Thanks. > Umbrella jira for improved HDFS rolling upgrades > > > Key: HDFS-5535 > URL: https://issues.apache.org/jira/browse/HDFS-5535 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, ha, hdfs-client, namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Nathan Roberts > Attachments: HDFSRollingUpgradesHighLevelDesign.pdf > > > In order to roll a new HDFS release through a large cluster quickly and > safely, a few enhancements are needed in HDFS. An initial High level design > document will be attached to this jira, and sub-jiras will itemize the > individual tasks. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13836540#comment-13836540 ] Nathan Roberts commented on HDFS-5535: -- Hi. Initial draft is still a little rough. Will try to get it up Tuesday for initial comments and suggestions. > Umbrella jira for improved HDFS rolling upgrades > > > Key: HDFS-5535 > URL: https://issues.apache.org/jira/browse/HDFS-5535 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, ha, hdfs-client, namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Nathan Roberts > > In order to roll a new HDFS release through a large cluster quickly and > safely, a few enhancements are needed in HDFS. An initial High level design > document will be attached to this jira, and sub-jiras will itemize the > individual tasks. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13836389#comment-13836389 ] Junping Du commented on HDFS-5535: -- Hi [~azuryy], Definitely. Sharing ideas, design and implementations are always welcomed! > Umbrella jira for improved HDFS rolling upgrades > > > Key: HDFS-5535 > URL: https://issues.apache.org/jira/browse/HDFS-5535 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, ha, hdfs-client, namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Nathan Roberts > > In order to roll a new HDFS release through a large cluster quickly and > safely, a few enhancements are needed in HDFS. An initial High level design > document will be attached to this jira, and sub-jiras will itemize the > individual tasks. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HDFS-5535) Umbrella jira for improved HDFS rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13836334#comment-13836334 ] Fengdong Yu commented on HDFS-5535: --- Hi folks, can we put a high level design doc or draft here, then we can push this important feature ahead. > Umbrella jira for improved HDFS rolling upgrades > > > Key: HDFS-5535 > URL: https://issues.apache.org/jira/browse/HDFS-5535 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, ha, hdfs-client, namenode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Nathan Roberts > > In order to roll a new HDFS release through a large cluster quickly and > safely, a few enhancements are needed in HDFS. An initial High level design > document will be attached to this jira, and sub-jiras will itemize the > individual tasks. -- This message was sent by Atlassian JIRA (v6.1#6144)
