[jira] [Commented] (HDFS-1335) HDFS side of HADOOP-6904: first step towards inter-version communications between dfs client and NameNode
[ https://issues.apache.org/jira/browse/HDFS-1335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155761#comment-13155761 ] Denny Ye commented on HDFS-1335: Does Hadoop-0.23 contains this patch? HDFS side of HADOOP-6904: first step towards inter-version communications between dfs client and NameNode - Key: HDFS-1335 URL: https://issues.apache.org/jira/browse/HDFS-1335 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs client, name-node Affects Versions: 0.22.0 Reporter: Hairong Kuang Assignee: Hairong Kuang Fix For: 0.23.0 Attachments: hdfsRPC.patch, hdfsRpcVersion.patch The idea is that for getProtocolVersion, NameNode checks if the client and server versions are compatible if the server version is greater than the client version. If no, throws a VersionIncompatible exception; otherwise, returns the server version. On the dfs client side, when creating a NameNode proxy, catches the VersionMismatch exception and then checks if the client version and the server version are compatible if the client version is greater than the server version. If not compatible, throws exception VersionIncomptible; otherwise, records the server version and continues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2552) Add WebHdfs Forrest doc
[ https://issues.apache.org/jira/browse/HDFS-2552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155848#comment-13155848 ] Hudson commented on HDFS-2552: -- Integrated in Hadoop-Hdfs-trunk #872 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/872/]) HDFS-2552. Add Forrest doc for WebHDFS REST API. szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1205146 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/docs/src/documentation/content/xdocs/site.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/docs/src/documentation/content/xdocs/webhdfs.xml Add WebHdfs Forrest doc --- Key: HDFS-2552 URL: https://issues.apache.org/jira/browse/HDFS-2552 Project: Hadoop HDFS Issue Type: Task Components: documentation Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Fix For: 0.20.205.1, 0.20.206.0, 0.24.0, 0.23.1 Attachments: h2552_2015.patch, h2552_2015b.patch, h2552_2015b.pdf, h2552_2017.patch, h2552_2017.pdf, h2552_2021.patch, h2552_2021.pdf, h2552_2021_0.20s.patch, h2552_2021_0.20s.pdf -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2552) Add WebHdfs Forrest doc
[ https://issues.apache.org/jira/browse/HDFS-2552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155855#comment-13155855 ] Hudson commented on HDFS-2552: -- Integrated in Hadoop-Mapreduce-0.23-Build #102 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/102/]) svn merge -c 1205146 from trunk for HDFS-2552. szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1205147 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/docs/src/documentation/content/xdocs/site.xml * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/docs/src/documentation/content/xdocs/webhdfs.xml * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java Add WebHdfs Forrest doc --- Key: HDFS-2552 URL: https://issues.apache.org/jira/browse/HDFS-2552 Project: Hadoop HDFS Issue Type: Task Components: documentation Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Fix For: 0.20.205.1, 0.20.206.0, 0.24.0, 0.23.1 Attachments: h2552_2015.patch, h2552_2015b.patch, h2552_2015b.pdf, h2552_2017.patch, h2552_2017.pdf, h2552_2021.patch, h2552_2021.pdf, h2552_2021_0.20s.patch, h2552_2021_0.20s.pdf -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2552) Add WebHdfs Forrest doc
[ https://issues.apache.org/jira/browse/HDFS-2552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13155866#comment-13155866 ] Hudson commented on HDFS-2552: -- Integrated in Hadoop-Mapreduce-trunk #906 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/906/]) HDFS-2552. Add Forrest doc for WebHDFS REST API. szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1205146 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/docs/src/documentation/content/xdocs/site.xml * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/docs/src/documentation/content/xdocs/webhdfs.xml Add WebHdfs Forrest doc --- Key: HDFS-2552 URL: https://issues.apache.org/jira/browse/HDFS-2552 Project: Hadoop HDFS Issue Type: Task Components: documentation Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Fix For: 0.20.205.1, 0.20.206.0, 0.24.0, 0.23.1 Attachments: h2552_2015.patch, h2552_2015b.patch, h2552_2015b.pdf, h2552_2017.patch, h2552_2017.pdf, h2552_2021.patch, h2552_2021.pdf, h2552_2021_0.20s.patch, h2552_2021_0.20s.pdf -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2588) hdfs jsp pages missing DOCTYPE
hdfs jsp pages missing DOCTYPE -- Key: HDFS-2588 URL: https://issues.apache.org/jira/browse/HDFS-2588 Project: Hadoop HDFS Issue Type: Bug Components: scripts Reporter: Dave Vronay Assignee: Dave Vronay Priority: Trivial Please see parent bug. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2588) hdfs jsp pages missing DOCTYPE
[ https://issues.apache.org/jira/browse/HDFS-2588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HDFS-2588: - Description: Please see parent bug HADOOP-7827. (was: Please see parent bug.) hdfs jsp pages missing DOCTYPE -- Key: HDFS-2588 URL: https://issues.apache.org/jira/browse/HDFS-2588 Project: Hadoop HDFS Issue Type: Bug Components: scripts Reporter: Dave Vronay Assignee: Dave Vronay Priority: Trivial Please see parent bug HADOOP-7827. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-2588) hdfs jsp pages missing DOCTYPE
[ https://issues.apache.org/jira/browse/HDFS-2588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley resolved HDFS-2588. -- Resolution: Fixed Fix Version/s: 0.23.1 0.20.205.1 Hadoop Flags: Reviewed Patch is in parent bug HADOOP-7827. hdfs jsp pages missing DOCTYPE -- Key: HDFS-2588 URL: https://issues.apache.org/jira/browse/HDFS-2588 Project: Hadoop HDFS Issue Type: Bug Components: scripts Reporter: Dave Vronay Assignee: Dave Vronay Priority: Trivial Fix For: 0.20.205.1, 0.23.1 Please see parent bug HADOOP-7827. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2588) hdfs jsp pages missing DOCTYPE
[ https://issues.apache.org/jira/browse/HDFS-2588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HDFS-2588: - Description: Some jsp pages in the UI are missing a DOCTYPE declaration. This causes the pages to render incorrectly on some browsers, such as IE9. Please see parent bug HADOOP-7827 for details and patch. (was: Please see parent bug HADOOP-7827.) hdfs jsp pages missing DOCTYPE -- Key: HDFS-2588 URL: https://issues.apache.org/jira/browse/HDFS-2588 Project: Hadoop HDFS Issue Type: Bug Components: scripts Reporter: Dave Vronay Assignee: Dave Vronay Priority: Trivial Fix For: 0.20.205.1, 0.23.1 Some jsp pages in the UI are missing a DOCTYPE declaration. This causes the pages to render incorrectly on some browsers, such as IE9. Please see parent bug HADOOP-7827 for details and patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2588) hdfs jsp pages missing DOCTYPE
[ https://issues.apache.org/jira/browse/HDFS-2588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13156032#comment-13156032 ] Hudson commented on HDFS-2588: -- Integrated in Hadoop-Hdfs-trunk-Commit #1391 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1391/]) HADOOP-7827 HDFS-2588 MAPREDUCE-3464. jsp pages missing DOCTYPE. Contributed by Dave Vronay. mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1205519 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode/browseBlock.jsp * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode/browseDirectory.jsp * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode/tail.jsp * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/corrupt_files.jsp * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.jsp * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfsnodelist.jsp * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/nn_browsedfscontent.jsp * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary/status.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/analysejobhistory.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/job_authorization_error.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/jobblacklistedtrackers.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/jobconf.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/jobconf_history.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/jobdetails.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/jobdetailshistory.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/jobfailures.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/jobhistory.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/jobqueue_details.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/jobtasks.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/jobtaskshistory.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/jobtracker.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/machines.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/queueinfo.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/queuetable.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/taskdetails.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/taskdetailshistory.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/taskstats.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/taskstatshistory.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/task/tasktracker.jsp hdfs jsp pages missing DOCTYPE -- Key: HDFS-2588 URL: https://issues.apache.org/jira/browse/HDFS-2588 Project: Hadoop HDFS Issue Type: Bug Components: scripts Reporter: Dave Vronay Assignee: Dave Vronay Priority: Trivial Fix For: 0.20.205.1, 0.23.1 Some jsp pages in the UI are missing a DOCTYPE declaration. This causes the pages to render incorrectly on some browsers, such as IE9. Please see parent bug HADOOP-7827 for details and patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2588) hdfs jsp pages missing DOCTYPE
[ https://issues.apache.org/jira/browse/HDFS-2588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13156039#comment-13156039 ] Hudson commented on HDFS-2588: -- Integrated in Hadoop-Common-trunk-Commit #1318 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1318/]) HADOOP-7827 HDFS-2588 MAPREDUCE-3464. jsp pages missing DOCTYPE. Contributed by Dave Vronay. mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1205519 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode/browseBlock.jsp * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode/browseDirectory.jsp * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode/tail.jsp * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/corrupt_files.jsp * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.jsp * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfsnodelist.jsp * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/nn_browsedfscontent.jsp * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary/status.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/analysejobhistory.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/job_authorization_error.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/jobblacklistedtrackers.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/jobconf.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/jobconf_history.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/jobdetails.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/jobdetailshistory.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/jobfailures.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/jobhistory.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/jobqueue_details.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/jobtasks.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/jobtaskshistory.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/jobtracker.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/machines.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/queueinfo.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/queuetable.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/taskdetails.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/taskdetailshistory.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/taskstats.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/taskstatshistory.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/task/tasktracker.jsp hdfs jsp pages missing DOCTYPE -- Key: HDFS-2588 URL: https://issues.apache.org/jira/browse/HDFS-2588 Project: Hadoop HDFS Issue Type: Bug Components: scripts Reporter: Dave Vronay Assignee: Dave Vronay Priority: Trivial Fix For: 0.20.205.1, 0.23.1 Some jsp pages in the UI are missing a DOCTYPE declaration. This causes the pages to render incorrectly on some browsers, such as IE9. Please see parent bug HADOOP-7827 for details and patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2588) hdfs jsp pages missing DOCTYPE
[ https://issues.apache.org/jira/browse/HDFS-2588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13156046#comment-13156046 ] Hudson commented on HDFS-2588: -- Integrated in Hadoop-Hdfs-0.23-Commit #207 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/207/]) HADOOP-7827 HDFS-2588 MAPREDUCE-3464. jsp pages missing DOCTYPE. Contributed by Dave Vronay. mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1205529 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode/browseBlock.jsp * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode/browseDirectory.jsp * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode/tail.jsp * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/corrupt_files.jsp * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.jsp * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfsnodelist.jsp * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/nn_browsedfscontent.jsp * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary/status.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/analysejobhistory.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/job_authorization_error.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/jobblacklistedtrackers.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/jobconf.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/jobconf_history.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/jobdetails.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/jobdetailshistory.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/jobfailures.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/jobhistory.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/jobqueue_details.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/jobtasks.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/jobtaskshistory.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/jobtracker.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/machines.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/queueinfo.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/queuetable.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/taskdetails.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/taskdetailshistory.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/taskstats.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/taskstatshistory.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/task/tasktracker.jsp hdfs jsp pages missing DOCTYPE -- Key: HDFS-2588 URL: https://issues.apache.org/jira/browse/HDFS-2588 Project: Hadoop HDFS Issue Type: Bug Components: scripts Reporter: Dave Vronay Assignee: Dave Vronay Priority: Trivial Fix For: 0.20.205.1, 0.23.1 Some jsp pages in the UI are missing a DOCTYPE declaration. This causes the pages to render incorrectly on some browsers, such as IE9. Please see parent bug HADOOP-7827 for details and patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2588) hdfs jsp pages missing DOCTYPE
[ https://issues.apache.org/jira/browse/HDFS-2588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13156049#comment-13156049 ] Hudson commented on HDFS-2588: -- Integrated in Hadoop-Common-0.23-Commit #210 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/210/]) HADOOP-7827 HDFS-2588 MAPREDUCE-3464. jsp pages missing DOCTYPE. Contributed by Dave Vronay. mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1205529 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode/browseBlock.jsp * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode/browseDirectory.jsp * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode/tail.jsp * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/corrupt_files.jsp * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.jsp * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfsnodelist.jsp * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/nn_browsedfscontent.jsp * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary/status.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/analysejobhistory.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/job_authorization_error.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/jobblacklistedtrackers.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/jobconf.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/jobconf_history.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/jobdetails.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/jobdetailshistory.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/jobfailures.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/jobhistory.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/jobqueue_details.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/jobtasks.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/jobtaskshistory.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/jobtracker.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/machines.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/queueinfo.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/queuetable.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/taskdetails.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/taskdetailshistory.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/taskstats.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/taskstatshistory.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/task/tasktracker.jsp hdfs jsp pages missing DOCTYPE -- Key: HDFS-2588 URL: https://issues.apache.org/jira/browse/HDFS-2588 Project: Hadoop HDFS Issue Type: Bug Components: scripts Reporter: Dave Vronay Assignee: Dave Vronay Priority: Trivial Fix For: 0.20.205.1, 0.23.1 Some jsp pages in the UI are missing a DOCTYPE declaration. This causes the pages to render incorrectly on some browsers, such as IE9. Please see parent bug HADOOP-7827 for details and patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2588) hdfs jsp pages missing DOCTYPE
[ https://issues.apache.org/jira/browse/HDFS-2588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13156059#comment-13156059 ] Hudson commented on HDFS-2588: -- Integrated in Hadoop-Mapreduce-0.23-Commit #221 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/221/]) HADOOP-7827 HDFS-2588 MAPREDUCE-3464. jsp pages missing DOCTYPE. Contributed by Dave Vronay. mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1205529 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode/browseBlock.jsp * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode/browseDirectory.jsp * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode/tail.jsp * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/corrupt_files.jsp * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.jsp * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfsnodelist.jsp * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/nn_browsedfscontent.jsp * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary/status.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/analysejobhistory.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/job_authorization_error.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/jobblacklistedtrackers.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/jobconf.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/jobconf_history.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/jobdetails.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/jobdetailshistory.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/jobfailures.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/jobhistory.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/jobqueue_details.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/jobtasks.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/jobtaskshistory.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/jobtracker.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/machines.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/queueinfo.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/queuetable.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/taskdetails.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/taskdetailshistory.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/taskstats.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/job/taskstatshistory.jsp * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/webapps/task/tasktracker.jsp hdfs jsp pages missing DOCTYPE -- Key: HDFS-2588 URL: https://issues.apache.org/jira/browse/HDFS-2588 Project: Hadoop HDFS Issue Type: Bug Components: scripts Reporter: Dave Vronay Assignee: Dave Vronay Priority: Trivial Fix For: 0.20.205.1, 0.23.1 Some jsp pages in the UI are missing a DOCTYPE declaration. This causes the pages to render incorrectly on some browsers, such as IE9. Please see parent bug HADOOP-7827 for details and patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2588) hdfs jsp pages missing DOCTYPE
[ https://issues.apache.org/jira/browse/HDFS-2588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13156063#comment-13156063 ] Hudson commented on HDFS-2588: -- Integrated in Hadoop-Mapreduce-trunk-Commit #1341 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1341/]) HADOOP-7827 HDFS-2588 MAPREDUCE-3464. jsp pages missing DOCTYPE. Contributed by Dave Vronay. mattf : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1205519 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode/browseBlock.jsp * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode/browseDirectory.jsp * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/datanode/tail.jsp * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/corrupt_files.jsp * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfshealth.jsp * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/dfsnodelist.jsp * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/hdfs/nn_browsedfscontent.jsp * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/webapps/secondary/status.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/analysejobhistory.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/job_authorization_error.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/jobblacklistedtrackers.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/jobconf.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/jobconf_history.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/jobdetails.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/jobdetailshistory.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/jobfailures.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/jobhistory.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/jobqueue_details.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/jobtasks.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/jobtaskshistory.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/jobtracker.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/machines.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/queueinfo.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/queuetable.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/taskdetails.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/taskdetailshistory.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/taskstats.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/job/taskstatshistory.jsp * /hadoop/common/trunk/hadoop-mapreduce-project/src/webapps/task/tasktracker.jsp hdfs jsp pages missing DOCTYPE -- Key: HDFS-2588 URL: https://issues.apache.org/jira/browse/HDFS-2588 Project: Hadoop HDFS Issue Type: Bug Components: scripts Reporter: Dave Vronay Assignee: Dave Vronay Priority: Trivial Fix For: 0.20.205.1, 0.23.1 Some jsp pages in the UI are missing a DOCTYPE declaration. This causes the pages to render incorrectly on some browsers, such as IE9. Please see parent bug HADOOP-7827 for details and patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2246) Shortcut a local client reads to a Datanodes files directly
[ https://issues.apache.org/jira/browse/HDFS-2246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDFS-2246: --- Target Version/s: 0.20.205.1, 0.24.0, 0.23.1 (was: 0.23.1, 0.24.0, 0.20.205.1) Release Note: #New configurations ## dfs.block.local-path-access.user is the key in datanode configuration to specify the user allowed to do short circuit read. ## dfs.client.read.shortcircuit is the key to enable short circuit read at the client side configuration. ## dfs.client.read.shortcircuit.skip.checksum is the key to bypass checksum check at the client side. #By default none of the above are enabled and short circuit read will not kick in. #The feature can be used only for user that has kerberos credentials at the client, therefore map reduce tasks cannot benefit from it in general. was: # dfs.block.local-path-access.user is the key in datanode configuration to specify the user allowed to do short circuit read. # dfs.client.read.shortcircuit is the key to enable short circuit read at the client side configuration. # dfs.client.read.shortcircuit.checksum is the key to bypass checksum check at the client side. By default none of the above are enabled and short circuit read will not kick in. Shortcut a local client reads to a Datanodes files directly --- Key: HDFS-2246 URL: https://issues.apache.org/jira/browse/HDFS-2246 Project: Hadoop HDFS Issue Type: Improvement Reporter: Sanjay Radia Assignee: Jitendra Nath Pandey Fix For: 0.24.0 Attachments: 0001-HDFS-347.-Local-reads.patch, HDFS-2246-branch-0.20-security-205.1.patch, HDFS-2246-branch-0.20-security-205.2.patch, HDFS-2246-branch-0.20-security-205.patch, HDFS-2246-branch-0.20-security-205.patch, HDFS-2246-branch-0.20-security-205.patch, HDFS-2246-branch-0.20-security.3.patch, HDFS-2246-branch-0.20-security.no-softref.patch, HDFS-2246-branch-0.20-security.patch, HDFS-2246-branch-0.20-security.patch, HDFS-2246-branch-0.20-security.patch, HDFS-2246-branch-0.20-security.patch, HDFS-2246-trunk.patch, HDFS-2246-trunk.patch, HDFS-2246-trunk.patch, HDFS-2246-trunk.patch, HDFS-2246-trunk.patch, HDFS-2246-trunk.patch, HDFS-2246.20s.1.patch, HDFS-2246.20s.2.txt, HDFS-2246.20s.3.txt, HDFS-2246.20s.4.txt, HDFS-2246.20s.patch, TestShortCircuitLocalRead.java, localReadShortcut20-security.2patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2246) Shortcut a local client reads to a Datanodes files directly
[ https://issues.apache.org/jira/browse/HDFS-2246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDFS-2246: --- Target Version/s: 0.20.205.1, 0.24.0, 0.23.1 (was: 0.23.1, 0.24.0, 0.20.205.1) Release Note: 1. New configurations a. dfs.block.local-path-access.user is the key in datanode configuration to specify the user allowed to do short circuit read. b. dfs.client.read.shortcircuit is the key to enable short circuit read at the client side configuration. c. dfs.client.read.shortcircuit.skip.checksum is the key to bypass checksum check at the client side. 2. By default none of the above are enabled and short circuit read will not kick in. 3. If security is on, the feature can be used only for user that has kerberos credentials at the client, therefore map reduce tasks cannot benefit from it in general. was: #New configurations ## dfs.block.local-path-access.user is the key in datanode configuration to specify the user allowed to do short circuit read. ## dfs.client.read.shortcircuit is the key to enable short circuit read at the client side configuration. ## dfs.client.read.shortcircuit.skip.checksum is the key to bypass checksum check at the client side. #By default none of the above are enabled and short circuit read will not kick in. #The feature can be used only for user that has kerberos credentials at the client, therefore map reduce tasks cannot benefit from it in general. Shortcut a local client reads to a Datanodes files directly --- Key: HDFS-2246 URL: https://issues.apache.org/jira/browse/HDFS-2246 Project: Hadoop HDFS Issue Type: Improvement Reporter: Sanjay Radia Assignee: Jitendra Nath Pandey Fix For: 0.24.0 Attachments: 0001-HDFS-347.-Local-reads.patch, HDFS-2246-branch-0.20-security-205.1.patch, HDFS-2246-branch-0.20-security-205.2.patch, HDFS-2246-branch-0.20-security-205.patch, HDFS-2246-branch-0.20-security-205.patch, HDFS-2246-branch-0.20-security-205.patch, HDFS-2246-branch-0.20-security.3.patch, HDFS-2246-branch-0.20-security.no-softref.patch, HDFS-2246-branch-0.20-security.patch, HDFS-2246-branch-0.20-security.patch, HDFS-2246-branch-0.20-security.patch, HDFS-2246-branch-0.20-security.patch, HDFS-2246-trunk.patch, HDFS-2246-trunk.patch, HDFS-2246-trunk.patch, HDFS-2246-trunk.patch, HDFS-2246-trunk.patch, HDFS-2246-trunk.patch, HDFS-2246.20s.1.patch, HDFS-2246.20s.2.txt, HDFS-2246.20s.3.txt, HDFS-2246.20s.4.txt, HDFS-2246.20s.patch, TestShortCircuitLocalRead.java, localReadShortcut20-security.2patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2589) unnecessary hftp token fetch and renewal thread
unnecessary hftp token fetch and renewal thread
---
Key: HDFS-2589
URL: https://issues.apache.org/jira/browse/HDFS-2589
Project: Hadoop HDFS
Issue Type: Bug
Components: security
Affects Versions: 0.20.205.1
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Instantiation of the hftp filesystem is causing a token to be implicitly
created and added to a custom token renewal thread. With the new token renewal
feature in the JT, this causes the mapreduce {{obtainTokensForNamenodes}} to
fetch two tokens (an implicit and uncancelled token, and an explicit token) and
leave a spurious renewal thread running. This thread should not be running in
the JT.
After speaking with Owen, the quick solution is to lazy fetch the token, and to
lazy start the renewer thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2589) unnecessary hftp token fetch and renewal thread
[
https://issues.apache.org/jira/browse/HDFS-2589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daryn Sharp updated HDFS-2589:
--
Status: Patch Available (was: Open)
unnecessary hftp token fetch and renewal thread
---
Key: HDFS-2589
URL: https://issues.apache.org/jira/browse/HDFS-2589
Project: Hadoop HDFS
Issue Type: Bug
Components: security
Affects Versions: 0.20.205.1
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Attachments: HDFS-2589.patch
Instantiation of the hftp filesystem is causing a token to be implicitly
created and added to a custom token renewal thread. With the new token
renewal feature in the JT, this causes the mapreduce
{{obtainTokensForNamenodes}} to fetch two tokens (an implicit and uncancelled
token, and an explicit token) and leave a spurious renewal thread running.
This thread should not be running in the JT.
After speaking with Owen, the quick solution is to lazy fetch the token, and
to lazy start the renewer thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2589) unnecessary hftp token fetch and renewal thread
[
https://issues.apache.org/jira/browse/HDFS-2589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daryn Sharp updated HDFS-2589:
--
Attachment: HDFS-2589.patch
implements lazy fetch and thread
unnecessary hftp token fetch and renewal thread
---
Key: HDFS-2589
URL: https://issues.apache.org/jira/browse/HDFS-2589
Project: Hadoop HDFS
Issue Type: Bug
Components: security
Affects Versions: 0.20.205.1
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Attachments: HDFS-2589.patch
Instantiation of the hftp filesystem is causing a token to be implicitly
created and added to a custom token renewal thread. With the new token
renewal feature in the JT, this causes the mapreduce
{{obtainTokensForNamenodes}} to fetch two tokens (an implicit and uncancelled
token, and an explicit token) and leave a spurious renewal thread running.
This thread should not be running in the JT.
After speaking with Owen, the quick solution is to lazy fetch the token, and
to lazy start the renewer thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2257) HftpFileSystem should implement getDelegationTokens
[
https://issues.apache.org/jira/browse/HDFS-2257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13156215#comment-13156215
]
Daryn Sharp commented on HDFS-2257:
---
Would a simple transition approach be for a default impl of
{{getDelegationToken(s)}} to call {{getDelegationToken}}? Once all filesystems
implement {{getDelegationTokens}}, then that method could be abstract?
HftpFileSystem should implement getDelegationTokens
---
Key: HDFS-2257
URL: https://issues.apache.org/jira/browse/HDFS-2257
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Assignee: Daryn Sharp
Attachments: HDFS-2257.patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2589) unnecessary hftp token fetch and renewal thread
[
https://issues.apache.org/jira/browse/HDFS-2589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13156217#comment-13156217
]
Hadoop QA commented on HDFS-2589:
-
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12504914/HDFS-2589.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1607//console
This message is automatically generated.
unnecessary hftp token fetch and renewal thread
---
Key: HDFS-2589
URL: https://issues.apache.org/jira/browse/HDFS-2589
Project: Hadoop HDFS
Issue Type: Bug
Components: security
Affects Versions: 0.20.205.1
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Attachments: HDFS-2589.patch
Instantiation of the hftp filesystem is causing a token to be implicitly
created and added to a custom token renewal thread. With the new token
renewal feature in the JT, this causes the mapreduce
{{obtainTokensForNamenodes}} to fetch two tokens (an implicit and uncancelled
token, and an explicit token) and leave a spurious renewal thread running.
This thread should not be running in the JT.
After speaking with Owen, the quick solution is to lazy fetch the token, and
to lazy start the renewer thread.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2582) Scope dfs.ha.namenodes config by nameservice
[
https://issues.apache.org/jira/browse/HDFS-2582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13156260#comment-13156260
]
Eli Collins commented on HDFS-2582:
---
You're proposed scheme makes sense to me. Perhaps the namespace was left in
the config so each namespace could have its own web UI at some point?
Scope dfs.ha.namenodes config by nameservice
Key: HDFS-2582
URL: https://issues.apache.org/jira/browse/HDFS-2582
Project: Hadoop HDFS
Issue Type: Sub-task
Components: data-node, name-node
Affects Versions: HA branch (HDFS-1623)
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Fix For: HA branch (HDFS-1623)
HDFS-2231 started the process of adding configuration for HA, but one piece
is missing. The current state of the configuration is, I believe:
{{dfs.ha.namenodes}} - a list of identifiers for HA namenodes
{{dfs.federation.nameservices}} - a list of federated nameservices
{{dfs.namenode.rpc-address[.nameservice-id][.namenode-id]}} - some specific
config for the given namenode. If HA or federation is disabled, the extra
components can be elided for backwards compatibility.
The issue here is that there is no easy way to discern which NN is paired
with which other NN. Additionally, adding a new federated nameservice to a
config will require changes to {{dfs.ha.namenodes}} which makes templating
harder. It would be simpler to change {{dfs.ha.namenodes}} to be
nameservice-scoped: {{dfs.ha.namenodes.nameservice-id}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2257) HftpFileSystem should implement getDelegationTokens
[
https://issues.apache.org/jira/browse/HDFS-2257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13156268#comment-13156268
]
Siddharth Seth commented on HDFS-2257:
--
Sounds reasonable to me. We should probably maintain the current behaviour of
returning an empty list if {{getDelegationToken}} returns a null.
HftpFileSystem should implement getDelegationTokens
---
Key: HDFS-2257
URL: https://issues.apache.org/jira/browse/HDFS-2257
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Assignee: Daryn Sharp
Attachments: HDFS-2257.patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2587) Add WebHDFS apt doc
[ https://issues.apache.org/jira/browse/HDFS-2587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-2587: - Attachment: h2587_2023site.tar.gz h2587_2023.patch h2587_2023.patch h2587_2023site.tar.gz - Added everything but the links to javadoc do not work. - The gz file is for reviewing. It was generated by mvn site under hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site. Add WebHDFS apt doc --- Key: HDFS-2587 URL: https://issues.apache.org/jira/browse/HDFS-2587 Project: Hadoop HDFS Issue Type: Task Components: documentation Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Attachments: h2587_2022.patch, h2587_2023.patch, h2587_2023site.tar.gz This issue is to add a WebHDFS doc in apt format in additional to the forrest doc (HDFS-2552). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2498) TestParallelRead times out consistently on Jenkins
[
https://issues.apache.org/jira/browse/HDFS-2498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13156308#comment-13156308
]
Konstantin Shvachko commented on HDFS-2498:
---
See the following assert in the logs when the test actually succeeded:
{code}
Exception in thread ReadWorker-7-/TestParallelRead.dat.0
java.lang.AssertionError: Bad args: 262098 + 46 should be 262144
at org.junit.Assert.fail(Assert.java:91)
at org.junit.Assert.assertTrue(Assert.java:43)
at
org.apache.hadoop.hdfs.TestParallelRead$ReadWorker.read(TestParallelRead.java:137)
at
org.apache.hadoop.hdfs.TestParallelRead$ReadWorker.run(TestParallelRead.java:104)
{code}
This is a wrong condition, should be =
TestParallelRead times out consistently on Jenkins
--
Key: HDFS-2498
URL: https://issues.apache.org/jira/browse/HDFS-2498
Project: Hadoop HDFS
Issue Type: Bug
Components: test
Affects Versions: 0.22.0
Reporter: Konstantin Shvachko
Assignee: Jakob Homan
Priority: Blocker
Fix For: 0.22.0
Attachments: TestParallelRead.txt
During last several Jenkins builds TestParallelRead consistently fails. See
Hadoop-Hdfs-22-branch for logs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2498) TestParallelRead times out consistently on Jenkins
[ https://issues.apache.org/jira/browse/HDFS-2498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13156312#comment-13156312 ] Todd Lipcon commented on HDFS-2498: --- Dup of HDFS-2287 TestParallelRead times out consistently on Jenkins -- Key: HDFS-2498 URL: https://issues.apache.org/jira/browse/HDFS-2498 Project: Hadoop HDFS Issue Type: Bug Components: test Affects Versions: 0.22.0 Reporter: Konstantin Shvachko Assignee: Jakob Homan Priority: Blocker Fix For: 0.22.0 Attachments: TestParallelRead.txt During last several Jenkins builds TestParallelRead consistently fails. See Hadoop-Hdfs-22-branch for logs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2587) Add WebHDFS apt doc
[ https://issues.apache.org/jira/browse/HDFS-2587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13156320#comment-13156320 ] Suresh Srinivas commented on HDFS-2587: --- +1 for the patch. Thanks for fixing federation doc also. Add WebHDFS apt doc --- Key: HDFS-2587 URL: https://issues.apache.org/jira/browse/HDFS-2587 Project: Hadoop HDFS Issue Type: Task Components: documentation Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Attachments: h2587_2022.patch, h2587_2023.patch, h2587_2023b.patch, h2587_2023b_site.tar.gz, h2587_2023site.tar.gz This issue is to add a WebHDFS doc in apt format in additional to the forrest doc (HDFS-2552). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2498) TestParallelRead times out consistently on Jenkins
[
https://issues.apache.org/jira/browse/HDFS-2498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13156343#comment-13156343
]
Konstantin Shvachko commented on HDFS-2498:
---
You are right, Todd. I'll incorporate.
I see that even when TestParallelRead succeeds (rather rarely) on Jenkins its
running time is around 14 minutes. While on my linux box it completes in less
than 20 seconds.
And in the log attached I see
{code}
[junit] 2011-10-24 19:58:35,474 INFO hdfs.TestParallelRead
(TestParallelRead.java:runParallelRead(269)) - === Report: 16 threads read
941298 KB (across 1 file(s)) in 550.051s; average 1711.292225629987 KB/s
{code}
So the second runParallelRead() is actually running more than 9 minutes
(550.051s). Again, on my box it completes in only 2-3 seconds.
Is it possible that because ReadWorker,run() does not catch Error (only
Exception) the thread cannot complete?
TestParallelRead times out consistently on Jenkins
--
Key: HDFS-2498
URL: https://issues.apache.org/jira/browse/HDFS-2498
Project: Hadoop HDFS
Issue Type: Bug
Components: test
Affects Versions: 0.22.0
Reporter: Konstantin Shvachko
Assignee: Jakob Homan
Priority: Blocker
Fix For: 0.22.0
Attachments: TestParallelRead.txt
During last several Jenkins builds TestParallelRead consistently fails. See
Hadoop-Hdfs-22-branch for logs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2587) Add WebHDFS apt doc
[ https://issues.apache.org/jira/browse/HDFS-2587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13156345#comment-13156345 ] Hudson commented on HDFS-2587: -- Integrated in Hadoop-Mapreduce-0.23-Commit #222 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/222/]) svn merge -c 1205626 from trunk for HDFS-2587. szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1205630 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/Federation.apt.vm * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/WebHDFS.apt.vm * /hadoop/common/branches/branch-0.23/hadoop-project/src/site * /hadoop/common/branches/branch-0.23/hadoop-project/src/site/site.xml Add WebHDFS apt doc --- Key: HDFS-2587 URL: https://issues.apache.org/jira/browse/HDFS-2587 Project: Hadoop HDFS Issue Type: Task Components: documentation Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Fix For: 0.24.0, 0.23.1 Attachments: h2587_2022.patch, h2587_2023.patch, h2587_2023b.patch, h2587_2023b_site.tar.gz, h2587_2023site.tar.gz This issue is to add a WebHDFS doc in apt format in additional to the forrest doc (HDFS-2552). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2498) TestParallelRead times out consistently on Jenkins
[ https://issues.apache.org/jira/browse/HDFS-2498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13156349#comment-13156349 ] Todd Lipcon commented on HDFS-2498: --- That's strange that it's so slow - hundreds of times slower. Is the Jenkins build directory hosted on NFS or something? Is this the Apache jenkins boxes or one of your internal hosts? TestParallelRead times out consistently on Jenkins -- Key: HDFS-2498 URL: https://issues.apache.org/jira/browse/HDFS-2498 Project: Hadoop HDFS Issue Type: Bug Components: test Affects Versions: 0.22.0 Reporter: Konstantin Shvachko Assignee: Jakob Homan Priority: Blocker Fix For: 0.22.0 Attachments: TestParallelRead.txt During last several Jenkins builds TestParallelRead consistently fails. See Hadoop-Hdfs-22-branch for logs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2498) TestParallelRead times out consistently on Jenkins
[
https://issues.apache.org/jira/browse/HDFS-2498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13156353#comment-13156353
]
Konstantin Shvachko commented on HDFS-2498:
---
This is Apache Jenkins. Here is how I just reproduced it:
https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Hdfs-22-branch/113/console
Login to Jenkins.
Go to: https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Hdfs-22-branch/
Click on Build now
In the BUILD_ARGS add {{-Dtestcase=TestParallelRead -Dtest.output=yes}}
In the BUILD_TARGETS replace with {{clean test}}
TestParallelRead times out consistently on Jenkins
--
Key: HDFS-2498
URL: https://issues.apache.org/jira/browse/HDFS-2498
Project: Hadoop HDFS
Issue Type: Bug
Components: test
Affects Versions: 0.22.0
Reporter: Konstantin Shvachko
Assignee: Jakob Homan
Priority: Blocker
Fix For: 0.22.0
Attachments: TestParallelRead.txt
During last several Jenkins builds TestParallelRead consistently fails. See
Hadoop-Hdfs-22-branch for logs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2587) Add WebHDFS apt doc
[ https://issues.apache.org/jira/browse/HDFS-2587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13156396#comment-13156396 ] Hudson commented on HDFS-2587: -- Integrated in Hadoop-Common-trunk-Commit #1319 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1319/]) HDFS-2587. Add apt doc for WebHDFS REST API. szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1205626 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/Federation.apt.vm * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/WebHDFS.apt.vm * /hadoop/common/trunk/hadoop-project/src/site/site.xml Add WebHDFS apt doc --- Key: HDFS-2587 URL: https://issues.apache.org/jira/browse/HDFS-2587 Project: Hadoop HDFS Issue Type: Task Components: documentation Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Fix For: 0.24.0, 0.23.1 Attachments: h2587_2022.patch, h2587_2023.patch, h2587_2023b.patch, h2587_2023b_site.tar.gz, h2587_2023site.tar.gz This issue is to add a WebHDFS doc in apt format in additional to the forrest doc (HDFS-2552). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2587) Add WebHDFS apt doc
[ https://issues.apache.org/jira/browse/HDFS-2587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13156403#comment-13156403 ] Hudson commented on HDFS-2587: -- Integrated in Hadoop-Common-0.23-Commit #211 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/211/]) svn merge -c 1205626 from trunk for HDFS-2587. szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1205630 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/Federation.apt.vm * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/WebHDFS.apt.vm * /hadoop/common/branches/branch-0.23/hadoop-project/src/site * /hadoop/common/branches/branch-0.23/hadoop-project/src/site/site.xml Add WebHDFS apt doc --- Key: HDFS-2587 URL: https://issues.apache.org/jira/browse/HDFS-2587 Project: Hadoop HDFS Issue Type: Task Components: documentation Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Fix For: 0.24.0, 0.23.1 Attachments: h2587_2022.patch, h2587_2023.patch, h2587_2023b.patch, h2587_2023b_site.tar.gz, h2587_2023site.tar.gz This issue is to add a WebHDFS doc in apt format in additional to the forrest doc (HDFS-2552). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2587) Add WebHDFS apt doc
[ https://issues.apache.org/jira/browse/HDFS-2587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13156406#comment-13156406 ] Hudson commented on HDFS-2587: -- Integrated in Hadoop-Hdfs-trunk-Commit #1392 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1392/]) HDFS-2587. Add apt doc for WebHDFS REST API. szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1205626 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/Federation.apt.vm * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/WebHDFS.apt.vm * /hadoop/common/trunk/hadoop-project/src/site/site.xml Add WebHDFS apt doc --- Key: HDFS-2587 URL: https://issues.apache.org/jira/browse/HDFS-2587 Project: Hadoop HDFS Issue Type: Task Components: documentation Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Fix For: 0.24.0, 0.23.1 Attachments: h2587_2022.patch, h2587_2023.patch, h2587_2023b.patch, h2587_2023b_site.tar.gz, h2587_2023site.tar.gz This issue is to add a WebHDFS doc in apt format in additional to the forrest doc (HDFS-2552). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1417) Add @Override annotation to SimulatedFSDataset methods that implement FSDatasetInterface
[ https://issues.apache.org/jira/browse/HDFS-1417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-1417: -- Fix Version/s: 0.22.0 Add @Override annotation to SimulatedFSDataset methods that implement FSDatasetInterface Key: HDFS-1417 URL: https://issues.apache.org/jira/browse/HDFS-1417 Project: Hadoop HDFS Issue Type: Improvement Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 0.22.0 Attachments: HDFS-1417.patch @Override annotations are inconsistently added to methods implementing the interface. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1500) TestOfflineImageViewer failing on trunk
[ https://issues.apache.org/jira/browse/HDFS-1500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-1500: -- Fix Version/s: 0.22.0 TestOfflineImageViewer failing on trunk --- Key: HDFS-1500 URL: https://issues.apache.org/jira/browse/HDFS-1500 Project: Hadoop HDFS Issue Type: Bug Components: test, tools Affects Versions: 0.22.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 0.22.0 Attachments: hdfs-1500.txt Testcase: testOIV took 22.679 sec FAILED Failed reading valid file: No image processor to read version -26 is available. junit.framework.AssertionFailedError: Failed reading valid file: No image processor to read version -26 is available. at org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer.outputOfLSVisitor(TestOfflineImageViewer.java:171) at org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer.testOIV(TestOfflineImageViewer.java:86) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1055) Improve thread naming for DataXceivers
[ https://issues.apache.org/jira/browse/HDFS-1055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-1055: -- Fix Version/s: 0.22.0 Improve thread naming for DataXceivers -- Key: HDFS-1055 URL: https://issues.apache.org/jira/browse/HDFS-1055 Project: Hadoop HDFS Issue Type: Improvement Components: data-node Affects Versions: 0.22.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 0.22.0 Attachments: dataxceiver-merged.patch, dataxceiver.patch, hdfs-1055-1.patch, hdfs-1055-branch20.txt The DataXceiver threads are named using the default Daemon naming, which is Runnable.toString(). Currently this isn't implemented, so threads have names like org.apache.hadoop.hdfs.server.datanode.DataXceiver@579c9a6b. It would be very handy for debugging (and even ops maybe) to have a better name like DataXceiver for client 1.2.3.4 [reading block_234254242] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1036) in DelegationTokenFetch dfs.getURI returns no port
[ https://issues.apache.org/jira/browse/HDFS-1036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-1036: -- Fix Version/s: 0.22.0 in DelegationTokenFetch dfs.getURI returns no port -- Key: HDFS-1036 URL: https://issues.apache.org/jira/browse/HDFS-1036 Project: Hadoop HDFS Issue Type: Bug Reporter: Boris Shkolnik Assignee: Boris Shkolnik Fix For: 0.22.0 Attachments: HDFS-1036-BP20-1.patch, HDFS-1036-BP20-Fix.patch, HDFS-1036-BP20.patch, HDFS-1036-doc.patch, HDFS-1036.patch, fetchdt_doc.patch dfs.getUri().getPort() returns -1. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1038) In nn_browsedfscontent.jsp fetch delegation token only if security is enabled.
[ https://issues.apache.org/jira/browse/HDFS-1038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-1038: -- Fix Version/s: 0.22.0 In nn_browsedfscontent.jsp fetch delegation token only if security is enabled. -- Key: HDFS-1038 URL: https://issues.apache.org/jira/browse/HDFS-1038 Project: Hadoop HDFS Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Fix For: 0.22.0 Attachments: HDFS-1038-y20.1.patch, HDFS-1038.1.patch, HDFS-1038.2.patch, HDFS-1038.3.patch nn_browsedfscontent.jsp calls getDelegationToken even if security is disabled, which causes NPE. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1039) Service should be set in the token in JspHelper.getUGI
[ https://issues.apache.org/jira/browse/HDFS-1039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-1039: -- Fix Version/s: 0.22.0 Service should be set in the token in JspHelper.getUGI -- Key: HDFS-1039 URL: https://issues.apache.org/jira/browse/HDFS-1039 Project: Hadoop HDFS Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Fix For: 0.22.0 Attachments: HDFS-1039-y20.1.patch, HDFS-1039-y20.2.1.patch, HDFS-1039-y20.2.patch, HDFS-1039.2.patch, HDFS-1039.4.patch The delegation token added to the UGI in getUGI method in the JspHelper does not have service set. Therefore, this token cannot be used to connect to the namenode. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2514) Link resolution bug for intermediate symlinks with relative targets
[ https://issues.apache.org/jira/browse/HDFS-2514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-2514: -- Fix Version/s: 0.22.0 Link resolution bug for intermediate symlinks with relative targets --- Key: HDFS-2514 URL: https://issues.apache.org/jira/browse/HDFS-2514 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.21.0, 0.22.0, 0.23.0 Reporter: Eli Collins Assignee: Eli Collins Fix For: 0.22.0 Attachments: hdfs-2514-1.patch, hdfs-2514-2.patch, hdfs-2514-3.patch There's a bug in the way the Namenode resolves intermediate symlinks (ie the symlink is not the final path component) in paths when the symlink's target is a relative path. Will post the full description in the first comment. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1146) Javadoc for getDelegationTokenSecretManager in FSNamesystem
[ https://issues.apache.org/jira/browse/HDFS-1146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-1146: -- Fix Version/s: 0.22.0 Javadoc for getDelegationTokenSecretManager in FSNamesystem --- Key: HDFS-1146 URL: https://issues.apache.org/jira/browse/HDFS-1146 Project: Hadoop HDFS Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Fix For: 0.22.0 Attachments: HDFS-1146-y20.1.patch, HDFS-1146.1.patch Javadoc is missing for public method getDelegationTokenSecretManager in FSNamesystem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1334) open in HftpFileSystem does not add delegation tokens to the url.
[ https://issues.apache.org/jira/browse/HDFS-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-1334: -- Fix Version/s: 0.22.0 open in HftpFileSystem does not add delegation tokens to the url. - Key: HDFS-1334 URL: https://issues.apache.org/jira/browse/HDFS-1334 Project: Hadoop HDFS Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Fix For: 0.22.0 Attachments: HDFS-1334.1.patch open method in HftpFileSystem uses ByteRangeInputStream for url connection. But it does not add the delegation tokens, even if security is enabled, to the url before passing it to the ByteRangeInputStream. Therefore request fails if security is enabled. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1946) HDFS part of HADOOP-7291
[ https://issues.apache.org/jira/browse/HDFS-1946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-1946: -- Fix Version/s: 0.22.0 HDFS part of HADOOP-7291 Key: HDFS-1946 URL: https://issues.apache.org/jira/browse/HDFS-1946 Project: Hadoop HDFS Issue Type: Task Reporter: Eli Collins Assignee: Eli Collins Fix For: 0.22.0 Attachments: hdfs-1946-1.patch The hudson-test-patch target needs to be updated to not pass python.home since this argument is no longer needed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1301) TestHDFSProxy need to use server side conf for ProxyUser stuff.
[ https://issues.apache.org/jira/browse/HDFS-1301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-1301: -- Fix Version/s: 0.22.0 TestHDFSProxy need to use server side conf for ProxyUser stuff. --- Key: HDFS-1301 URL: https://issues.apache.org/jira/browse/HDFS-1301 Project: Hadoop HDFS Issue Type: Bug Reporter: Boris Shkolnik Assignee: Boris Shkolnik Fix For: 0.22.0 Attachments: HDFS-1301-1.patch, HDFS-1301-BP20-1.patch, HDFS-1301-BP20.patch, HDFS-1301.patch currently TestHdfsProxy sets hadoop.proxyuser.USER.groups in local copy of configuration. But ProxyUsers only looks at the server side config. For test we can uses static method in ProxyUsers to load the config. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1190) Remove unused getNamenode() method from DataNode.
[ https://issues.apache.org/jira/browse/HDFS-1190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-1190: -- Fix Version/s: 0.22.0 Remove unused getNamenode() method from DataNode. - Key: HDFS-1190 URL: https://issues.apache.org/jira/browse/HDFS-1190 Project: Hadoop HDFS Issue Type: Improvement Components: data-node Reporter: Jeff Ames Assignee: Jeff Ames Priority: Minor Fix For: 0.22.0 Attachments: HDFS-1190.patch This patch removes the getNamenode() method from DataNode, which appears to be unused. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1408) Herriot NN and DN clients should vend statistics
[ https://issues.apache.org/jira/browse/HDFS-1408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-1408: -- Fix Version/s: 0.22.0 Herriot NN and DN clients should vend statistics Key: HDFS-1408 URL: https://issues.apache.org/jira/browse/HDFS-1408 Project: Hadoop HDFS Issue Type: Improvement Components: test Affects Versions: 0.22.0 Reporter: Al Thompson Assignee: Konstantin Boudnik Fix For: 0.22.0 Attachments: HADOOP-6927.y20s.patch, HDFS-1408.patch, HDFS-1408.patch, HDFS-1408.patch, HDFS-1408.patch, add_JMXlisteners.patch, jmx.patch, jmx.patch, jmx.patch, jmx.patch The HDFS web user interface serves useful information through dfshealth.jsp and dfsnodelist.jsp. The Herriot interface to the namenode and datanode (as implemented in NNClient and DNClient, respectively) would benefit from the addition of some way to channel this information. In the case of DNClient this can be an injected method that returns a DatanodeDescriptor relevant to the underlying datanode. There seems to be no analagous NamenodeDescriptor. It may be useful to add this as a facade to a visitor that aggregates values across the filesystem datanodes. These values are (from dfshealth JSP): Configured Capacity DFS Used Non DFS Used DFS Remaining DFS Used% DFS Remaining% Live Nodes Dead Nodes Decommissioning Nodes Number of Under-Replicated Blocks Attributes reflecting the web user interface header may also be useful such as When-Started, Version, When-Compiled, and Upgrade-Status. A NamenodeDescriptor would essentially push down the code in dfshealth web UI behind a more general abstraction. If it is objectionable to make this class available in HDFS, perhaps this could be packaged in a Herriot specific way. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1387) Update HDFS permissions guide for security
[ https://issues.apache.org/jira/browse/HDFS-1387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-1387: -- Fix Version/s: 0.22.0 Update HDFS permissions guide for security -- Key: HDFS-1387 URL: https://issues.apache.org/jira/browse/HDFS-1387 Project: Hadoop HDFS Issue Type: Improvement Components: documentation, security Affects Versions: 0.22.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 0.22.0 Attachments: hdfs-1387.txt, hdfs_permissions_guide.pdf The HDFS permissions guide currently makes several statements that are no longer correct now that we provide security. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1003) authorization checks for inter-server protocol (based on HADOOP-6600)
[ https://issues.apache.org/jira/browse/HDFS-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-1003: -- Fix Version/s: 0.22.0 authorization checks for inter-server protocol (based on HADOOP-6600) - Key: HDFS-1003 URL: https://issues.apache.org/jira/browse/HDFS-1003 Project: Hadoop HDFS Issue Type: New Feature Reporter: Boris Shkolnik Assignee: Boris Shkolnik Fix For: 0.22.0 Attachments: HDFS-1003.patch authorization checks for inter-server protocol (based on HADOOP-6600) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2577) HA: NN fails to start since it tries to start secret manager in safemode
[ https://issues.apache.org/jira/browse/HDFS-2577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13156432#comment-13156432 ] Todd Lipcon commented on HDFS-2577: --- The LM changes don't need to be done on trunk, because on the non-HA branch, FSNamesystem.isRunning is an accurate gauge of whether the LM should run. On HDFS it's not the case anymore since in standby mode, the LM doesn't run, even though the FSN is running. I'll check this into the HA branch, thanks! HA: NN fails to start since it tries to start secret manager in safemode Key: HDFS-2577 URL: https://issues.apache.org/jira/browse/HDFS-2577 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Affects Versions: HA branch (HDFS-1623) Reporter: Todd Lipcon Assignee: Todd Lipcon Attachments: hdfs-2577.txt After HDFS-2301, the NN fails to start with the following: Caused by: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot log master key update in safe mode. Name node is in safe mode. The reported blocks 0 needs additional 5 blocks to reach the threshold 1. of total blocks 4. Safe mode will be turned off automatically. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.logUpdateMasterKey(FSNamesystem.java:4259) at org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenSecretManager.logUpdateMasterKey(DelegationTokenSecretManager.java:285) at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.updateCurrentKey(AbstractDelegationTokenSecretManager.java:143) at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.startThreads(AbstractDelegationTokenSecretManager.java:98) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startSecretManager(FSNamesystem.java:386) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startActiveServices(FSNamesystem.java:440) at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.startActiveServices(NameNode.java:937) at org.apache.hadoop.hdfs.server.namenode.ha.ActiveState.enterState(ActiveState.java:57) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-503) Implement erasure coding as a layer on HDFS
[ https://issues.apache.org/jira/browse/HDFS-503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13156434#comment-13156434 ] Hemanth Makkapati commented on HDFS-503: Hey, I am a beginner with hadoop and started delving into the code only lately. As I was trying to get RAID up and running, I observed the following exception in the log ERROR org.apache.hadoop.raid.RaidNode: java.lang.NullPointerException at org.apache.hadoop.raid.RaidNode.tmpHarPathForCode(RaidNode.java:1491) at org.apache.hadoop.raid.RaidNode.doHar(RaidNode.java:1217) at org.apache.hadoop.raid.RaidNode.access$300(RaidNode.java:73) at org.apache.hadoop.raid.RaidNode$HarMonitor.run(RaidNode.java:1371) at java.lang.Thread.run(Thread.java:636) The reason for this seems to be the absence of 'erasurecode' tag in raid configuration file which, in my case, is very similar to the sample configuration file provided. Once the tag is introduced, which is allowed to assume either XOR or RS, I didn't see any exception. Also, the README file also doesn't mention anything about such a tag. Please confirm if my observation is correct. Thought of posting it here for the benefit of others. BTW, I checked out code from the trunk. Thank you. Implement erasure coding as a layer on HDFS --- Key: HDFS-503 URL: https://issues.apache.org/jira/browse/HDFS-503 Project: Hadoop HDFS Issue Type: New Feature Components: contrib/raid Reporter: dhruba borthakur Assignee: dhruba borthakur Fix For: 0.21.0 Attachments: raid1.txt, raid2.txt The goal of this JIRA is to discuss how the cost of raw storage for a HDFS file system can be reduced. Keeping three copies of the same data is very costly, especially when the size of storage is huge. One idea is to reduce the replication factor and do erasure coding of a set of blocks so that the over probability of failure of a block remains the same as before. Many forms of error-correcting codes are available, see http://en.wikipedia.org/wiki/Erasure_code. Also, recent research from CMU has described DiskReduce https://opencirrus.org/system/files/Gibson-OpenCirrus-June9-09.ppt. My opinion is to discuss implementation strategies that are not part of base HDFS, but is a layer on top of HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-2577) HA: NN fails to start since it tries to start secret manager in safemode
[ https://issues.apache.org/jira/browse/HDFS-2577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon resolved HDFS-2577. --- Resolution: Fixed Fix Version/s: HA branch (HDFS-1623) Hadoop Flags: Reviewed HA: NN fails to start since it tries to start secret manager in safemode Key: HDFS-2577 URL: https://issues.apache.org/jira/browse/HDFS-2577 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Affects Versions: HA branch (HDFS-1623) Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: HA branch (HDFS-1623) Attachments: hdfs-2577.txt After HDFS-2301, the NN fails to start with the following: Caused by: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot log master key update in safe mode. Name node is in safe mode. The reported blocks 0 needs additional 5 blocks to reach the threshold 1. of total blocks 4. Safe mode will be turned off automatically. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.logUpdateMasterKey(FSNamesystem.java:4259) at org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenSecretManager.logUpdateMasterKey(DelegationTokenSecretManager.java:285) at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.updateCurrentKey(AbstractDelegationTokenSecretManager.java:143) at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.startThreads(AbstractDelegationTokenSecretManager.java:98) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startSecretManager(FSNamesystem.java:386) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startActiveServices(FSNamesystem.java:440) at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.startActiveServices(NameNode.java:937) at org.apache.hadoop.hdfs.server.namenode.ha.ActiveState.enterState(ActiveState.java:57) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-1617) CLONE to COMMON - Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file
[ https://issues.apache.org/jira/browse/HDFS-1617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko reopened HDFS-1617: --- CLONE to COMMON - Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file Key: HDFS-1617 URL: https://issues.apache.org/jira/browse/HDFS-1617 Project: Hadoop HDFS Issue Type: Sub-task Components: data-node Affects Versions: 0.20.2 Reporter: Matt Foley Assignee: Matt Foley Fix For: 0.22.0 It was a bit of a puzzle why we can do a full scan of a disk in about 30 seconds during FSDir() or getVolumeMap(), but the same disk took 11 minutes to do Upgrade replication via hardlinks. It turns out that the org.apache.hadoop.fs.FileUtil.createHardLink() method does an outcall to Runtime.getRuntime().exec(), to utilize native filesystem hardlink capability. So it is forking a full-weight external process, and we call it on each individual file to be replicated. As a simple check on the possible cost of this approach, I built a Perl test script (under Linux on a production-class datanode). Perl also uses a compiled and optimized p-code engine, and it has both native support for hardlinks and the ability to do exec. - A simple script to create 256,000 files in a directory tree organized like the Datanode, took 10 seconds to run. - Replicating that directory tree using hardlinks, the same way as the Datanode, took 12 seconds using native hardlink support. - The same replication using outcalls to exec, one per file, took 256 seconds! - Batching the calls, and doing 'exec' once per directory instead of once per file, took 16 seconds. Obviously, your mileage will vary based on the number of blocks per volume. A volume with less than about 4000 blocks will have only 65 directories. A volume with more than 4K and less than about 250K blocks will have 4200 directories (more or less). And there are two files per block (the data file and the .meta file). So the average number of files per directory may vary from 2:1 to 500:1. A node with 50K blocks and four volumes will have 25K files per volume, or an average of about 6:1. So this change may be expected to take it down from, say, 12 minutes per volume to 2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-1617) CLONE to COMMON - Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file
[ https://issues.apache.org/jira/browse/HDFS-1617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko resolved HDFS-1617. --- Resolution: Duplicate CLONE to COMMON - Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file Key: HDFS-1617 URL: https://issues.apache.org/jira/browse/HDFS-1617 Project: Hadoop HDFS Issue Type: Sub-task Components: data-node Affects Versions: 0.20.2 Reporter: Matt Foley Assignee: Matt Foley Fix For: 0.22.0 It was a bit of a puzzle why we can do a full scan of a disk in about 30 seconds during FSDir() or getVolumeMap(), but the same disk took 11 minutes to do Upgrade replication via hardlinks. It turns out that the org.apache.hadoop.fs.FileUtil.createHardLink() method does an outcall to Runtime.getRuntime().exec(), to utilize native filesystem hardlink capability. So it is forking a full-weight external process, and we call it on each individual file to be replicated. As a simple check on the possible cost of this approach, I built a Perl test script (under Linux on a production-class datanode). Perl also uses a compiled and optimized p-code engine, and it has both native support for hardlinks and the ability to do exec. - A simple script to create 256,000 files in a directory tree organized like the Datanode, took 10 seconds to run. - Replicating that directory tree using hardlinks, the same way as the Datanode, took 12 seconds using native hardlink support. - The same replication using outcalls to exec, one per file, took 256 seconds! - Batching the calls, and doing 'exec' once per directory instead of once per file, took 16 seconds. Obviously, your mileage will vary based on the number of blocks per volume. A volume with less than about 4000 blocks will have only 65 directories. A volume with more than 4K and less than about 250K blocks will have 4200 directories (more or less). And there are two files per block (the data file and the .meta file). So the average number of files per directory may vary from 2:1 to 500:1. A node with 50K blocks and four volumes will have 25K files per volume, or an average of about 6:1. So this change may be expected to take it down from, say, 12 minutes per volume to 2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-1496) TestStorageRestore is failing after HDFS-903 fix
[ https://issues.apache.org/jira/browse/HDFS-1496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko reopened HDFS-1496: --- TestStorageRestore is failing after HDFS-903 fix Key: HDFS-1496 URL: https://issues.apache.org/jira/browse/HDFS-1496 Project: Hadoop HDFS Issue Type: Bug Components: test Affects Versions: 0.22.0, 0.23.0 Reporter: Konstantin Boudnik Assignee: Hairong Kuang Priority: Blocker Fix For: 0.22.0 Attachments: HDFS-1496.sh, HDFS-1496.sh, HDFS-1496.sh TestStorageRestore seems to be failing after HDFS-903 commit. Running git bisect confirms it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-1496) TestStorageRestore is failing after HDFS-903 fix
[ https://issues.apache.org/jira/browse/HDFS-1496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko resolved HDFS-1496. --- Resolution: Duplicate TestStorageRestore is failing after HDFS-903 fix Key: HDFS-1496 URL: https://issues.apache.org/jira/browse/HDFS-1496 Project: Hadoop HDFS Issue Type: Bug Components: test Affects Versions: 0.22.0, 0.23.0 Reporter: Konstantin Boudnik Assignee: Hairong Kuang Priority: Blocker Fix For: 0.22.0 Attachments: HDFS-1496.sh, HDFS-1496.sh, HDFS-1496.sh TestStorageRestore seems to be failing after HDFS-903 commit. Running git bisect confirms it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-1084) TestDFSShell fails in trunk.
[
https://issues.apache.org/jira/browse/HDFS-1084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko reopened HDFS-1084:
---
TestDFSShell fails in trunk.
Key: HDFS-1084
URL: https://issues.apache.org/jira/browse/HDFS-1084
Project: Hadoop HDFS
Issue Type: Bug
Components: test
Affects Versions: 0.22.0
Reporter: Konstantin Shvachko
Assignee: Po Cheung
Priority: Blocker
Fix For: 0.22.0
{{TestDFSShell.testFilePermissions()}} fails on an assert attached below. I
see it on my Linux box. Don't see it failing with Hudson, and the same test
runs fine in 0.21 branch.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-1084) TestDFSShell fails in trunk.
[
https://issues.apache.org/jira/browse/HDFS-1084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko resolved HDFS-1084.
---
Resolution: Duplicate
TestDFSShell fails in trunk.
Key: HDFS-1084
URL: https://issues.apache.org/jira/browse/HDFS-1084
Project: Hadoop HDFS
Issue Type: Bug
Components: test
Affects Versions: 0.22.0
Reporter: Konstantin Shvachko
Assignee: Po Cheung
Priority: Blocker
Fix For: 0.22.0
{{TestDFSShell.testFilePermissions()}} fails on an assert attached below. I
see it on my Linux box. Don't see it failing with Hudson, and the same test
runs fine in 0.21 branch.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2582) Scope dfs.ha.namenodes config by nameservice
[
https://issues.apache.org/jira/browse/HDFS-2582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated HDFS-2582:
--
Attachment: hdfs-2582.txt
Here's a patch implementing the proposal.
There's one TODO in there for fixing TestGetConf - I'm working on fixing that
now.
I haven't run all of the unit tests yet, but I did run some of the
federation-related tests as well as some of the HA-related tests and they pass.
I also ran a selection of general HDFS tests and they pass.
The patch does introduce some more TODOs in places like the balancer, where we
are currently doing some custom RPC connections to NNs -- will file a followup
JIRA for that when this is committed.
Scope dfs.ha.namenodes config by nameservice
Key: HDFS-2582
URL: https://issues.apache.org/jira/browse/HDFS-2582
Project: Hadoop HDFS
Issue Type: Sub-task
Components: data-node, name-node
Affects Versions: HA branch (HDFS-1623)
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Fix For: HA branch (HDFS-1623)
Attachments: hdfs-2582.txt
HDFS-2231 started the process of adding configuration for HA, but one piece
is missing. The current state of the configuration is, I believe:
{{dfs.ha.namenodes}} - a list of identifiers for HA namenodes
{{dfs.federation.nameservices}} - a list of federated nameservices
{{dfs.namenode.rpc-address[.nameservice-id][.namenode-id]}} - some specific
config for the given namenode. If HA or federation is disabled, the extra
components can be elided for backwards compatibility.
The issue here is that there is no easy way to discern which NN is paired
with which other NN. Additionally, adding a new federated nameservice to a
config will require changes to {{dfs.ha.namenodes}} which makes templating
harder. It would be simpler to change {{dfs.ha.namenodes}} to be
nameservice-scoped: {{dfs.ha.namenodes.nameservice-id}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2246) Shortcut a local client reads to a Datanodes files directly
[ https://issues.apache.org/jira/browse/HDFS-2246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13156471#comment-13156471 ] Hudson commented on HDFS-2246: -- Integrated in Hadoop-Mapreduce-0.23-Commit #223 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/223/]) Merged r1204792 from trunk for HDFS-2246. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1205694 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/BlockLocalPathInfo.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockMetadataHeader.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetInterface.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestConnCache.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java Shortcut a local client reads to a Datanodes files directly --- Key: HDFS-2246 URL: https://issues.apache.org/jira/browse/HDFS-2246 Project: Hadoop HDFS Issue Type: Improvement Reporter: Sanjay Radia Assignee: Jitendra Nath Pandey Fix For: 0.24.0 Attachments: 0001-HDFS-347.-Local-reads.patch, HDFS-2246-branch-0.20-security-205.1.patch, HDFS-2246-branch-0.20-security-205.2.patch, HDFS-2246-branch-0.20-security-205.patch, HDFS-2246-branch-0.20-security-205.patch, HDFS-2246-branch-0.20-security-205.patch, HDFS-2246-branch-0.20-security.3.patch, HDFS-2246-branch-0.20-security.no-softref.patch, HDFS-2246-branch-0.20-security.patch, HDFS-2246-branch-0.20-security.patch, HDFS-2246-branch-0.20-security.patch, HDFS-2246-branch-0.20-security.patch, HDFS-2246-trunk.patch, HDFS-2246-trunk.patch, HDFS-2246-trunk.patch, HDFS-2246-trunk.patch, HDFS-2246-trunk.patch, HDFS-2246-trunk.patch, HDFS-2246.20s.1.patch, HDFS-2246.20s.2.txt, HDFS-2246.20s.3.txt, HDFS-2246.20s.4.txt, HDFS-2246.20s.patch, TestShortCircuitLocalRead.java, localReadShortcut20-security.2patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2590) Some links in WebHDFS forrest doc do not work
Some links in WebHDFS forrest doc do not work - Key: HDFS-2590 URL: https://issues.apache.org/jira/browse/HDFS-2590 Project: Hadoop HDFS Issue Type: Bug Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Some links are pointing to DistributedFileSystem javadoc but the javadoc of DistributedFileSystem is not generated by default. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2590) Some links in WebHDFS forrest doc do not work
[ https://issues.apache.org/jira/browse/HDFS-2590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-2590: - Component/s: documentation Some links in WebHDFS forrest doc do not work - Key: HDFS-2590 URL: https://issues.apache.org/jira/browse/HDFS-2590 Project: Hadoop HDFS Issue Type: Bug Components: documentation Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Some links are pointing to DistributedFileSystem javadoc but the javadoc of DistributedFileSystem is not generated by default. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2582) Scope dfs.ha.namenodes config by nameservice
[
https://issues.apache.org/jira/browse/HDFS-2582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated HDFS-2582:
--
Attachment: hdfs-2582-v2.txt
v2 addresses GetConf
Scope dfs.ha.namenodes config by nameservice
Key: HDFS-2582
URL: https://issues.apache.org/jira/browse/HDFS-2582
Project: Hadoop HDFS
Issue Type: Sub-task
Components: data-node, name-node
Affects Versions: HA branch (HDFS-1623)
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Fix For: HA branch (HDFS-1623)
Attachments: hdfs-2582-v2.txt, hdfs-2582.txt
HDFS-2231 started the process of adding configuration for HA, but one piece
is missing. The current state of the configuration is, I believe:
{{dfs.ha.namenodes}} - a list of identifiers for HA namenodes
{{dfs.federation.nameservices}} - a list of federated nameservices
{{dfs.namenode.rpc-address[.nameservice-id][.namenode-id]}} - some specific
config for the given namenode. If HA or federation is disabled, the extra
components can be elided for backwards compatibility.
The issue here is that there is no easy way to discern which NN is paired
with which other NN. Additionally, adding a new federated nameservice to a
config will require changes to {{dfs.ha.namenodes}} which makes templating
harder. It would be simpler to change {{dfs.ha.namenodes}} to be
nameservice-scoped: {{dfs.ha.namenodes.nameservice-id}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2590) Some links in WebHDFS forrest doc do not work
[ https://issues.apache.org/jira/browse/HDFS-2590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-2590: - Attachment: h2590_2023.patch h2590_2023_site.tar.gz h2590_2023_0.20s.patch h2590_2023_0.20s.patch h2590_2023_site.tar.gz h2590_2023.patch Fixed the links. The gz is the generated doc. Some links in WebHDFS forrest doc do not work - Key: HDFS-2590 URL: https://issues.apache.org/jira/browse/HDFS-2590 Project: Hadoop HDFS Issue Type: Bug Components: documentation Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Attachments: h2590_2023.patch, h2590_2023_0.20s.patch, h2590_2023_site.tar.gz Some links are pointing to DistributedFileSystem javadoc but the javadoc of DistributedFileSystem is not generated by default. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2590) Some links in WebHDFS forrest doc do not work
[ https://issues.apache.org/jira/browse/HDFS-2590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-2590: - Attachment: h2590_2023_0.20s.patch Some links in WebHDFS forrest doc do not work - Key: HDFS-2590 URL: https://issues.apache.org/jira/browse/HDFS-2590 Project: Hadoop HDFS Issue Type: Bug Components: documentation Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Attachments: h2590_2023.patch, h2590_2023.patch, h2590_2023_0.20s.patch, h2590_2023_0.20s.patch, h2590_2023_site.tar.gz Some links are pointing to DistributedFileSystem javadoc but the javadoc of DistributedFileSystem is not generated by default. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2590) Some links in WebHDFS forrest doc do not work
[ https://issues.apache.org/jira/browse/HDFS-2590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-2590: - Attachment: h2590_2023.patch Some links in WebHDFS forrest doc do not work - Key: HDFS-2590 URL: https://issues.apache.org/jira/browse/HDFS-2590 Project: Hadoop HDFS Issue Type: Bug Components: documentation Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Attachments: h2590_2023.patch, h2590_2023.patch, h2590_2023_0.20s.patch, h2590_2023_0.20s.patch, h2590_2023_site.tar.gz Some links are pointing to DistributedFileSystem javadoc but the javadoc of DistributedFileSystem is not generated by default. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2246) Shortcut a local client reads to a Datanodes files directly
[ https://issues.apache.org/jira/browse/HDFS-2246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13156484#comment-13156484 ] Hudson commented on HDFS-2246: -- Integrated in Hadoop-Common-0.23-Commit #212 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/212/]) Merged r1204792 from trunk for HDFS-2246. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1205694 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/BlockLocalPathInfo.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockMetadataHeader.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetInterface.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestConnCache.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java Shortcut a local client reads to a Datanodes files directly --- Key: HDFS-2246 URL: https://issues.apache.org/jira/browse/HDFS-2246 Project: Hadoop HDFS Issue Type: Improvement Reporter: Sanjay Radia Assignee: Jitendra Nath Pandey Fix For: 0.24.0 Attachments: 0001-HDFS-347.-Local-reads.patch, HDFS-2246-branch-0.20-security-205.1.patch, HDFS-2246-branch-0.20-security-205.2.patch, HDFS-2246-branch-0.20-security-205.patch, HDFS-2246-branch-0.20-security-205.patch, HDFS-2246-branch-0.20-security-205.patch, HDFS-2246-branch-0.20-security.3.patch, HDFS-2246-branch-0.20-security.no-softref.patch, HDFS-2246-branch-0.20-security.patch, HDFS-2246-branch-0.20-security.patch, HDFS-2246-branch-0.20-security.patch, HDFS-2246-branch-0.20-security.patch, HDFS-2246-trunk.patch, HDFS-2246-trunk.patch, HDFS-2246-trunk.patch, HDFS-2246-trunk.patch, HDFS-2246-trunk.patch, HDFS-2246-trunk.patch, HDFS-2246.20s.1.patch, HDFS-2246.20s.2.txt, HDFS-2246.20s.3.txt, HDFS-2246.20s.4.txt, HDFS-2246.20s.patch, TestShortCircuitLocalRead.java, localReadShortcut20-security.2patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2590) Some links in WebHDFS forrest doc do not work
[ https://issues.apache.org/jira/browse/HDFS-2590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-2590: - Attachment: (was: h2590_2023.patch) Some links in WebHDFS forrest doc do not work - Key: HDFS-2590 URL: https://issues.apache.org/jira/browse/HDFS-2590 Project: Hadoop HDFS Issue Type: Bug Components: documentation Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Attachments: h2590_2023.patch, h2590_2023_0.20s.patch, h2590_2023_site.tar.gz Some links are pointing to DistributedFileSystem javadoc but the javadoc of DistributedFileSystem is not generated by default. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2590) Some links in WebHDFS forrest doc do not work
[ https://issues.apache.org/jira/browse/HDFS-2590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-2590: - Attachment: h2590_2023.patch Some links in WebHDFS forrest doc do not work - Key: HDFS-2590 URL: https://issues.apache.org/jira/browse/HDFS-2590 Project: Hadoop HDFS Issue Type: Bug Components: documentation Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Attachments: h2590_2023.patch, h2590_2023_0.20s.patch, h2590_2023_site.tar.gz Some links are pointing to DistributedFileSystem javadoc but the javadoc of DistributedFileSystem is not generated by default. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2590) Some links in WebHDFS forrest doc do not work
[ https://issues.apache.org/jira/browse/HDFS-2590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-2590: - Attachment: (was: h2590_2023.patch) Some links in WebHDFS forrest doc do not work - Key: HDFS-2590 URL: https://issues.apache.org/jira/browse/HDFS-2590 Project: Hadoop HDFS Issue Type: Bug Components: documentation Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Attachments: h2590_2023.patch, h2590_2023_0.20s.patch, h2590_2023_site.tar.gz Some links are pointing to DistributedFileSystem javadoc but the javadoc of DistributedFileSystem is not generated by default. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2590) Some links in WebHDFS forrest doc do not work
[ https://issues.apache.org/jira/browse/HDFS-2590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-2590: - Attachment: (was: h2590_2023_0.20s.patch) Some links in WebHDFS forrest doc do not work - Key: HDFS-2590 URL: https://issues.apache.org/jira/browse/HDFS-2590 Project: Hadoop HDFS Issue Type: Bug Components: documentation Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Attachments: h2590_2023.patch, h2590_2023_0.20s.patch, h2590_2023_site.tar.gz Some links are pointing to DistributedFileSystem javadoc but the javadoc of DistributedFileSystem is not generated by default. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2246) Shortcut a local client reads to a Datanodes files directly
[ https://issues.apache.org/jira/browse/HDFS-2246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13156485#comment-13156485 ] Hudson commented on HDFS-2246: -- Integrated in Hadoop-Hdfs-0.23-Commit #209 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/209/]) Merged r1204792 from trunk for HDFS-2246. jitendra : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1205694 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/BlockReaderLocal.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/RemoteBlockReader2.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/BlockLocalPathInfo.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockMetadataHeader.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDataset.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FSDatasetInterface.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestConnCache.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestShortCircuitLocalRead.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java Shortcut a local client reads to a Datanodes files directly --- Key: HDFS-2246 URL: https://issues.apache.org/jira/browse/HDFS-2246 Project: Hadoop HDFS Issue Type: Improvement Reporter: Sanjay Radia Assignee: Jitendra Nath Pandey Fix For: 0.24.0 Attachments: 0001-HDFS-347.-Local-reads.patch, HDFS-2246-branch-0.20-security-205.1.patch, HDFS-2246-branch-0.20-security-205.2.patch, HDFS-2246-branch-0.20-security-205.patch, HDFS-2246-branch-0.20-security-205.patch, HDFS-2246-branch-0.20-security-205.patch, HDFS-2246-branch-0.20-security.3.patch, HDFS-2246-branch-0.20-security.no-softref.patch, HDFS-2246-branch-0.20-security.patch, HDFS-2246-branch-0.20-security.patch, HDFS-2246-branch-0.20-security.patch, HDFS-2246-branch-0.20-security.patch, HDFS-2246-trunk.patch, HDFS-2246-trunk.patch, HDFS-2246-trunk.patch, HDFS-2246-trunk.patch, HDFS-2246-trunk.patch, HDFS-2246-trunk.patch, HDFS-2246.20s.1.patch, HDFS-2246.20s.2.txt, HDFS-2246.20s.3.txt, HDFS-2246.20s.4.txt, HDFS-2246.20s.patch, TestShortCircuitLocalRead.java, localReadShortcut20-security.2patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2591) HA: MiniDFSCluster support to mix and match federation with HA
HA: MiniDFSCluster support to mix and match federation with HA -- Key: HDFS-2591 URL: https://issues.apache.org/jira/browse/HDFS-2591 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, test Affects Versions: HA branch (HDFS-1623) Reporter: Todd Lipcon Assignee: Todd Lipcon Right now the MiniDFS builder is somewhat inflexible - it just takes a numNameNodes parameter, which is used to specify federated nameservices. In order to add HA support, we need to be able to be more specific when configuring the NNs -- eg to test the case where there is one nameservice that is HA and another which is not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2592) HA: Balancer support for HA namenodes
HA: Balancer support for HA namenodes - Key: HDFS-2592 URL: https://issues.apache.org/jira/browse/HDFS-2592 Project: Hadoop HDFS Issue Type: Sub-task Components: balancer, ha Affects Versions: HA branch (HDFS-1623) Reporter: Todd Lipcon The balancer currently interacts directly with namenode InetSocketAddresses and makes its own IPC proxies. We need to integrate it with HA so that it uses the same client failover infrastructure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2582) Scope dfs.ha.namenodes config by nameservice
[
https://issues.apache.org/jira/browse/HDFS-2582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated HDFS-2582:
--
Attachment: hdfs-2582-v3.txt
v3 addresses a small NPE I discovered building HDFS-2591 on top of this patch.
Scope dfs.ha.namenodes config by nameservice
Key: HDFS-2582
URL: https://issues.apache.org/jira/browse/HDFS-2582
Project: Hadoop HDFS
Issue Type: Sub-task
Components: data-node, name-node
Affects Versions: HA branch (HDFS-1623)
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Fix For: HA branch (HDFS-1623)
Attachments: hdfs-2582-v2.txt, hdfs-2582-v3.txt, hdfs-2582.txt
HDFS-2231 started the process of adding configuration for HA, but one piece
is missing. The current state of the configuration is, I believe:
{{dfs.ha.namenodes}} - a list of identifiers for HA namenodes
{{dfs.federation.nameservices}} - a list of federated nameservices
{{dfs.namenode.rpc-address[.nameservice-id][.namenode-id]}} - some specific
config for the given namenode. If HA or federation is disabled, the extra
components can be elided for backwards compatibility.
The issue here is that there is no easy way to discern which NN is paired
with which other NN. Additionally, adding a new federated nameservice to a
config will require changes to {{dfs.ha.namenodes}} which makes templating
harder. It would be simpler to change {{dfs.ha.namenodes}} to be
nameservice-scoped: {{dfs.ha.namenodes.nameservice-id}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-2592) HA: Balancer support for HA namenodes
[ https://issues.apache.org/jira/browse/HDFS-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G reassigned HDFS-2592: - Assignee: Uma Maheswara Rao G HA: Balancer support for HA namenodes - Key: HDFS-2592 URL: https://issues.apache.org/jira/browse/HDFS-2592 Project: Hadoop HDFS Issue Type: Sub-task Components: balancer, ha Affects Versions: HA branch (HDFS-1623) Reporter: Todd Lipcon Assignee: Uma Maheswara Rao G The balancer currently interacts directly with namenode InetSocketAddresses and makes its own IPC proxies. We need to integrate it with HA so that it uses the same client failover infrastructure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2591) HA: MiniDFSCluster support to mix and match federation with HA
[ https://issues.apache.org/jira/browse/HDFS-2591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2591: -- Attachment: hdfs-2591.txt Attached patch introduces a new MiniDFSNNTopology class which is used to describe the layout of NNs in the cluster. Most unit tests are passing, including those that make federated miniclusters. (ran tests with this applied on top of HDFS-2582, though it may also work without that patch) HA: MiniDFSCluster support to mix and match federation with HA -- Key: HDFS-2591 URL: https://issues.apache.org/jira/browse/HDFS-2591 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, test Affects Versions: HA branch (HDFS-1623) Reporter: Todd Lipcon Assignee: Todd Lipcon Attachments: hdfs-2591.txt Right now the MiniDFS builder is somewhat inflexible - it just takes a numNameNodes parameter, which is used to specify federated nameservices. In order to add HA support, we need to be able to be more specific when configuring the NNs -- eg to test the case where there is one nameservice that is HA and another which is not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2591) HA: MiniDFSCluster support to mix and match federation with HA
[ https://issues.apache.org/jira/browse/HDFS-2591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13156539#comment-13156539 ] Todd Lipcon commented on HDFS-2591: --- (this is still a work in progress - testing it by updating the latest HDFS-1975 patch to run on top of this infrastructure, and identifying a few missing pieces) HA: MiniDFSCluster support to mix and match federation with HA -- Key: HDFS-2591 URL: https://issues.apache.org/jira/browse/HDFS-2591 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, test Affects Versions: HA branch (HDFS-1623) Reporter: Todd Lipcon Assignee: Todd Lipcon Attachments: hdfs-2591.txt Right now the MiniDFS builder is somewhat inflexible - it just takes a numNameNodes parameter, which is used to specify federated nameservices. In order to add HA support, we need to be able to be more specific when configuring the NNs -- eg to test the case where there is one nameservice that is HA and another which is not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1924) Block information displayed in UI is incorrect
[ https://issues.apache.org/jira/browse/HDFS-1924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-1924: -- Resolution: Duplicate Status: Resolved (was: Patch Available) After the analysis, the root cause for this issue should be HDFS-140, So, closing this as duplicate. Block information displayed in UI is incorrect --- Key: HDFS-1924 URL: https://issues.apache.org/jira/browse/HDFS-1924 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.20-append Reporter: ramkrishna.s.vasudevan Assignee: Uma Maheswara Rao G Priority: Minor Fix For: 0.20-append Attachments: HDFS-1924.patch Problem statement Deleted blocks are not removed from the blockmap. Solution === Whenever delete is called the block entry must be removed from the block map and also move it to invalidates. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-2287) TestParallelRead has a small off-by-one bug
[ https://issues.apache.org/jira/browse/HDFS-2287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko reopened HDFS-2287: --- Second assert need to be fixed. TestParallelRead has a small off-by-one bug --- Key: HDFS-2287 URL: https://issues.apache.org/jira/browse/HDFS-2287 Project: Hadoop HDFS Issue Type: Bug Components: test Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Trivial Fix For: 0.24.0 Attachments: hdfs-2287.txt Noticed this bug when I was running TestParallelRead - a simple off-by-one error in some internal bounds checking. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1975) HA: Support for sharing the namenode state from active to standby.
[
https://issues.apache.org/jira/browse/HDFS-1975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13156568#comment-13156568
]
Todd Lipcon commented on HDFS-1975:
---
I'm trying to write a test that combines this with HDFS-1971, HDFS-2591, and
HDFS-1108, and ran into an issue with the design:
With HDFS-1108, we persist blocks, but the order of the edits in the edit log
is:
{code}
2: SET_GENERATION_STAMP 1001
3: ADD /testStandbyIsHot with no blocks
4: SET_GENERATION_STAMP 1002
5: ADD /testStandbyIsHot with the block id blk_486254483591558448_1002 length=0
6: CLOSE /testStandbyIsHot with the block id blk_486254483591558448_1002
length=1000
{code}
the issue is that, when the StandbyNode is receiving the edits, it gets
SET_GENERATION_STAMP=1002 _before_ it gets the mapping of that block into the
file. So, it processes the blockReceived, sees it as an invalid block not
belonging to any file, and ignores it.
Will think about a solution
HA: Support for sharing the namenode state from active to standby.
--
Key: HDFS-1975
URL: https://issues.apache.org/jira/browse/HDFS-1975
Project: Hadoop HDFS
Issue Type: Sub-task
Components: name-node
Reporter: Suresh Srinivas
Assignee: Jitendra Nath Pandey
Attachments: HDFS-1975-HA.2.patch, HDFS-1975-HA.patch,
HDFS-1975-HDFS-1623.patch, HDFS-1975-HDFS-1623.patch, hdfs-1975.txt,
hdfs-1975.txt
To enable hot standby namenode, the standby node must have current
information for - namenode state (image + edits) and block location
information. This jira addresses keeping the namenode state current in the
standby node. To do this, the proposed solution in this jira is to use a
shared storage to store the namenode state.
Note one could also build an alternative solution by augmenting the backup
node. A seperate jira could explore this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
