[jira] [Resolved] (HDFS-361) Convert FSImage.removedStorageDirs into a map.
[
https://issues.apache.org/jira/browse/HDFS-361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Boris Shkolnik resolved HDFS-361.
-
Resolution: Won't Fix
Release Note: expired
> Convert FSImage.removedStorageDirs into a map.
> --
>
> Key: HDFS-361
> URL: https://issues.apache.org/jira/browse/HDFS-361
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Konstantin Shvachko
>Assignee: Boris Shkolnik
> Attachments: HADOOP-5619.patch
>
>
> {{FSImage.removedStorageDirs}} is declared as an {{ArrayList}}. In order to
> avoid adding the same directory twice into {{removedStorageDirs}} we should
> convert it into a map.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-1860) when renewing/canceling DelegationToken over http we need to pass exception information back to the caller.
[ https://issues.apache.org/jira/browse/HDFS-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1860: - Attachment: MR2420.20-1.patch patch for previous release (.20) when renewing/canceling DelegationToken over http we need to pass exception information back to the caller. --- Key: HDFS-1860 URL: https://issues.apache.org/jira/browse/HDFS-1860 Project: Hadoop HDFS Issue Type: Bug Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1860-2.patch, HDFS-1860-4.patch, HDFS-1860-5.patch, HDFS-1860.patch, MR2420.20-1.patch Current implementation is not using XML for that, so we will pass it as a part of response message. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1860) when renewing/canceling DelegationToken over http we need to pass exception information back to the caller.
[ https://issues.apache.org/jira/browse/HDFS-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1860: - Attachment: HDFS-1860.22-1.patch patch for earlier release (.22) when renewing/canceling DelegationToken over http we need to pass exception information back to the caller. --- Key: HDFS-1860 URL: https://issues.apache.org/jira/browse/HDFS-1860 Project: Hadoop HDFS Issue Type: Bug Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1860-2.patch, HDFS-1860-4.patch, HDFS-1860-5.patch, HDFS-1860.20-1.patch, HDFS-1860.22-1.patch, HDFS-1860.patch, MR2420.20-1.patch Current implementation is not using XML for that, so we will pass it as a part of response message. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1860) when renewing/canceling DelegationToken over http we need to pass exception information back to the caller.
[ https://issues.apache.org/jira/browse/HDFS-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1860: - Attachment: HDFS-1860.20-1.patch renamed patch for previous version (.20) when renewing/canceling DelegationToken over http we need to pass exception information back to the caller. --- Key: HDFS-1860 URL: https://issues.apache.org/jira/browse/HDFS-1860 Project: Hadoop HDFS Issue Type: Bug Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1860-2.patch, HDFS-1860-4.patch, HDFS-1860-5.patch, HDFS-1860.20-1.patch, HDFS-1860.22-1.patch, HDFS-1860.patch, MR2420.20-1.patch Current implementation is not using XML for that, so we will pass it as a part of response message. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1860) when renewing/canceling DelegationToken over http we need to pass exception information back to the caller.
[ https://issues.apache.org/jira/browse/HDFS-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1860: - Status: Open (was: Patch Available) when renewing/canceling DelegationToken over http we need to pass exception information back to the caller. --- Key: HDFS-1860 URL: https://issues.apache.org/jira/browse/HDFS-1860 Project: Hadoop HDFS Issue Type: Bug Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1860-2.patch, HDFS-1860-4.patch, HDFS-1860-5.patch, HDFS-1860.20-1.patch, HDFS-1860.22-1.patch, HDFS-1860.patch, MR2420.20-1.patch Current implementation is not using XML for that, so we will pass it as a part of response message. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1860) when renewing/canceling DelegationToken over http we need to pass exception information back to the caller.
[ https://issues.apache.org/jira/browse/HDFS-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1860: - Attachment: HDFS-1860-4.patch changed config string to DFSConfigKeys value. when renewing/canceling DelegationToken over http we need to pass exception information back to the caller. --- Key: HDFS-1860 URL: https://issues.apache.org/jira/browse/HDFS-1860 Project: Hadoop HDFS Issue Type: Bug Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1860-2.patch, HDFS-1860-4.patch, HDFS-1860.patch Current implementation is not using XML for that, so we will pass it as a part of response message. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1860) when renewing/canceling DelegationToken over http we need to pass exception information back to the caller.
[ https://issues.apache.org/jira/browse/HDFS-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1860: - Status: Patch Available (was: Open) when renewing/canceling DelegationToken over http we need to pass exception information back to the caller. --- Key: HDFS-1860 URL: https://issues.apache.org/jira/browse/HDFS-1860 Project: Hadoop HDFS Issue Type: Bug Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1860-2.patch, HDFS-1860-4.patch, HDFS-1860.patch Current implementation is not using XML for that, so we will pass it as a part of response message. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1860) when renewing/canceling DelegationToken over http we need to pass exception information back to the caller.
[ https://issues.apache.org/jira/browse/HDFS-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1860: - Attachment: HDFS-1860-5.patch fixed javac warning and findbug. Ran the failed test manually and it passed. when renewing/canceling DelegationToken over http we need to pass exception information back to the caller. --- Key: HDFS-1860 URL: https://issues.apache.org/jira/browse/HDFS-1860 Project: Hadoop HDFS Issue Type: Bug Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1860-2.patch, HDFS-1860-4.patch, HDFS-1860-5.patch, HDFS-1860.patch Current implementation is not using XML for that, so we will pass it as a part of response message. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1860) when renewing/canceling DelegationToken over http we need to pass exception information back to the caller.
[ https://issues.apache.org/jira/browse/HDFS-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1860: - Hadoop Flags: [Reviewed] Status: Patch Available (was: Open) when renewing/canceling DelegationToken over http we need to pass exception information back to the caller. --- Key: HDFS-1860 URL: https://issues.apache.org/jira/browse/HDFS-1860 Project: Hadoop HDFS Issue Type: Bug Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1860-2.patch, HDFS-1860-4.patch, HDFS-1860-5.patch, HDFS-1860.patch Current implementation is not using XML for that, so we will pass it as a part of response message. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1860) when renewing/canceling DelegationToken over http we need to pass exception information back to the caller.
[ https://issues.apache.org/jira/browse/HDFS-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1860: - Status: Open (was: Patch Available) when renewing/canceling DelegationToken over http we need to pass exception information back to the caller. --- Key: HDFS-1860 URL: https://issues.apache.org/jira/browse/HDFS-1860 Project: Hadoop HDFS Issue Type: Bug Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1860-2.patch, HDFS-1860-4.patch, HDFS-1860-5.patch, HDFS-1860.patch Current implementation is not using XML for that, so we will pass it as a part of response message. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1860) when renewing/canceling DelegationToken over http we need to pass exception information back to the caller.
[ https://issues.apache.org/jira/browse/HDFS-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13026756#comment-13026756 ] Boris Shkolnik commented on HDFS-1860: -- committed to trunk. when renewing/canceling DelegationToken over http we need to pass exception information back to the caller. --- Key: HDFS-1860 URL: https://issues.apache.org/jira/browse/HDFS-1860 Project: Hadoop HDFS Issue Type: Bug Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1860-2.patch, HDFS-1860-4.patch, HDFS-1860-5.patch, HDFS-1860.patch Current implementation is not using XML for that, so we will pass it as a part of response message. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1860) when renewing/canceling DelegationToken over http we need to pass exception information back to the caller.
[ https://issues.apache.org/jira/browse/HDFS-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1860: - Attachment: HDFS-1860-2.patch implemented review comment and also added creation of exception passed thru HTTP using Class.from() method. when renewing/canceling DelegationToken over http we need to pass exception information back to the caller. --- Key: HDFS-1860 URL: https://issues.apache.org/jira/browse/HDFS-1860 Project: Hadoop HDFS Issue Type: Bug Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1860-2.patch, HDFS-1860.patch Current implementation is not using XML for that, so we will pass it as a part of response message. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-1860) when renewing/canceling DelegationToken over http we need to pass exception information back to the caller.
[ https://issues.apache.org/jira/browse/HDFS-1860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik reassigned HDFS-1860: Assignee: Boris Shkolnik when renewing/canceling DelegationToken over http we need to pass exception information back to the caller. --- Key: HDFS-1860 URL: https://issues.apache.org/jira/browse/HDFS-1860 Project: Hadoop HDFS Issue Type: Bug Reporter: Boris Shkolnik Assignee: Boris Shkolnik Current implementation is not using XML for that, so we will pass it as a part of response message. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-1860) when renewing/canceling DelegationToken over http we need to pass exception information back to the caller.
when renewing/canceling DelegationToken over http we need to pass exception information back to the caller. --- Key: HDFS-1860 URL: https://issues.apache.org/jira/browse/HDFS-1860 Project: Hadoop HDFS Issue Type: Bug Reporter: Boris Shkolnik Current implementation is not using XML for that, so we will pass it as a part of response message. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1751) Intrinsic limits for HDFS files, directories
[ https://issues.apache.org/jira/browse/HDFS-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13022850#comment-13022850 ] Boris Shkolnik commented on HDFS-1751: -- +1. Intrinsic limits for HDFS files, directories Key: HDFS-1751 URL: https://issues.apache.org/jira/browse/HDFS-1751 Project: Hadoop HDFS Issue Type: New Feature Components: data-node Affects Versions: 0.22.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Fix For: 0.23.0 Attachments: HDFS-1751-2.patch, HDFS-1751-3.patch, HDFS-1751-4.patch, HDFS-1751-5.patch, HDFS-1751-6.patch, HDFS-1751.patch Enforce a configurable limit on: the length of a path component the number of names in a directory The intention is to prevent a too-long name or a too-full directory. This is not about RPC buffers, the length of command lines, etc. There may be good reasons for those kinds of limits, but that is not the intended scope of this feature. Consequently, a reasonable implementation might be to extend the existing quota checker so that it faults the creation of a name that violates the limits. This strategy of faulting new creation evades the problem of existing names or directories that violate the limits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1751) Intrinsic limits for HDFS files, directories
[ https://issues.apache.org/jira/browse/HDFS-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13022858#comment-13022858 ] Boris Shkolnik commented on HDFS-1751: -- Committed to trunk. Thanks Daryn. Intrinsic limits for HDFS files, directories Key: HDFS-1751 URL: https://issues.apache.org/jira/browse/HDFS-1751 Project: Hadoop HDFS Issue Type: New Feature Components: data-node Affects Versions: 0.22.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Fix For: 0.23.0 Attachments: HDFS-1751-2.patch, HDFS-1751-3.patch, HDFS-1751-4.patch, HDFS-1751-5.patch, HDFS-1751-6.patch, HDFS-1751.patch Enforce a configurable limit on: the length of a path component the number of names in a directory The intention is to prevent a too-long name or a too-full directory. This is not about RPC buffers, the length of command lines, etc. There may be good reasons for those kinds of limits, but that is not the intended scope of this feature. Consequently, a reasonable implementation might be to extend the existing quota checker so that it faults the creation of a name that violates the limits. This strategy of faulting new creation evades the problem of existing names or directories that violate the limits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1824) delay instantiation of file system object until it is needed (linked to HADOOP-7207)
[ https://issues.apache.org/jira/browse/HDFS-1824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1824: - Resolution: Fixed Status: Resolved (was: Patch Available) delay instantiation of file system object until it is needed (linked to HADOOP-7207) Key: HDFS-1824 URL: https://issues.apache.org/jira/browse/HDFS-1824 Project: Hadoop HDFS Issue Type: Bug Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1824-1-22.patch, HDFS-1824-1.patch, HDFS-1824.patch also re-factor the code little bit to avoid checking for instance of DFS in multiple places. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1824) delay instantiation of file system object until it is needed (linked to HADOOP-7207)
[ https://issues.apache.org/jira/browse/HDFS-1824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13020044#comment-13020044 ] Boris Shkolnik commented on HDFS-1824: -- Failures from the comment above are caused by the fact that this change required changes in Common which were not committed yet. I ran test-patch and tests manually and all passed. delay instantiation of file system object until it is needed (linked to HADOOP-7207) Key: HDFS-1824 URL: https://issues.apache.org/jira/browse/HDFS-1824 Project: Hadoop HDFS Issue Type: Bug Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1824-1.patch, HDFS-1824.patch also re-factor the code little bit to avoid checking for instance of DFS in multiple places. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1824) delay instantiation of file system object until it is needed (linked to HADOOP-7207)
[ https://issues.apache.org/jira/browse/HDFS-1824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13020045#comment-13020045 ] Boris Shkolnik commented on HDFS-1824: -- committed to trunk. delay instantiation of file system object until it is needed (linked to HADOOP-7207) Key: HDFS-1824 URL: https://issues.apache.org/jira/browse/HDFS-1824 Project: Hadoop HDFS Issue Type: Bug Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1824-1.patch, HDFS-1824.patch also re-factor the code little bit to avoid checking for instance of DFS in multiple places. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1824) delay instantiation of file system object until it is needed (linked to HADOOP-7207)
[ https://issues.apache.org/jira/browse/HDFS-1824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1824: - Attachment: HDFS-1824-1-22.patch not for commit. For previous version. delay instantiation of file system object until it is needed (linked to HADOOP-7207) Key: HDFS-1824 URL: https://issues.apache.org/jira/browse/HDFS-1824 Project: Hadoop HDFS Issue Type: Bug Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1824-1-22.patch, HDFS-1824-1.patch, HDFS-1824.patch also re-factor the code little bit to avoid checking for instance of DFS in multiple places. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1824) delay instantiation of file system object until it is needed (linked to HADOOP-7207)
[ https://issues.apache.org/jira/browse/HDFS-1824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1824: - Status: Patch Available (was: Open) delay instantiation of file system object until it is needed (linked to HADOOP-7207) Key: HDFS-1824 URL: https://issues.apache.org/jira/browse/HDFS-1824 Project: Hadoop HDFS Issue Type: Bug Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1824-1.patch, HDFS-1824.patch also re-factor the code little bit to avoid checking for instance of DFS in multiple places. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1824) delay instantiation of file system object until it is needed (linked to HADOOP-7207)
[ https://issues.apache.org/jira/browse/HDFS-1824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1824: - Attachment: HDFS-1824-1.patch You are right, now we throw the exception and run() catches it. And it is where the returned code is set to -1. I've added a test for return code for invalid (no DFS) file system. delay instantiation of file system object until it is needed (linked to HADOOP-7207) Key: HDFS-1824 URL: https://issues.apache.org/jira/browse/HDFS-1824 Project: Hadoop HDFS Issue Type: Bug Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1824-1.patch, HDFS-1824.patch also re-factor the code little bit to avoid checking for instance of DFS in multiple places. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-1638) HDFS Federation: DataNode.handleDiskError needs to inform ALL namenodes if a disk failed
[ https://issues.apache.org/jira/browse/HDFS-1638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik resolved HDFS-1638. -- Resolution: Fixed Hadoop Flags: [Reviewed] HDFS Federation: DataNode.handleDiskError needs to inform ALL namenodes if a disk failed - Key: HDFS-1638 URL: https://issues.apache.org/jira/browse/HDFS-1638 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1638-1.patch, HDFS-1638-2.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-1641) HDFS Federation: Datanode fields that are no longer used should be removed
[ https://issues.apache.org/jira/browse/HDFS-1641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik resolved HDFS-1641. -- Resolution: Fixed HDFS Federation: Datanode fields that are no longer used should be removed -- Key: HDFS-1641 URL: https://issues.apache.org/jira/browse/HDFS-1641 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1641-1.patch, HDFS-1641-2.patch, HDFS-1641.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-1642) HDFS Federation: add Datanode.getDNRegistration(String bpid) method
[ https://issues.apache.org/jira/browse/HDFS-1642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik resolved HDFS-1642. -- Resolution: Fixed HDFS Federation: add Datanode.getDNRegistration(String bpid) method Key: HDFS-1642 URL: https://issues.apache.org/jira/browse/HDFS-1642 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1642-1.patch, HDFS-1642.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-1660) HDFS Federation: Datanode doesn't start with two namenodes
[ https://issues.apache.org/jira/browse/HDFS-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik resolved HDFS-1660. -- Resolution: Fixed HDFS Federation: Datanode doesn't start with two namenodes -- Key: HDFS-1660 URL: https://issues.apache.org/jira/browse/HDFS-1660 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1660.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-1643) HDFS Federation: remove namenode argument from DataNode constructor
[ https://issues.apache.org/jira/browse/HDFS-1643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik resolved HDFS-1643. -- Resolution: Fixed HDFS Federation: remove namenode argument from DataNode constructor --- Key: HDFS-1643 URL: https://issues.apache.org/jira/browse/HDFS-1643 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1643-1.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-1696) HDFS Federation: when build version doesn't match - datanode should wait (keep connecting) untill NN comes up with the right version
[ https://issues.apache.org/jira/browse/HDFS-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik resolved HDFS-1696. -- Resolution: Fixed HDFS Federation: when build version doesn't match - datanode should wait (keep connecting) untill NN comes up with the right version - Key: HDFS-1696 URL: https://issues.apache.org/jira/browse/HDFS-1696 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1696-1.patch, HDFS-1696.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-1671) HDFS Federation: shutdown in DataNode should be able to shutdown individual BP threads as well as the whole DN
[ https://issues.apache.org/jira/browse/HDFS-1671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik resolved HDFS-1671. -- Resolution: Fixed HDFS Federation: shutdown in DataNode should be able to shutdown individual BP threads as well as the whole DN -- Key: HDFS-1671 URL: https://issues.apache.org/jira/browse/HDFS-1671 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1671-1.patch, HDFS-1671.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-1672) HDFS Federation: refactor stopDatanode(name) to work with multiple Block Pools
[ https://issues.apache.org/jira/browse/HDFS-1672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik resolved HDFS-1672. -- Resolution: Fixed HDFS Federation: refactor stopDatanode(name) to work with multiple Block Pools -- Key: HDFS-1672 URL: https://issues.apache.org/jira/browse/HDFS-1672 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1672-1.patch, HDFS-1672-2.patch, HDFS-1672.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-1697) HDFS federation: fix testBlockRecovery
[ https://issues.apache.org/jira/browse/HDFS-1697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik resolved HDFS-1697. -- Resolution: Fixed HDFS federation: fix testBlockRecovery --- Key: HDFS-1697 URL: https://issues.apache.org/jira/browse/HDFS-1697 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1697.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-1711) HDFS Federation: create method for updating machine name in DataNode.java
[ https://issues.apache.org/jira/browse/HDFS-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik resolved HDFS-1711. -- Resolution: Fixed HDFS Federation: create method for updating machine name in DataNode.java - Key: HDFS-1711 URL: https://issues.apache.org/jira/browse/HDFS-1711 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1711-1.patch, HDFS-1711.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-1712) HDFS Federation: when looking up datanode we should use machineNmae (in testOverReplicatedBlocks)
[ https://issues.apache.org/jira/browse/HDFS-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik resolved HDFS-1712. -- Resolution: Fixed HDFS Federation: when looking up datanode we should use machineNmae (in testOverReplicatedBlocks) -- Key: HDFS-1712 URL: https://issues.apache.org/jira/browse/HDFS-1712 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1712.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-1714) HDFS Federation: refactor upgrade object in DataNode
[ https://issues.apache.org/jira/browse/HDFS-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik resolved HDFS-1714. -- Resolution: Fixed HDFS Federation: refactor upgrade object in DataNode Key: HDFS-1714 URL: https://issues.apache.org/jira/browse/HDFS-1714 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1714-1.patch, HDFS-1714.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-1715) HDFS Federation: warning/error not generated when datanode sees inconsistent/different Cluster ID between namenodes
[ https://issues.apache.org/jira/browse/HDFS-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik resolved HDFS-1715. -- Resolution: Fixed HDFS Federation: warning/error not generated when datanode sees inconsistent/different Cluster ID between namenodes Key: HDFS-1715 URL: https://issues.apache.org/jira/browse/HDFS-1715 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1715-1.patch, HDFS-1715.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-1824) delay instantiation of file system object until it is needed (linked to HADOOP-7207)
delay instantiation of file system object until it is needed (linked to HADOOP-7207) Key: HDFS-1824 URL: https://issues.apache.org/jira/browse/HDFS-1824 Project: Hadoop HDFS Issue Type: Bug Reporter: Boris Shkolnik Assignee: Boris Shkolnik also re-factor the code little bit to avoid checking for instance of DFS in multiple places. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1824) delay instantiation of file system object until it is needed (linked to HADOOP-7207)
[ https://issues.apache.org/jira/browse/HDFS-1824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1824: - Attachment: HDFS-1824.patch delay instantiation of file system object until it is needed (linked to HADOOP-7207) Key: HDFS-1824 URL: https://issues.apache.org/jira/browse/HDFS-1824 Project: Hadoop HDFS Issue Type: Bug Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1824.patch also re-factor the code little bit to avoid checking for instance of DFS in multiple places. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1692) In secure mode, Datanode process doesn't exit when disks fail.
[ https://issues.apache.org/jira/browse/HDFS-1692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13015671#comment-13015671 ] Boris Shkolnik commented on HDFS-1692: -- committed to branch-0.20-security. Thanks Bharath. In secure mode, Datanode process doesn't exit when disks fail. -- Key: HDFS-1692 URL: https://issues.apache.org/jira/browse/HDFS-1692 Project: Hadoop HDFS Issue Type: Bug Components: data-node Affects Versions: 0.20.204.0 Reporter: Bharath Mundlapudi Assignee: Bharath Mundlapudi Fix For: 0.20.204.0 Attachments: HDFS-1692-1.patch In secure mode, when disks fail more than volumes tolerated, datanode process doesn't exit properly and it just hangs even though shutdown method is called. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1692) In secure mode, Datanode process doesn't exit when disks fail.
[ https://issues.apache.org/jira/browse/HDFS-1692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13014844#comment-13014844 ] Boris Shkolnik commented on HDFS-1692: -- +1 In secure mode, Datanode process doesn't exit when disks fail. -- Key: HDFS-1692 URL: https://issues.apache.org/jira/browse/HDFS-1692 Project: Hadoop HDFS Issue Type: Bug Components: data-node Affects Versions: 0.20.204.0 Reporter: Bharath Mundlapudi Assignee: Bharath Mundlapudi Fix For: 0.20.204.0 Attachments: HDFS-1692-1.patch In secure mode, when disks fail more than volumes tolerated, datanode process doesn't exit properly and it just hangs even though shutdown method is called. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1172) Blocks in newly completed files are considered under-replicated too quickly
[ https://issues.apache.org/jira/browse/HDFS-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1172: - Attachment: HDFS-1172.patch Does this patch looks like what has been discussed here? It puts underreplicated blocks into pending replication queue in case of newly created file. Blocks in newly completed files are considered under-replicated too quickly --- Key: HDFS-1172 URL: https://issues.apache.org/jira/browse/HDFS-1172 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.21.0 Reporter: Todd Lipcon Attachments: HDFS-1172.patch I've seen this for a long time, and imagine it's a known issue, but couldn't find an existing JIRA. It often happens that we see the NN schedule replication on the last block of files very quickly after they're completed, before the other DNs in the pipeline have a chance to report the new block. This results in a lot of extra replication work on the cluster, as we replicate the block and then end up with multiple excess replicas which are very quickly deleted. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1172) Blocks in newly completed files are considered under-replicated too quickly
[ https://issues.apache.org/jira/browse/HDFS-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13008703#comment-13008703 ] Boris Shkolnik commented on HDFS-1172: -- I agree that 5 minutes is too long, but putting it into pendingReplication still seems to be more appropriate. May be we can modify pendingReplication monitor to adjust check interval dynamically to the next 'timing out' replication. This would, of course, require having timeOut value per replication (or we can reuse timeStamp for that). Blocks in newly completed files are considered under-replicated too quickly --- Key: HDFS-1172 URL: https://issues.apache.org/jira/browse/HDFS-1172 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.21.0 Reporter: Todd Lipcon Attachments: HDFS-1172.patch I've seen this for a long time, and imagine it's a known issue, but couldn't find an existing JIRA. It often happens that we see the NN schedule replication on the last block of files very quickly after they're completed, before the other DNs in the pipeline have a chance to report the new block. This results in a lot of extra replication work on the cluster, as we replicate the block and then end up with multiple excess replicas which are very quickly deleted. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1751) Intrinsic limits for HDFS files, directories
[ https://issues.apache.org/jira/browse/HDFS-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13008173#comment-13008173 ] Boris Shkolnik commented on HDFS-1751: -- Using two different mechanisms makes sense to me. It is like 'ulimit' and quota on a unix system. They solve different problems. In regards to the quote mentioned by Jakob, it looks like this comment is related to the 'reusing' of the boolean flag only. Intrinsic limits for HDFS files, directories Key: HDFS-1751 URL: https://issues.apache.org/jira/browse/HDFS-1751 Project: Hadoop HDFS Issue Type: New Feature Components: data-node Affects Versions: 0.22.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Fix For: 0.23.0 Attachments: HDFS-1751-2.patch, HDFS-1751-3.patch, HDFS-1751-4.patch, HDFS-1751.patch Enforce a configurable limit on: the length of a path component the number of names in a directory The intention is to prevent a too-long name or a too-full directory. This is not about RPC buffers, the length of command lines, etc. There may be good reasons for those kinds of limits, but that is not the intended scope of this feature. Consequently, a reasonable implementation might be to extend the existing quota checker so that it faults the creation of a name that violates the limits. This strategy of faulting new creation evades the problem of existing names or directories that violate the limits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1751) Intrinsic limits for HDFS files, directories
[ https://issues.apache.org/jira/browse/HDFS-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13007518#comment-13007518 ] Boris Shkolnik commented on HDFS-1751: -- This seems to be a bit different kind of limitation. It is a cluster wide setting and it is separate from quota. Quotas are per specific entity - user, directory and so on. If the operator of a cluster needs to impose such a cluster wide limitation he/she may want to use this new configuration. And that's why it needs to be done thru config. So we shouldn't view this setting as a quota at all. I agree with the other Jacob's comment - regarding the name of the config setting. If you guys think using specific types of exceptions is more preferable - sure, go ahead. Intrinsic limits for HDFS files, directories Key: HDFS-1751 URL: https://issues.apache.org/jira/browse/HDFS-1751 Project: Hadoop HDFS Issue Type: New Feature Components: data-node Affects Versions: 0.22.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Fix For: 0.23.0 Attachments: HDFS-1751-2.patch, HDFS-1751.patch Enforce a configurable limit on: the length of a path component the number of names in a directory The intention is to prevent a too-long name or a too-full directory. This is not about RPC buffers, the length of command lines, etc. There may be good reasons for those kinds of limits, but that is not the intended scope of this feature. Consequently, a reasonable implementation might be to extend the existing quota checker so that it faults the creation of a name that violates the limits. This strategy of faulting new creation evades the problem of existing names or directories that violate the limits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1751) Intrinsic limits for HDFS files, directories
[ https://issues.apache.org/jira/browse/HDFS-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13006976#comment-13006976 ] Boris Shkolnik commented on HDFS-1751: -- Non code questions: 1. why do we need limit on minimum component length. Some code notes: 1. Please use @Override when overriding methods. 2. verifyFSLimits should throw FSLimitException (most specific exception). Also update the java doc for this method. 3. Do we really need so specific exceptions, can't we just use FSLimitException with different messages. 4. Please use junit 4 for test. Intrinsic limits for HDFS files, directories Key: HDFS-1751 URL: https://issues.apache.org/jira/browse/HDFS-1751 Project: Hadoop HDFS Issue Type: New Feature Components: data-node Affects Versions: 0.22.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Fix For: 0.23.0 Attachments: HDFS-1751.patch Enforce a configurable limit on: the length of a path component the number of names in a directory The intention is to prevent a too-long name or a too-full directory. This is not about RPC buffers, the length of command lines, etc. There may be good reasons for those kinds of limits, but that is not the intended scope of this feature. Consequently, a reasonable implementation might be to extend the existing quota checker so that it faults the creation of a name that violates the limits. This strategy of faulting new creation evades the problem of existing names or directories that violate the limits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1755) Hdfs Federation: The BPOfferService must always connect to namenode as the login user.
[ https://issues.apache.org/jira/browse/HDFS-1755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13007238#comment-13007238 ] Boris Shkolnik commented on HDFS-1755: -- Yeah, I like this solution better (using loginUser.doAs() always) This way everything is created same way. +1 Hdfs Federation: The BPOfferService must always connect to namenode as the login user. -- Key: HDFS-1755 URL: https://issues.apache.org/jira/browse/HDFS-1755 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Affects Versions: Federation Branch Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Fix For: Federation Branch Attachments: HDFS-1755.2.patch, HDFS-1755.3.patch The BPOfferService thread when started, normally connects to the namenode as the login user. But in case of refreshNamenodes a new BPOfferService may be started in the context of an rpc call where the user is dfsadmin. In such a case the connection to the namenode will fail. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1751) Intrinsic limits for HDFS files, directories
[ https://issues.apache.org/jira/browse/HDFS-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13007266#comment-13007266 ] Boris Shkolnik commented on HDFS-1751: -- Ok, so my understanding is that these changes allow to impose some cluster wide limitations on the length of the names and number of items in a directory. Unlike quota system - which is per directory. I still cannot imagine a case where we need a minimum name length check. Couple more comments: 1. Could you see if you can use mock() for FSNamesystem instead of changing the constructor for FSDirectory. 2. in your tests seems like testMinComponentsAndMaxDirContents() test covers all the cases. Do we really need the others (testNoLimits, testMinComponentLength, testMaxComponentLength and testMaxDirContents). Intrinsic limits for HDFS files, directories Key: HDFS-1751 URL: https://issues.apache.org/jira/browse/HDFS-1751 Project: Hadoop HDFS Issue Type: New Feature Components: data-node Affects Versions: 0.22.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Fix For: 0.23.0 Attachments: HDFS-1751-2.patch, HDFS-1751.patch Enforce a configurable limit on: the length of a path component the number of names in a directory The intention is to prevent a too-long name or a too-full directory. This is not about RPC buffers, the length of command lines, etc. There may be good reasons for those kinds of limits, but that is not the intended scope of this feature. Consequently, a reasonable implementation might be to extend the existing quota checker so that it faults the creation of a name that violates the limits. This strategy of faulting new creation evades the problem of existing names or directories that violate the limits. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Created: (HDFS-1754) testFsck fails
testFsck fails -- Key: HDFS-1754 URL: https://issues.apache.org/jira/browse/HDFS-1754 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Boris Shkolnik Assignee: Boris Shkolnik -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1754) Federation HDFS: :testFsck fails
[ https://issues.apache.org/jira/browse/HDFS-1754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1754: - Description: getting config value for HTTP server is changed to read new (federation)style configuration, but it is missing backward compatibility case. Summary: Federation HDFS: :testFsck fails (was: testFsck fails) Federation HDFS: :testFsck fails Key: HDFS-1754 URL: https://issues.apache.org/jira/browse/HDFS-1754 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1754.patch getting config value for HTTP server is changed to read new (federation)style configuration, but it is missing backward compatibility case. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1754) Federation HDFS: :testFsck fails
[ https://issues.apache.org/jira/browse/HDFS-1754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1754: - Attachment: HDFS-1754.patch Federation HDFS: :testFsck fails Key: HDFS-1754 URL: https://issues.apache.org/jira/browse/HDFS-1754 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1754.patch getting config value for HTTP server is changed to read new (federation)style configuration, but it is missing backward compatibility case. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1754) Federation HDFS: :testFsck fails
[ https://issues.apache.org/jira/browse/HDFS-1754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1754: - Attachment: HDFS-1754-1.patch clean up and review comments Federation HDFS: :testFsck fails Key: HDFS-1754 URL: https://issues.apache.org/jira/browse/HDFS-1754 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1754-1.patch, HDFS-1754.patch getting config value for HTTP server is changed to read new (federation)style configuration, but it is missing backward compatibility case. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1754) Federation HDFS: :testFsck fails
[ https://issues.apache.org/jira/browse/HDFS-1754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1754: - Attachment: HDFS-1754-2.patch added @param Federation HDFS: :testFsck fails Key: HDFS-1754 URL: https://issues.apache.org/jira/browse/HDFS-1754 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1754-1.patch, HDFS-1754-2.patch, HDFS-1754.patch getting config value for HTTP server is changed to read new (federation)style configuration, but it is missing backward compatibility case. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1754) Federation HDFS: testFsck fails
[ https://issues.apache.org/jira/browse/HDFS-1754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1754: - Hadoop Flags: [Reviewed] Summary: Federation HDFS: testFsck fails (was: Federation HDFS: :testFsck fails) committed to Federation branch Federation HDFS: testFsck fails --- Key: HDFS-1754 URL: https://issues.apache.org/jira/browse/HDFS-1754 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1754-1.patch, HDFS-1754-2.patch, HDFS-1754.patch getting config value for HTTP server is changed to read new (federation)style configuration, but it is missing backward compatibility case. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (HDFS-1754) Federation HDFS: testFsck fails
[ https://issues.apache.org/jira/browse/HDFS-1754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik resolved HDFS-1754. -- Resolution: Fixed Federation HDFS: testFsck fails --- Key: HDFS-1754 URL: https://issues.apache.org/jira/browse/HDFS-1754 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1754-1.patch, HDFS-1754-2.patch, HDFS-1754.patch getting config value for HTTP server is changed to read new (federation)style configuration, but it is missing backward compatibility case. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1744) Federation: Add new layout version to offline image/edits viewer
[ https://issues.apache.org/jira/browse/HDFS-1744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005221#comment-13005221 ] Boris Shkolnik commented on HDFS-1744: -- +1 Federation: Add new layout version to offline image/edits viewer Key: HDFS-1744 URL: https://issues.apache.org/jira/browse/HDFS-1744 Project: Hadoop HDFS Issue Type: Sub-task Components: test Affects Versions: Federation Branch Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: Federation Branch Attachments: HDFS-1744.patch Federation introduces new layout version -28, that needs to be added to offline image and edits viewer. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1745) Federation: fix fault injection test build failure
[ https://issues.apache.org/jira/browse/HDFS-1745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005291#comment-13005291 ] Boris Shkolnik commented on HDFS-1745: -- +1 Federation: fix fault injection test build failure -- Key: HDFS-1745 URL: https://issues.apache.org/jira/browse/HDFS-1745 Project: Hadoop HDFS Issue Type: Sub-task Components: data-node, test Affects Versions: Federation Branch Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: Federation Branch Attachments: HDFS-1745.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1738) change hdfs jmxget to return an empty string instead of null when an attribute value is not available.
[ https://issues.apache.org/jira/browse/HDFS-1738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005367#comment-13005367 ] Boris Shkolnik commented on HDFS-1738: -- committed to trunk. Thanks Tanping. change hdfs jmxget to return an empty string instead of null when an attribute value is not available. -- Key: HDFS-1738 URL: https://issues.apache.org/jira/browse/HDFS-1738 Project: Hadoop HDFS Issue Type: Improvement Components: tools Reporter: Tanping Wang Assignee: Tanping Wang Priority: Minor Attachments: HDFS-1738.patch Currently the tool, hdfs jmx returns null in case of an attribute value is not available. A null pointer exception is thrown and the value of the rest attributes are not printed. It makes more sense to return an empty string and continue to print out the values for the rest attributes. Example of current behavior $ hdfs jmxget -server gsbl90629.blue.ygrid.yahoo.com -port 8004 -service NameNode,name=NameNodeActivity jmx name: name=NameNodeActivity,service=NameNode tag.ProcessName=NameNode java.lang.NullPointerException at org.apache.hadoop.hdfs.tools.JMXGet.printAllValues(JMXGet.java:106) at org.apache.hadoop.hdfs.tools.JMXGet.main(JMXGet.java:329) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Created: (HDFS-1747) HDFS Federation: TestBackupNode Fails on Federation branch
HDFS Federation: TestBackupNode Fails on Federation branch -- Key: HDFS-1747 URL: https://issues.apache.org/jira/browse/HDFS-1747 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Boris Shkolnik Assignee: Boris Shkolnik -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1747) HDFS Federation: TestBackupNode Fails on Federation branch
[ https://issues.apache.org/jira/browse/HDFS-1747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005386#comment-13005386 ] Boris Shkolnik commented on HDFS-1747: -- Test gets into infinite waiting loop waiting for checkpoint to complete, while checkpoint fails because of a mismatch of the signature. HDFS Federation: TestBackupNode Fails on Federation branch -- Key: HDFS-1747 URL: https://issues.apache.org/jira/browse/HDFS-1747 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1747) HDFS Federation: TestBackupNode Fails on Federation branch
[ https://issues.apache.org/jira/browse/HDFS-1747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1747: - Attachment: HDFS-1747.patch Seems like the problem is missing BlockPoolId in the signature. HDFS Federation: TestBackupNode Fails on Federation branch -- Key: HDFS-1747 URL: https://issues.apache.org/jira/browse/HDFS-1747 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1747.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (HDFS-1747) HDFS Federation: TestBackupNode Fails on Federation branch
[ https://issues.apache.org/jira/browse/HDFS-1747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik resolved HDFS-1747. -- Resolution: Fixed HDFS Federation: TestBackupNode Fails on Federation branch -- Key: HDFS-1747 URL: https://issues.apache.org/jira/browse/HDFS-1747 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1747.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1747) HDFS Federation: TestBackupNode Fails on Federation branch
[ https://issues.apache.org/jira/browse/HDFS-1747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13005397#comment-13005397 ] Boris Shkolnik commented on HDFS-1747: -- committed to Federation branch. HDFS Federation: TestBackupNode Fails on Federation branch -- Key: HDFS-1747 URL: https://issues.apache.org/jira/browse/HDFS-1747 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1747.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1696) HDFS Federation: when build version doesn't match - datanode should wait (keep connecting) untill NN comes up with the right version
[ https://issues.apache.org/jira/browse/HDFS-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13004245#comment-13004245 ] Boris Shkolnik commented on HDFS-1696: -- I guess it should work. You should create a separate JIRA for it. So others can comment on it. HDFS Federation: when build version doesn't match - datanode should wait (keep connecting) untill NN comes up with the right version - Key: HDFS-1696 URL: https://issues.apache.org/jira/browse/HDFS-1696 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1696-1.patch, HDFS-1696.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1737) Update the layout version for federation changes
[ https://issues.apache.org/jira/browse/HDFS-1737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13004306#comment-13004306 ] Boris Shkolnik commented on HDFS-1737: -- +1 Update the layout version for federation changes Key: HDFS-1737 URL: https://issues.apache.org/jira/browse/HDFS-1737 Project: Hadoop HDFS Issue Type: Sub-task Components: data-node, name-node Affects Versions: Federation Branch Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: Federation Branch Attachments: HDFS-1737.patch Layout version needs to be bumped up for federation changes. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1738) change hdfs jmxget to return an empty string instead of null when an attribute value is not available.
[ https://issues.apache.org/jira/browse/HDFS-1738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13004325#comment-13004325 ] Boris Shkolnik commented on HDFS-1738: -- +1 change hdfs jmxget to return an empty string instead of null when an attribute value is not available. -- Key: HDFS-1738 URL: https://issues.apache.org/jira/browse/HDFS-1738 Project: Hadoop HDFS Issue Type: Improvement Components: tools Reporter: Tanping Wang Assignee: Tanping Wang Priority: Minor Attachments: HDFS-1738.patch Currently the tool, hdfs jmx returns null in case of an attribute value is not available. A null pointer exception is thrown and the value of the rest attributes are not printed. It makes more sense to return an empty string and continue to print out the values for the rest attributes. Example of current behavior $ hdfs jmxget -server gsbl90629.blue.ygrid.yahoo.com -port 8004 -service NameNode,name=NameNodeActivity jmx name: name=NameNodeActivity,service=NameNode tag.ProcessName=NameNode java.lang.NullPointerException at org.apache.hadoop.hdfs.tools.JMXGet.printAllValues(JMXGet.java:106) at org.apache.hadoop.hdfs.tools.JMXGet.main(JMXGet.java:329) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1719) Federation: TestDFSRemove fails intermittently
[ https://issues.apache.org/jira/browse/HDFS-1719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13002747#comment-13002747 ] Boris Shkolnik commented on HDFS-1719: -- +1 Federation: TestDFSRemove fails intermittently --- Key: HDFS-1719 URL: https://issues.apache.org/jira/browse/HDFS-1719 Project: Hadoop HDFS Issue Type: Sub-task Components: data-node Affects Versions: Federation Branch Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: Federation Branch Attachments: HDFS-1719.1.patch, HDFS-1719.patch This failure is due to Datanode#datanodeId being seen as null by another thread. Datanode#datanodeId access is not synchronized. It has many fields, datanode name, info port, ipc port, storageID etc. These fields are some times set and some times read without being synchronized. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (HDFS-1708) Federation: remove datanode's datanodeId TODOs
[ https://issues.apache.org/jira/browse/HDFS-1708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik resolved HDFS-1708. -- Resolution: Invalid this issue is invalid, because most of the stuff is fixed somewhere else. Federation: remove datanode's datanodeId TODOs --- Key: HDFS-1708 URL: https://issues.apache.org/jira/browse/HDFS-1708 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1711) create method for updating machine name in DataNode.java
[ https://issues.apache.org/jira/browse/HDFS-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1711: - Attachment: HDFS-1711.patch create method for updating machine name in DataNode.java Key: HDFS-1711 URL: https://issues.apache.org/jira/browse/HDFS-1711 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1711.patch -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1711) create method for updating machine name in DataNode.java
[ https://issues.apache.org/jira/browse/HDFS-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1711: - Attachment: HDFS-1711-1.patch merged with branch create method for updating machine name in DataNode.java Key: HDFS-1711 URL: https://issues.apache.org/jira/browse/HDFS-1711 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1711-1.patch, HDFS-1711.patch -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Created: (HDFS-1712) when looking up datanode we should use machineNmae (in testOverReplicatedBlocks)
when looking up datanode we should use machineNmae (in testOverReplicatedBlocks) - Key: HDFS-1712 URL: https://issues.apache.org/jira/browse/HDFS-1712 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Boris Shkolnik Assignee: Boris Shkolnik -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1712) when looking up datanode we should use machineNmae (in testOverReplicatedBlocks)
[ https://issues.apache.org/jira/browse/HDFS-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1712: - Attachment: HDFS-1712.patch currently , in this test, we compare DataNodeInfo, but we should compare datanodoe machine name when looking up datanode we should use machineNmae (in testOverReplicatedBlocks) - Key: HDFS-1712 URL: https://issues.apache.org/jira/browse/HDFS-1712 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1712.patch -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1711) HDFS Federation: create method for updating machine name in DataNode.java
[ https://issues.apache.org/jira/browse/HDFS-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13002158#comment-13002158 ] Boris Shkolnik commented on HDFS-1711: -- committed to Federation branch HDFS Federation: create method for updating machine name in DataNode.java - Key: HDFS-1711 URL: https://issues.apache.org/jira/browse/HDFS-1711 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1711-1.patch, HDFS-1711.patch -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1711) HDFS Federation: create method for updating machine name in DataNode.java
[ https://issues.apache.org/jira/browse/HDFS-1711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1711: - Summary: HDFS Federation: create method for updating machine name in DataNode.java (was: create method for updating machine name in DataNode.java) HDFS Federation: create method for updating machine name in DataNode.java - Key: HDFS-1711 URL: https://issues.apache.org/jira/browse/HDFS-1711 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1711-1.patch, HDFS-1711.patch -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1712) HDFS Federation: when looking up datanode we should use machineNmae (in testOverReplicatedBlocks)
[ https://issues.apache.org/jira/browse/HDFS-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1712: - Hadoop Flags: [Reviewed] Summary: HDFS Federation: when looking up datanode we should use machineNmae (in testOverReplicatedBlocks) (was: when looking up datanode we should use machineNmae (in testOverReplicatedBlocks) ) committed to Federatin Branch. HDFS Federation: when looking up datanode we should use machineNmae (in testOverReplicatedBlocks) -- Key: HDFS-1712 URL: https://issues.apache.org/jira/browse/HDFS-1712 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1712.patch -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1697) HDFS federation: fix testBlockRecovery
[ https://issues.apache.org/jira/browse/HDFS-1697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13002172#comment-13002172 ] Boris Shkolnik commented on HDFS-1697: -- committed to Federation branch HDFS federation: fix testBlockRecovery --- Key: HDFS-1697 URL: https://issues.apache.org/jira/browse/HDFS-1697 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1697.patch -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1697) HDFS federation: fix testBlockRecovery
[ https://issues.apache.org/jira/browse/HDFS-1697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13002171#comment-13002171 ] Boris Shkolnik commented on HDFS-1697: -- Datanode doesn't have namenode field anymore, so we need to add ability to update namenode per block pool. HDFS federation: fix testBlockRecovery --- Key: HDFS-1697 URL: https://issues.apache.org/jira/browse/HDFS-1697 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1697.patch -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (HDFS-1634) HDFS Federation: Convert single threaded DataNode into per BlockPool thread model.
[ https://issues.apache.org/jira/browse/HDFS-1634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik resolved HDFS-1634. -- Resolution: Fixed Hadoop Flags: [Reviewed] HDFS Federation: Convert single threaded DataNode into per BlockPool thread model. -- Key: HDFS-1634 URL: https://issues.apache.org/jira/browse/HDFS-1634 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1634-1.patch, HDFS-1634-2.patch -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Created: (HDFS-1714) HDFS Federation: refactor upgrade object in DataNode
HDFS Federation: refactor upgrade object in DataNode Key: HDFS-1714 URL: https://issues.apache.org/jira/browse/HDFS-1714 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Boris Shkolnik Assignee: Boris Shkolnik -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1714) HDFS Federation: refactor upgrade object in DataNode
[ https://issues.apache.org/jira/browse/HDFS-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1714: - Attachment: HDFS-1714.patch Currently there is single Upgrade object in the DataNode. And it makes use of assumption that there is only one NameNode by accessing datanode.namenode member. We remove this memeber and make UpgradeManager member of BPOffService. One instance will be created per BPOferService at start. HDFS Federation: refactor upgrade object in DataNode Key: HDFS-1714 URL: https://issues.apache.org/jira/browse/HDFS-1714 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1714.patch -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Created: (HDFS-1715) HDFS Federation: warning/error not generated when datanode sees inconsistent/different Cluster ID between namenodes
HDFS Federation: warning/error not generated when datanode sees inconsistent/different Cluster ID between namenodes Key: HDFS-1715 URL: https://issues.apache.org/jira/browse/HDFS-1715 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Boris Shkolnik Assignee: Boris Shkolnik -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1715) HDFS Federation: warning/error not generated when datanode sees inconsistent/different Cluster ID between namenodes
[ https://issues.apache.org/jira/browse/HDFS-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1715: - Attachment: HDFS-1715.patch HDFS Federation: warning/error not generated when datanode sees inconsistent/different Cluster ID between namenodes Key: HDFS-1715 URL: https://issues.apache.org/jira/browse/HDFS-1715 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1715.patch -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1714) HDFS Federation: refactor upgrade object in DataNode
[ https://issues.apache.org/jira/browse/HDFS-1714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13002191#comment-13002191 ] Boris Shkolnik commented on HDFS-1714: -- committed to Federation branch. HDFS Federation: refactor upgrade object in DataNode Key: HDFS-1714 URL: https://issues.apache.org/jira/browse/HDFS-1714 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1714-1.patch, HDFS-1714.patch -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1718) HDFS Federation: MiniDFSCluster#waitActive() bug causes some tests to fail
[ https://issues.apache.org/jira/browse/HDFS-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13002263#comment-13002263 ] Boris Shkolnik commented on HDFS-1718: -- +1 HDFS Federation: MiniDFSCluster#waitActive() bug causes some tests to fail --- Key: HDFS-1718 URL: https://issues.apache.org/jira/browse/HDFS-1718 Project: Hadoop HDFS Issue Type: Sub-task Components: test Affects Versions: Federation Branch Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: Federation Branch Attachments: HDFS-1718.patch MiniDFSCluster#shouldWait() method waits for all the datanodes to come up and register with the namenode. Due to threading issues some of the tests fail for two reasons: # Datanode#isDatanodeUp() fails even if all the BPOfferService threads have exited. This is due to Thread.isAlive() returning true, even though the thread has exited. Adding a check to BPOfferService#shouldService run as an addition, fixes this issues. # shouldWait(), where isBPServiceAlive() is called, does not work when a BPOfferService thread fails before the datanode has discovered the BPID, from handshake with namenode. This can be fixed by checking the thread state using InetSocketAddress to determine the BPOfferService, instead of BPID. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1715) HDFS Federation: warning/error not generated when datanode sees inconsistent/different Cluster ID between namenodes
[ https://issues.apache.org/jira/browse/HDFS-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1715: - Attachment: HDFS-1715-1.patch also added test for mismatched cluster id. HDFS Federation: warning/error not generated when datanode sees inconsistent/different Cluster ID between namenodes Key: HDFS-1715 URL: https://issues.apache.org/jira/browse/HDFS-1715 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1715-1.patch, HDFS-1715.patch -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Created: (HDFS-1722) HDFS Federation: Add flag to MiniDFSCluster to differentiate between two different modes-Federation and not.
HDFS Federation: Add flag to MiniDFSCluster to differentiate between two different modes-Federation and not. - Key: HDFS-1722 URL: https://issues.apache.org/jira/browse/HDFS-1722 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Boris Shkolnik Assignee: Boris Shkolnik -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1722) HDFS Federation: Add flag to MiniDFSCluster to differentiate between two different modes-Federation and not.
[ https://issues.apache.org/jira/browse/HDFS-1722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1722: - Attachment: HDFS-1722.patch For non-federated mode we keep old style set-up for backward compatibility. HDFS Federation: Add flag to MiniDFSCluster to differentiate between two different modes-Federation and not. - Key: HDFS-1722 URL: https://issues.apache.org/jira/browse/HDFS-1722 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1722.patch -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1722) HDFS Federation: Add flag to MiniDFSCluster to differentiate between two different modes-Federation and not.
[ https://issues.apache.org/jira/browse/HDFS-1722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1722: - Attachment: HDFS-1722.patch HDFS Federation: Add flag to MiniDFSCluster to differentiate between two different modes-Federation and not. - Key: HDFS-1722 URL: https://issues.apache.org/jira/browse/HDFS-1722 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1722.patch, HDFS-1722.patch -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1722) HDFS Federation: Add flag to MiniDFSCluster to differentiate between two different modes-Federation and not.
[ https://issues.apache.org/jira/browse/HDFS-1722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13002388#comment-13002388 ] Boris Shkolnik commented on HDFS-1722: -- without a flag MiniDFScluster will try to figure it by itself - if number of namenodes is 1, then it is a classical case, if 1, then it is Federation. Federation flag is needed when you create cluster with 1 NameNode but plan to add more later. HDFS Federation: Add flag to MiniDFSCluster to differentiate between two different modes-Federation and not. - Key: HDFS-1722 URL: https://issues.apache.org/jira/browse/HDFS-1722 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1722.patch, HDFS-1722.patch -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1696) HDFS Federation: when build version doesn't match - datanode should wait (keep connecting) untill NN comes up with the right version
[ https://issues.apache.org/jira/browse/HDFS-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1696: - Attachment: HDFS-1696.patch HDFS Federation: when build version doesn't match - datanode should wait (keep connecting) untill NN comes up with the right version - Key: HDFS-1696 URL: https://issues.apache.org/jira/browse/HDFS-1696 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1696.patch -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1696) HDFS Federation: when build version doesn't match - datanode should wait (keep connecting) untill NN comes up with the right version
[ https://issues.apache.org/jira/browse/HDFS-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13001589#comment-13001589 ] Boris Shkolnik commented on HDFS-1696: -- in datanode, instead of exiting in case of mismatching buildVersion, now we just keep waiting for NN to connect with the right version. HDFS Federation: when build version doesn't match - datanode should wait (keep connecting) untill NN comes up with the right version - Key: HDFS-1696 URL: https://issues.apache.org/jira/browse/HDFS-1696 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1696-1.patch, HDFS-1696.patch -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1696) HDFS Federation: when build version doesn't match - datanode should wait (keep connecting) untill NN comes up with the right version
[ https://issues.apache.org/jira/browse/HDFS-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1696: - Attachment: HDFS-1696-1.patch HDFS Federation: when build version doesn't match - datanode should wait (keep connecting) untill NN comes up with the right version - Key: HDFS-1696 URL: https://issues.apache.org/jira/browse/HDFS-1696 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1696-1.patch, HDFS-1696.patch -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Created: (HDFS-1697) HDFS federation: fix testBlockRecovery
HDFS federation: fix testBlockRecovery --- Key: HDFS-1697 URL: https://issues.apache.org/jira/browse/HDFS-1697 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Boris Shkolnik Assignee: Boris Shkolnik -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1697) HDFS federation: fix testBlockRecovery
[ https://issues.apache.org/jira/browse/HDFS-1697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1697: - Attachment: HDFS-1697.patch HDFS federation: fix testBlockRecovery --- Key: HDFS-1697 URL: https://issues.apache.org/jira/browse/HDFS-1697 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1697.patch -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1696) HDFS Federation: when build version doesn't match - datanode should wait (keep connecting) untill NN comes up with the right version
[ https://issues.apache.org/jira/browse/HDFS-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13001659#comment-13001659 ] Boris Shkolnik commented on HDFS-1696: -- committed to Federation branch HDFS Federation: when build version doesn't match - datanode should wait (keep connecting) untill NN comes up with the right version - Key: HDFS-1696 URL: https://issues.apache.org/jira/browse/HDFS-1696 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1696-1.patch, HDFS-1696.patch -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1694) SimulatedFSDataset changes to work with federation and multiple block pools
[ https://issues.apache.org/jira/browse/HDFS-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13001706#comment-13001706 ] Boris Shkolnik commented on HDFS-1694: -- +1 SimulatedFSDataset changes to work with federation and multiple block pools Key: HDFS-1694 URL: https://issues.apache.org/jira/browse/HDFS-1694 Project: Hadoop HDFS Issue Type: Sub-task Components: test Affects Versions: Federation Branch Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: Federation Branch Attachments: HDFS-1694.patch SimulatedFSDataset does not have the notion of multiple blocks pools. Changes equivalent FSDataset for federation is needed in SimulatedFSDataset. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1697) HDFS federation: fix testBlockRecovery
[ https://issues.apache.org/jira/browse/HDFS-1697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13001778#comment-13001778 ] Boris Shkolnik commented on HDFS-1697: -- since the test doesn't start real namenode (only a mock), datanode needs to be setup manually. HDFS federation: fix testBlockRecovery --- Key: HDFS-1697 URL: https://issues.apache.org/jira/browse/HDFS-1697 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1697.patch -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1697) HDFS federation: fix testBlockRecovery
[ https://issues.apache.org/jira/browse/HDFS-1697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13001780#comment-13001780 ] Boris Shkolnik commented on HDFS-1697: -- [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 6 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] HDFS federation: fix testBlockRecovery --- Key: HDFS-1697 URL: https://issues.apache.org/jira/browse/HDFS-1697 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1697.patch -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1708) remove datanode's datanodeId TODOs
[ https://issues.apache.org/jira/browse/HDFS-1708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13001784#comment-13001784 ] Boris Shkolnik commented on HDFS-1708: -- and fix all the TODO items in DataNode.java remove datanode's datanodeId TODOs --- Key: HDFS-1708 URL: https://issues.apache.org/jira/browse/HDFS-1708 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Created: (HDFS-1708) remove datanode's datanodeId TODOs
remove datanode's datanodeId TODOs --- Key: HDFS-1708 URL: https://issues.apache.org/jira/browse/HDFS-1708 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Boris Shkolnik Assignee: Boris Shkolnik -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1708) remove datanode's datanodeId TODOs
[ https://issues.apache.org/jira/browse/HDFS-1708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13001835#comment-13001835 ] Boris Shkolnik commented on HDFS-1708: -- +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no new tests are needed for this patch. [exec] Also please list what manual steps were performed to verify this patch. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 remove datanode's datanodeId TODOs --- Key: HDFS-1708 URL: https://issues.apache.org/jira/browse/HDFS-1708 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1670) HDFS Federation: remove dnRegistration from Datanode
[ https://issues.apache.org/jira/browse/HDFS-1670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1670: - Attachment: HDFS-1670-1.patch merged with current branch HDFS Federation: remove dnRegistration from Datanode Key: HDFS-1670 URL: https://issues.apache.org/jira/browse/HDFS-1670 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1670-1.patch, HDFS-1670.patch -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
