[jira] [Assigned] (HDFS-1490) TransferFSImage should timeout
[ https://issues.apache.org/jira/browse/HDFS-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay reassigned HDFS-1490: --- Assignee: Vinay (was: Dmytro Molkov) > TransferFSImage should timeout > -- > > Key: HDFS-1490 > URL: https://issues.apache.org/jira/browse/HDFS-1490 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Reporter: Dmytro Molkov >Assignee: Vinay >Priority: Minor > Fix For: 3.0.0, 2.2.0-alpha > > Attachments: HDFS-1490.patch, HDFS-1490.patch, HDFS-1490.patch, > HDFS-1490.patch > > > Sometimes when primary crashes during image transfer secondary namenode would > hang trying to read the image from HTTP connection forever. > It would be great to set timeouts on the connection so if something like that > happens there is no need to restart the secondary itself. > In our case restarting components is handled by the set of scripts and since > the Secondary as the process is running it would just stay hung until we get > an alarm saying the checkpointing doesn't happen. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1490) TransferFSImage should timeout
[ https://issues.apache.org/jira/browse/HDFS-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448519#comment-13448519 ] Hudson commented on HDFS-1490: -- Integrated in Hadoop-Mapreduce-trunk-Commit #2707 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2707/]) HDFS-1490. TransferFSImage should timeout. Contributed by Dmytro Molkov and Vinay. (Revision 1380988) Result = FAILURE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1380988 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestTransferFsImage.java > TransferFSImage should timeout > -- > > Key: HDFS-1490 > URL: https://issues.apache.org/jira/browse/HDFS-1490 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Reporter: Dmytro Molkov >Assignee: Dmytro Molkov >Priority: Minor > Fix For: 3.0.0, 2.2.0-alpha > > Attachments: HDFS-1490.patch, HDFS-1490.patch, HDFS-1490.patch, > HDFS-1490.patch > > > Sometimes when primary crashes during image transfer secondary namenode would > hang trying to read the image from HTTP connection forever. > It would be great to set timeouts on the connection so if something like that > happens there is no need to restart the secondary itself. > In our case restarting components is handled by the set of scripts and since > the Secondary as the process is running it would just stay hung until we get > an alarm saying the checkpointing doesn't happen. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2793) Add an admin command to trigger an edit log roll
[ https://issues.apache.org/jira/browse/HDFS-2793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448500#comment-13448500 ] Hudson commented on HDFS-2793: -- Integrated in Hadoop-Mapreduce-trunk-Commit #2706 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2706/]) HDFS-2793. Add an admin command to trigger an edit log roll. Contributed by Todd Lipcon. (Revision 1380982) Result = FAILURE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1380982 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml > Add an admin command to trigger an edit log roll > > > Key: HDFS-2793 > URL: https://issues.apache.org/jira/browse/HDFS-2793 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Affects Versions: 0.24.0 >Reporter: Aaron T. Myers >Assignee: Todd Lipcon > Fix For: 3.0.0, 2.2.0-alpha > > Attachments: hdfs-2793.txt, hdfs-2793.txt, hdfs-2793.txt > > > This seems like it would also be helpful outside of the context of HA, but > especially so given that the standby can currently only read finalized log > segments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1490) TransferFSImage should timeout
[ https://issues.apache.org/jira/browse/HDFS-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448485#comment-13448485 ] Hudson commented on HDFS-1490: -- Integrated in Hadoop-Hdfs-trunk-Commit #2746 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2746/]) HDFS-1490. TransferFSImage should timeout. Contributed by Dmytro Molkov and Vinay. (Revision 1380988) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1380988 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestTransferFsImage.java > TransferFSImage should timeout > -- > > Key: HDFS-1490 > URL: https://issues.apache.org/jira/browse/HDFS-1490 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Reporter: Dmytro Molkov >Assignee: Dmytro Molkov >Priority: Minor > Fix For: 3.0.0, 2.2.0-alpha > > Attachments: HDFS-1490.patch, HDFS-1490.patch, HDFS-1490.patch, > HDFS-1490.patch > > > Sometimes when primary crashes during image transfer secondary namenode would > hang trying to read the image from HTTP connection forever. > It would be great to set timeouts on the connection so if something like that > happens there is no need to restart the secondary itself. > In our case restarting components is handled by the set of scripts and since > the Secondary as the process is running it would just stay hung until we get > an alarm saying the checkpointing doesn't happen. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1490) TransferFSImage should timeout
[ https://issues.apache.org/jira/browse/HDFS-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448484#comment-13448484 ] Hudson commented on HDFS-1490: -- Integrated in Hadoop-Common-trunk-Commit #2683 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2683/]) HDFS-1490. TransferFSImage should timeout. Contributed by Dmytro Molkov and Vinay. (Revision 1380988) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1380988 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/TransferFsImage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestTransferFsImage.java > TransferFSImage should timeout > -- > > Key: HDFS-1490 > URL: https://issues.apache.org/jira/browse/HDFS-1490 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Reporter: Dmytro Molkov >Assignee: Dmytro Molkov >Priority: Minor > Fix For: 3.0.0, 2.2.0-alpha > > Attachments: HDFS-1490.patch, HDFS-1490.patch, HDFS-1490.patch, > HDFS-1490.patch > > > Sometimes when primary crashes during image transfer secondary namenode would > hang trying to read the image from HTTP connection forever. > It would be great to set timeouts on the connection so if something like that > happens there is no need to restart the secondary itself. > In our case restarting components is handled by the set of scripts and since > the Secondary as the process is running it would just stay hung until we get > an alarm saying the checkpointing doesn't happen. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-1490) TransferFSImage should timeout
[ https://issues.apache.org/jira/browse/HDFS-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon resolved HDFS-1490. --- Resolution: Fixed Hadoop Flags: Reviewed > TransferFSImage should timeout > -- > > Key: HDFS-1490 > URL: https://issues.apache.org/jira/browse/HDFS-1490 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Reporter: Dmytro Molkov >Assignee: Dmytro Molkov >Priority: Minor > Fix For: 3.0.0, 2.2.0-alpha > > Attachments: HDFS-1490.patch, HDFS-1490.patch, HDFS-1490.patch, > HDFS-1490.patch > > > Sometimes when primary crashes during image transfer secondary namenode would > hang trying to read the image from HTTP connection forever. > It would be great to set timeouts on the connection so if something like that > happens there is no need to restart the secondary itself. > In our case restarting components is handled by the set of scripts and since > the Secondary as the process is running it would just stay hung until we get > an alarm saying the checkpointing doesn't happen. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1490) TransferFSImage should timeout
[ https://issues.apache.org/jira/browse/HDFS-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-1490: -- Fix Version/s: 2.2.0-alpha 3.0.0 Status: In Progress (was: Patch Available) Committed to branch-2 and trunk. Thanks Dmytro and Vinay! > TransferFSImage should timeout > -- > > Key: HDFS-1490 > URL: https://issues.apache.org/jira/browse/HDFS-1490 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Reporter: Dmytro Molkov >Assignee: Dmytro Molkov >Priority: Minor > Fix For: 3.0.0, 2.2.0-alpha > > Attachments: HDFS-1490.patch, HDFS-1490.patch, HDFS-1490.patch, > HDFS-1490.patch > > > Sometimes when primary crashes during image transfer secondary namenode would > hang trying to read the image from HTTP connection forever. > It would be great to set timeouts on the connection so if something like that > happens there is no need to restart the secondary itself. > In our case restarting components is handled by the set of scripts and since > the Secondary as the process is running it would just stay hung until we get > an alarm saying the checkpointing doesn't happen. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1490) TransferFSImage should timeout
[ https://issues.apache.org/jira/browse/HDFS-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448476#comment-13448476 ] Todd Lipcon commented on HDFS-1490: --- +1, looks good to me. Will commit momentarily. > TransferFSImage should timeout > -- > > Key: HDFS-1490 > URL: https://issues.apache.org/jira/browse/HDFS-1490 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Reporter: Dmytro Molkov >Assignee: Dmytro Molkov >Priority: Minor > Attachments: HDFS-1490.patch, HDFS-1490.patch, HDFS-1490.patch, > HDFS-1490.patch > > > Sometimes when primary crashes during image transfer secondary namenode would > hang trying to read the image from HTTP connection forever. > It would be great to set timeouts on the connection so if something like that > happens there is no need to restart the secondary itself. > In our case restarting components is handled by the set of scripts and since > the Secondary as the process is running it would just stay hung until we get > an alarm saying the checkpointing doesn't happen. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2793) Add an admin command to trigger an edit log roll
[ https://issues.apache.org/jira/browse/HDFS-2793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448473#comment-13448473 ] Hudson commented on HDFS-2793: -- Integrated in Hadoop-Hdfs-trunk-Commit #2745 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2745/]) HDFS-2793. Add an admin command to trigger an edit log roll. Contributed by Todd Lipcon. (Revision 1380982) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1380982 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml > Add an admin command to trigger an edit log roll > > > Key: HDFS-2793 > URL: https://issues.apache.org/jira/browse/HDFS-2793 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Affects Versions: 0.24.0 >Reporter: Aaron T. Myers >Assignee: Todd Lipcon > Fix For: 3.0.0, 2.2.0-alpha > > Attachments: hdfs-2793.txt, hdfs-2793.txt, hdfs-2793.txt > > > This seems like it would also be helpful outside of the context of HA, but > especially so given that the standby can currently only read finalized log > segments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2793) Add an admin command to trigger an edit log roll
[ https://issues.apache.org/jira/browse/HDFS-2793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448470#comment-13448470 ] Hudson commented on HDFS-2793: -- Integrated in Hadoop-Common-trunk-Commit #2682 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2682/]) HDFS-2793. Add an admin command to trigger an edit log roll. Contributed by Todd Lipcon. (Revision 1380982) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1380982 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolServerSideTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSAdmin.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/ClientNamenodeProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/testHDFSConf.xml > Add an admin command to trigger an edit log roll > > > Key: HDFS-2793 > URL: https://issues.apache.org/jira/browse/HDFS-2793 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Affects Versions: 0.24.0 >Reporter: Aaron T. Myers >Assignee: Todd Lipcon > Fix For: 3.0.0, 2.2.0-alpha > > Attachments: hdfs-2793.txt, hdfs-2793.txt, hdfs-2793.txt > > > This seems like it would also be helpful outside of the context of HA, but > especially so given that the standby can currently only read finalized log > segments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2793) Add an admin command to trigger an edit log roll
[ https://issues.apache.org/jira/browse/HDFS-2793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2793: -- Resolution: Fixed Fix Version/s: 2.2.0-alpha 3.0.0 Release Note: Introduced a new command, "hdfs dfsadmin -rollEdits" which requests that the active NameNode roll its edit log. This can be useful for administrators manually backing up log segments. Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to branch-2 and trunk. Thanks for the review, Aaron. > Add an admin command to trigger an edit log roll > > > Key: HDFS-2793 > URL: https://issues.apache.org/jira/browse/HDFS-2793 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Affects Versions: 0.24.0 >Reporter: Aaron T. Myers >Assignee: Todd Lipcon > Fix For: 3.0.0, 2.2.0-alpha > > Attachments: hdfs-2793.txt, hdfs-2793.txt, hdfs-2793.txt > > > This seems like it would also be helpful outside of the context of HA, but > especially so given that the standby can currently only read finalized log > segments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-3870) QJM: add metrics to JournalNode
[ https://issues.apache.org/jira/browse/HDFS-3870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon resolved HDFS-3870. --- Resolution: Fixed Fix Version/s: QuorumJournalManager (HDFS-3077) Hadoop Flags: Reviewed Committed to branch, thanks for the review. > QJM: add metrics to JournalNode > --- > > Key: HDFS-3870 > URL: https://issues.apache.org/jira/browse/HDFS-3870 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: QuorumJournalManager (HDFS-3077) >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: QuorumJournalManager (HDFS-3077) > > Attachments: hdfs-3870.txt > > > The JournalNode should expose some basic metrics through the usual interface. > In particular: > - the writer epoch, accepted epoch, > - the last written transaction ID and last committed txid (which may be newer > in case that it's in the process of catching up) > - latency information for how long the syncs are taking > Please feel free to suggest others that come to mind. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3884) QJM: Journal format() should reset cached values
[ https://issues.apache.org/jira/browse/HDFS-3884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448461#comment-13448461 ] Eli Collins commented on HDFS-3884: --- +1 to the delta, makes sense. > QJM: Journal format() should reset cached values > > > Key: HDFS-3884 > URL: https://issues.apache.org/jira/browse/HDFS-3884 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: QuorumJournalManager (HDFS-3077) >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: QuorumJournalManager (HDFS-3077) > > Attachments: hdfs-3884.txt, hdfs-3884.txt, hdfs-3884.txt > > > Simple bug in the JournalNode: it caches certain values (eg accepted epoch) > in memory, and the cached values aren't reset when the journal is formatted. > So, after a format, further calls to the same Journal will see the old value > for accepted epoch, writer epoch, etc, preventing the journal from being > re-used until the JN is restarted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3870) QJM: add metrics to JournalNode
[ https://issues.apache.org/jira/browse/HDFS-3870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448460#comment-13448460 ] Todd Lipcon commented on HDFS-3870: --- Just realized I never commented to respond to Chao's suggestion. It was a good one, so it's included in the patch. Will commit momentarily. > QJM: add metrics to JournalNode > --- > > Key: HDFS-3870 > URL: https://issues.apache.org/jira/browse/HDFS-3870 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: QuorumJournalManager (HDFS-3077) >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: hdfs-3870.txt > > > The JournalNode should expose some basic metrics through the usual interface. > In particular: > - the writer epoch, accepted epoch, > - the last written transaction ID and last committed txid (which may be newer > in case that it's in the process of catching up) > - latency information for how long the syncs are taking > Please feel free to suggest others that come to mind. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-3884) QJM: Journal format() should reset cached values
[ https://issues.apache.org/jira/browse/HDFS-3884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon resolved HDFS-3884. --- Resolution: Fixed Target Version/s: QuorumJournalManager (HDFS-3077) Hadoop Flags: Reviewed committed to branch, thx for reviews. > QJM: Journal format() should reset cached values > > > Key: HDFS-3884 > URL: https://issues.apache.org/jira/browse/HDFS-3884 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: QuorumJournalManager (HDFS-3077) >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: QuorumJournalManager (HDFS-3077) > > Attachments: hdfs-3884.txt, hdfs-3884.txt, hdfs-3884.txt > > > Simple bug in the JournalNode: it caches certain values (eg accepted epoch) > in memory, and the cached values aren't reset when the journal is formatted. > So, after a format, further calls to the same Journal will see the old value > for accepted epoch, writer epoch, etc, preventing the journal from being > re-used until the JN is restarted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3884) QJM: Journal format() should reset cached values
[
https://issues.apache.org/jira/browse/HDFS-3884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated HDFS-3884:
--
Attachment: hdfs-3884.txt
Attaching patch I'm about to commit. Had to slightly modify it due to changes
in the order the patches were staged vs committed. For it to compile, had to
add the following getter which was from the patch in HDFS-3870:
{code}
+ synchronized public long getLastWriterEpoch() throws IOException {
+checkFormatted();
+return lastWriterEpoch.get();
+ }
{code}
Since it's a trivial change, I'll commit based on the earlier +1s.
> QJM: Journal format() should reset cached values
>
>
> Key: HDFS-3884
> URL: https://issues.apache.org/jira/browse/HDFS-3884
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: QuorumJournalManager (HDFS-3077)
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
> Fix For: QuorumJournalManager (HDFS-3077)
>
> Attachments: hdfs-3884.txt, hdfs-3884.txt, hdfs-3884.txt
>
>
> Simple bug in the JournalNode: it caches certain values (eg accepted epoch)
> in memory, and the cached values aren't reset when the journal is formatted.
> So, after a format, further calls to the same Journal will see the old value
> for accepted epoch, writer epoch, etc, preventing the journal from being
> re-used until the JN is restarted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-3869) QJM: expose non-file journal manager details in web UI
[ https://issues.apache.org/jira/browse/HDFS-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon resolved HDFS-3869. --- Resolution: Fixed Fix Version/s: QuorumJournalManager (HDFS-3077) Hadoop Flags: Reviewed Committed to branch, thanks for the review. > QJM: expose non-file journal manager details in web UI > -- > > Key: HDFS-3869 > URL: https://issues.apache.org/jira/browse/HDFS-3869 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: QuorumJournalManager (HDFS-3077) >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: QuorumJournalManager (HDFS-3077) > > Attachments: dir-failed.png, hdfs-3869.txt, hdfs-3869.txt, > hdfs-3869.txt, hdfs-3869.txt, lagging-jn.png, open-for-read.png, > open-for-write.png > > > Currently, the NN web UI only contains NN storage directories on local disk. > It should also include details about any non-file JournalManagers in use. > This JIRA targets the QJM branch, but will be useful for BKJM as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-3863) QJM: track last "committed" txid
[ https://issues.apache.org/jira/browse/HDFS-3863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon resolved HDFS-3863. --- Resolution: Fixed Fix Version/s: QuorumJournalManager (HDFS-3077) Hadoop Flags: Reviewed Committed to branch, thanks for the reviews, Eli, and for the good ideas, Chao. > QJM: track last "committed" txid > > > Key: HDFS-3863 > URL: https://issues.apache.org/jira/browse/HDFS-3863 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha >Affects Versions: QuorumJournalManager (HDFS-3077) >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: QuorumJournalManager (HDFS-3077) > > Attachments: hdfs-3863-prelim.txt, hdfs-3863.txt, hdfs-3863.txt > > > Per some discussion with [~stepinto] > [here|https://issues.apache.org/jira/browse/HDFS-3077?focusedCommentId=13422579&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13422579], > we should keep track of the "last committed txid" on each JournalNode. Then > during any recovery operation, we can sanity-check that we aren't asked to > truncate a log to an earlier transaction. > This is also a necessary step if we want to support reading from in-progress > segments in the future (since we should only allow reads up to the commit > point) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2793) Add an admin command to trigger an edit log roll
[ https://issues.apache.org/jira/browse/HDFS-2793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448422#comment-13448422 ] Hadoop QA commented on HDFS-2793: - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12543787/hdfs-2793.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3147//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3147//console This message is automatically generated. > Add an admin command to trigger an edit log roll > > > Key: HDFS-2793 > URL: https://issues.apache.org/jira/browse/HDFS-2793 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Affects Versions: 0.24.0 >Reporter: Aaron T. Myers >Assignee: Todd Lipcon > Attachments: hdfs-2793.txt, hdfs-2793.txt, hdfs-2793.txt > > > This seems like it would also be helpful outside of the context of HA, but > especially so given that the standby can currently only read finalized log > segments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3889) distcp overwrites files even when there are missing checksums
[
https://issues.apache.org/jira/browse/HDFS-3889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448413#comment-13448413
]
Colin Patrick McCabe commented on HDFS-3889:
I guess I need to look more into why {{getFileChecksum}} would fail. If the
block file simply had no checksum to begin with, then skipping it is clearly
fine. I think that's the case this was designed to handle (although I could be
wrong here.) However, if we expected it to have a checksum and it didn't, then
that should be a red flag.
> distcp overwrites files even when there are missing checksums
> -
>
> Key: HDFS-3889
> URL: https://issues.apache.org/jira/browse/HDFS-3889
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tools
>Affects Versions: 2.2.0-alpha
>Reporter: Colin Patrick McCabe
>Priority: Minor
>
> If distcp can't read the checksum files for the source and destination
> files-- for any reason-- it ignores the checksums and overwrites the
> destination file. It does produce a log message, but I think the correct
> behavior would be to throw an error and stop the distcp.
> If the user really wants to ignore checksums, he or she can use
> {{-skipcrccheck}} to do so.
> The relevant code is in DistCpUtils#checksumsAreEquals:
> {code}
> try {
> sourceChecksum = sourceFS.getFileChecksum(source);
> targetChecksum = targetFS.getFileChecksum(target);
> } catch (IOException e) {
> LOG.error("Unable to retrieve checksum for " + source + " or " +
> target, e);
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2793) Add an admin command to trigger an edit log roll
[ https://issues.apache.org/jira/browse/HDFS-2793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448401#comment-13448401 ] Hadoop QA commented on HDFS-2793: - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12543780/hdfs-2793.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3146//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3146//console This message is automatically generated. > Add an admin command to trigger an edit log roll > > > Key: HDFS-2793 > URL: https://issues.apache.org/jira/browse/HDFS-2793 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Affects Versions: 0.24.0 >Reporter: Aaron T. Myers >Assignee: Todd Lipcon > Attachments: hdfs-2793.txt, hdfs-2793.txt, hdfs-2793.txt > > > This seems like it would also be helpful outside of the context of HA, but > especially so given that the standby can currently only read finalized log > segments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3888) BlockPlacementPolicyDefault code cleanup
[ https://issues.apache.org/jira/browse/HDFS-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448399#comment-13448399 ] Hadoop QA commented on HDFS-3888: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12543770/HDFS-3888.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics org.apache.hadoop.hdfs.TestPersistBlocks +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3145//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3145//console This message is automatically generated. > BlockPlacementPolicyDefault code cleanup > > > Key: HDFS-3888 > URL: https://issues.apache.org/jira/browse/HDFS-3888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Jing Zhao >Priority: Minor > Fix For: 2.2.0-alpha > > Attachments: HDFS-3888.patch, HDFS-3888.patch > > > BlockPlacementPolicyDefault#LOG should be removed as it hides LOG from the > base class BlockPlacementPolicy. Also, in > BlockPlacementPolicyDefault#chooseTarget method, the logic computing the > maxTargetPerLoc can be made a separate method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3889) distcp overwrites files even when there are missing checksums
[
https://issues.apache.org/jira/browse/HDFS-3889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448393#comment-13448393
]
Eli Collins commented on HDFS-3889:
---
Good find. Or perhaps have an option that checks CRCs but just logs. I imagine
the motivation for this was to not stop a large distcp job because one call to
getFileChecksum failed (though it's robust, eg checks multiple DNs, so that
should probably be rare).
> distcp overwrites files even when there are missing checksums
> -
>
> Key: HDFS-3889
> URL: https://issues.apache.org/jira/browse/HDFS-3889
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tools
>Affects Versions: 2.2.0-alpha
>Reporter: Colin Patrick McCabe
>Priority: Minor
>
> If distcp can't read the checksum files for the source and destination
> files-- for any reason-- it ignores the checksums and overwrites the
> destination file. It does produce a log message, but I think the correct
> behavior would be to throw an error and stop the distcp.
> If the user really wants to ignore checksums, he or she can use
> {{-skipcrccheck}} to do so.
> The relevant code is in DistCpUtils#checksumsAreEquals:
> {code}
> try {
> sourceChecksum = sourceFS.getFileChecksum(source);
> targetChecksum = targetFS.getFileChecksum(target);
> } catch (IOException e) {
> LOG.error("Unable to retrieve checksum for " + source + " or " +
> target, e);
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3869) QJM: expose non-file journal manager details in web UI
[ https://issues.apache.org/jira/browse/HDFS-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448392#comment-13448392 ] Eli Collins commented on HDFS-3869: --- +1 updated patch looks good > QJM: expose non-file journal manager details in web UI > -- > > Key: HDFS-3869 > URL: https://issues.apache.org/jira/browse/HDFS-3869 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Affects Versions: QuorumJournalManager (HDFS-3077) >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: dir-failed.png, hdfs-3869.txt, hdfs-3869.txt, > hdfs-3869.txt, hdfs-3869.txt, lagging-jn.png, open-for-read.png, > open-for-write.png > > > Currently, the NN web UI only contains NN storage directories on local disk. > It should also include details about any non-file JournalManagers in use. > This JIRA targets the QJM branch, but will be useful for BKJM as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3383) libhdfs does not build on ARM because jni_md.h is not found
[ https://issues.apache.org/jira/browse/HDFS-3383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448391#comment-13448391 ] Eli Collins commented on HDFS-3383: --- Happy to review if you post a patch for branch-1. > libhdfs does not build on ARM because jni_md.h is not found > --- > > Key: HDFS-3383 > URL: https://issues.apache.org/jira/browse/HDFS-3383 > Project: Hadoop HDFS > Issue Type: Bug > Components: libhdfs >Affects Versions: 0.23.1 > Environment: Linux 3.2.0-1412-omap4 #16-Ubuntu SMP PREEMPT Tue Apr 17 > 19:38:42 UTC 2012 armv7l armv7l armv7l GNU/Linux > java version "1.7.0_04-ea" > Java(TM) SE Runtime Environment for Embedded (build 1.7.0_04-ea-b20, headless) > Java HotSpot(TM) Embedded Server VM (build 23.0-b21, mixed mode, experimental) >Reporter: Trevor Robinson > Attachments: HDFS-3383.patch > > > The wrong include directory is used for jni_md.h: > [INFO] --- make-maven-plugin:1.0-beta-1:make-install (compile) @ hadoop-hdfs > --- > [INFO] /bin/bash ./libtool --tag=CC --mode=compile gcc > -DPACKAGE_NAME=\"libhdfs\" -DPACKAGE_TARNAME=\"libhdfs\" > -DPACKAGE_VERSION=\"0.1.0\" -DPACKAGE_STRING=\"libhdfs\ 0.1.0\" > -DPACKAGE_BUGREPORT=\"omal...@apache.org\" -DPACKAGE_URL=\"\" > -DPACKAGE=\"libhdfs\" -DVERSION=\"0.1.0\" -DSTDC_HEADERS=1 > -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 > -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 > -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=\".libs/\" -DHAVE_STRDUP=1 > -DHAVE_STRERROR=1 -DHAVE_STRTOUL=1 -DHAVE_FCNTL_H=1 -DHAVE__BOOL=1 > -DHAVE_STDBOOL_H=1 -I. -g -O2 -DOS_LINUX -DDSO_DLFCN -DCPU=\"arm\" > -I/usr/lib/jvm/ejdk1.7.0_04/include -I/usr/lib/jvm/ejdk1.7.0_04/include/arm > -Wall -Wstrict-prototypes -MT hdfs.lo -MD -MP -MF .deps/hdfs.Tpo -c -o > hdfs.lo hdfs.c > [INFO] libtool: compile: gcc -DPACKAGE_NAME=\"libhdfs\" > -DPACKAGE_TARNAME=\"libhdfs\" -DPACKAGE_VERSION=\"0.1.0\" > "-DPACKAGE_STRING=\"libhdfs 0.1.0\"" > -DPACKAGE_BUGREPORT=\"omal...@apache.org\" -DPACKAGE_URL=\"\" > -DPACKAGE=\"libhdfs\" -DVERSION=\"0.1.0\" -DSTDC_HEADERS=1 > -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 > -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 > -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=\".libs/\" -DHAVE_STRDUP=1 > -DHAVE_STRERROR=1 -DHAVE_STRTOUL=1 -DHAVE_FCNTL_H=1 -DHAVE__BOOL=1 > -DHAVE_STDBOOL_H=1 -I. -g -O2 -DOS_LINUX -DDSO_DLFCN -DCPU=\"arm\" > -I/usr/lib/jvm/ejdk1.7.0_04/include -I/usr/lib/jvm/ejdk1.7.0_04/include/arm > -Wall -Wstrict-prototypes -MT hdfs.lo -MD -MP -MF .deps/hdfs.Tpo -c hdfs.c > -fPIC -DPIC -o .libs/hdfs.o > [INFO] In file included from hdfs.h:33:0, > [INFO] from hdfs.c:19: > [INFO] /usr/lib/jvm/ejdk1.7.0_04/include/jni.h:45:20: fatal error: jni_md.h: > No such file or directory > [INFO] compilation terminated. > [INFO] make: *** [hdfs.lo] Error 1 > The problem is caused by > hadoop-hdfs-project/hadoop-hdfs/src/main/native/m4/apsupport.m4 overriding > supported_os=arm when host_cpu=arm*; supported_os should remain "linux", > since it determines the jni_md.h include path. OpenJDK 6 and 7 (in Ubuntu > 12.04, at least) and Oracle EJDK put jni_md.h in include/linux. Not sure > if/why this ever worked before. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3863) QJM: track last "committed" txid
[ https://issues.apache.org/jira/browse/HDFS-3863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448387#comment-13448387 ] Eli Collins commented on HDFS-3863: --- +1 looks great > QJM: track last "committed" txid > > > Key: HDFS-3863 > URL: https://issues.apache.org/jira/browse/HDFS-3863 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha >Affects Versions: QuorumJournalManager (HDFS-3077) >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: hdfs-3863-prelim.txt, hdfs-3863.txt, hdfs-3863.txt > > > Per some discussion with [~stepinto] > [here|https://issues.apache.org/jira/browse/HDFS-3077?focusedCommentId=13422579&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13422579], > we should keep track of the "last committed txid" on each JournalNode. Then > during any recovery operation, we can sanity-check that we aren't asked to > truncate a log to an earlier transaction. > This is also a necessary step if we want to support reading from in-progress > segments in the future (since we should only allow reads up to the commit > point) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3054) distcp -skipcrccheck has no effect
[
https://issues.apache.org/jira/browse/HDFS-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448386#comment-13448386
]
Colin Patrick McCabe commented on HDFS-3054:
testing done:
I confirmed that copying files from a cluster running branch-1 derived code to
a cluster running branch-2 derived code did *not* work unless {{-skipcrccheck}}
was supplied.
The exception was this:
{code}
Error: java.io.IOException: File copy failed: hftp://172.22.1.204:6001/a/xx
--> hdfs://localhost:6000/b/a/xx
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:267)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:229)
at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:45)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:726)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:153)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1367)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:148)
Caused by: java.io.IOException: Couldn't run retriable-command: Copying
hftp://172.22.1.204:6001/a/xx to hdfs://localhost:6000/b/a/xx
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:101)
at
org.apache.hadoop.tools.mapred.CopyMapper.copyFileWithRetry(CopyMapper.java:263)
... 10 more
Caused by: java.io.IOException: Check-sum mismatch between
hftp://172.22.1.204:6001/a/xx and
hdfs://localhost:6000/b/.distcp.tmp.attempt_1346456743556_0010_m_01_0
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.compareCheckSums(RetriableFileCopyCommand.java:145)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:107)
at
org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:83)
at
org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
... 11 more
{code}
With {{-skipcrccheck}}, this problem did not occur.
> distcp -skipcrccheck has no effect
> --
>
> Key: HDFS-3054
> URL: https://issues.apache.org/jira/browse/HDFS-3054
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tools
>Affects Versions: 0.23.2, 2.0.0-alpha, 2.0.1-alpha, 2.2.0-alpha
>Reporter: patrick white
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-3054.002.patch, hdfs-3054.patch
>
>
> Using distcp with '-skipcrccheck' still seems to cause CRC checksums to
> happen.
> Ran into this while debugging an issue associated with source and destination
> having different blocksizes, and not using the preserve blocksize parameter
> (-pb). In both 23.1 and 23.2 builds, trying to bypass the checksum
> verification by using the '-skipcrcrcheck' parameter had no effect, the
> distcp still failed on checksum errors.
> Test scenario to reproduce;
> do not use '-pb' and try a distcp from 20.205 (default blksize=128M) to .23
> (default blksize=256M), the distcp fails on checksum errors, which is
> expected due to checksum calculation (tiered aggregation of all blks). Trying
> the same distcp only providing '-skipcrccheck' still fails with the same
> checksum error, it is expected that checksum would now be bypassed and the
> distcp would proceed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2793) Add an admin command to trigger an edit log roll
[ https://issues.apache.org/jira/browse/HDFS-2793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448383#comment-13448383 ] Aaron T. Myers commented on HDFS-2793: -- The latest patch looks good to me. +1 pending Jenkings. > Add an admin command to trigger an edit log roll > > > Key: HDFS-2793 > URL: https://issues.apache.org/jira/browse/HDFS-2793 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Affects Versions: 0.24.0 >Reporter: Aaron T. Myers >Assignee: Todd Lipcon > Attachments: hdfs-2793.txt, hdfs-2793.txt, hdfs-2793.txt > > > This seems like it would also be helpful outside of the context of HA, but > especially so given that the standby can currently only read finalized log > segments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2793) Add an admin command to trigger an edit log roll
[ https://issues.apache.org/jira/browse/HDFS-2793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2793: -- Attachment: hdfs-2793.txt Oops. good catch. New rev. > Add an admin command to trigger an edit log roll > > > Key: HDFS-2793 > URL: https://issues.apache.org/jira/browse/HDFS-2793 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Affects Versions: 0.24.0 >Reporter: Aaron T. Myers >Assignee: Todd Lipcon > Attachments: hdfs-2793.txt, hdfs-2793.txt, hdfs-2793.txt > > > This seems like it would also be helpful outside of the context of HA, but > especially so given that the standby can currently only read finalized log > segments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3887) Remove redundant chooseTarget methods in BlockPlacementPolicy.java
[ https://issues.apache.org/jira/browse/HDFS-3887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448370#comment-13448370 ] Hudson commented on HDFS-3887: -- Integrated in Hadoop-Mapreduce-trunk-Commit #2704 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2704/]) HDFS-3887. Remove redundant chooseTarget methods in BlockPlacementPolicy. Contributed by Jing Zhao (Revision 1380934) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1380934 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java > Remove redundant chooseTarget methods in BlockPlacementPolicy.java > -- > > Key: HDFS-3887 > URL: https://issues.apache.org/jira/browse/HDFS-3887 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Jing Zhao >Priority: Trivial > Fix For: 2.2.0-alpha > > Attachments: HDFS-3887.patch > > > BlockPlacementPolicy.java contains multiple chooseTarget() methods with > different parameter lists. It is difficult to follow and understand the code > since some chooseTarget methods only have minor differences and some of them > are only invoked by testing code. > In this patch, I try to remove some of the chooseTarget methods and only keep > three of them: two abstract methods and the third one using BlockCollection > as its parameter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3888) BlockPlacementPolicyDefault code cleanup
[ https://issues.apache.org/jira/browse/HDFS-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448371#comment-13448371 ] Hudson commented on HDFS-3888: -- Integrated in Hadoop-Mapreduce-trunk-Commit #2704 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2704/]) HDFS-3888. Clean up BlockPlacementPolicyDefault. Contributed by Jing Zhao (Revision 1380939) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1380939 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java > BlockPlacementPolicyDefault code cleanup > > > Key: HDFS-3888 > URL: https://issues.apache.org/jira/browse/HDFS-3888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Jing Zhao >Priority: Minor > Fix For: 2.2.0-alpha > > Attachments: HDFS-3888.patch, HDFS-3888.patch > > > BlockPlacementPolicyDefault#LOG should be removed as it hides LOG from the > base class BlockPlacementPolicy. Also, in > BlockPlacementPolicyDefault#chooseTarget method, the logic computing the > maxTargetPerLoc can be made a separate method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3889) distcp silently ignores missing checksums
Colin Patrick McCabe created HDFS-3889:
--
Summary: distcp silently ignores missing checksums
Key: HDFS-3889
URL: https://issues.apache.org/jira/browse/HDFS-3889
Project: Hadoop HDFS
Issue Type: Bug
Components: tools
Affects Versions: 2.2.0-alpha
Reporter: Colin Patrick McCabe
Priority: Minor
If distcp can't read the checksum files for the source and destination files--
for any reason-- it ignores the checksums and overwrites the destination file.
It does produce a log message, but I think the correct behavior would be to
throw an error and stop the distcp.
If the user really wants to ignore checksums, he or she can use
{{-skipcrccheck}} to do so.
The relevant code is in DistCpUtils#checksumsAreEquals:
{code}
try {
sourceChecksum = sourceFS.getFileChecksum(source);
targetChecksum = targetFS.getFileChecksum(target);
} catch (IOException e) {
LOG.error("Unable to retrieve checksum for " + source + " or " + target,
e);
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3889) distcp overwrites files even when there are missing checksums
[
https://issues.apache.org/jira/browse/HDFS-3889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-3889:
---
Summary: distcp overwrites files even when there are missing checksums
(was: distcp silently ignores missing checksums)
> distcp overwrites files even when there are missing checksums
> -
>
> Key: HDFS-3889
> URL: https://issues.apache.org/jira/browse/HDFS-3889
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tools
>Affects Versions: 2.2.0-alpha
>Reporter: Colin Patrick McCabe
>Priority: Minor
>
> If distcp can't read the checksum files for the source and destination
> files-- for any reason-- it ignores the checksums and overwrites the
> destination file. It does produce a log message, but I think the correct
> behavior would be to throw an error and stop the distcp.
> If the user really wants to ignore checksums, he or she can use
> {{-skipcrccheck}} to do so.
> The relevant code is in DistCpUtils#checksumsAreEquals:
> {code}
> try {
> sourceChecksum = sourceFS.getFileChecksum(source);
> targetChecksum = targetFS.getFileChecksum(target);
> } catch (IOException e) {
> LOG.error("Unable to retrieve checksum for " + source + " or " +
> target, e);
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2793) Add an admin command to trigger an edit log roll
[
https://issues.apache.org/jira/browse/HDFS-2793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448364#comment-13448364
]
Aaron T. Myers commented on HDFS-2793:
--
One nit, I think this should just be "ClientProtocol":
{code}
+ @Override // ClientNamenodeProtocol
{code}
> Add an admin command to trigger an edit log roll
>
>
> Key: HDFS-2793
> URL: https://issues.apache.org/jira/browse/HDFS-2793
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: name-node
>Affects Versions: 0.24.0
>Reporter: Aaron T. Myers
>Assignee: Todd Lipcon
> Attachments: hdfs-2793.txt, hdfs-2793.txt
>
>
> This seems like it would also be helpful outside of the context of HA, but
> especially so given that the standby can currently only read finalized log
> segments.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2793) Add an admin command to trigger an edit log roll
[ https://issues.apache.org/jira/browse/HDFS-2793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2793: -- Attachment: hdfs-2793.txt How's this look? > Add an admin command to trigger an edit log roll > > > Key: HDFS-2793 > URL: https://issues.apache.org/jira/browse/HDFS-2793 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Affects Versions: 0.24.0 >Reporter: Aaron T. Myers >Assignee: Todd Lipcon > Attachments: hdfs-2793.txt, hdfs-2793.txt > > > This seems like it would also be helpful outside of the context of HA, but > especially so given that the standby can currently only read finalized log > segments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3540) Further improvement on recovery mode and edit log toleration in branch-1
[ https://issues.apache.org/jira/browse/HDFS-3540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448357#comment-13448357 ] Colin Patrick McCabe commented on HDFS-3540: Here's a table of the abilities of the two features (Recovery mode and edit log toleration): || ||skip over edits||discard the end of the edit log||requires administrator intervention|| |edit log toleration|no|yes|no| |edit log recovery mode|branch-2 and later|yes|yes| We have considered enhancing RM in branch-1 so that it can skip over certain edits. When handling corruptions caused by HDFS-3652, it would have been nice to have this capability. > Further improvement on recovery mode and edit log toleration in branch-1 > > > Key: HDFS-3540 > URL: https://issues.apache.org/jira/browse/HDFS-3540 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 1.2.0 >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > > *Recovery Mode*: HDFS-3479 backported HDFS-3335 to branch-1. However, the > recovery mode feature in branch-1 is dramatically different from the recovery > mode in trunk since the edit log implementations in these two branch are > different. For example, there is UNCHECKED_REGION_LENGTH in branch-1 but not > in trunk. > *Edit Log Toleration*: HDFS-3521 added this feature to branch-1 to remedy > UNCHECKED_REGION_LENGTH and to tolerate edit log corruption. > There are overlaps between these two features. We study potential further > improvement in this issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3540) Further improvement on recovery mode and edit log toleration in branch-1
[ https://issues.apache.org/jira/browse/HDFS-3540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448277#comment-13448277 ] Colin Patrick McCabe commented on HDFS-3540: bq. I should be more specific: For Recovery Mode without HDFS-3479, if there is corruption in the middle of the edit log and the user chooses "stop reading" in recovery mode, then the remaining data of the edit log will not be checked. Is it correct? Yes, if the user chooses "stop reading" in Recovery Mode, the rest of the edit log will be discarded. > Further improvement on recovery mode and edit log toleration in branch-1 > > > Key: HDFS-3540 > URL: https://issues.apache.org/jira/browse/HDFS-3540 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 1.2.0 >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > > *Recovery Mode*: HDFS-3479 backported HDFS-3335 to branch-1. However, the > recovery mode feature in branch-1 is dramatically different from the recovery > mode in trunk since the edit log implementations in these two branch are > different. For example, there is UNCHECKED_REGION_LENGTH in branch-1 but not > in trunk. > *Edit Log Toleration*: HDFS-3521 added this feature to branch-1 to remedy > UNCHECKED_REGION_LENGTH and to tolerate edit log corruption. > There are overlaps between these two features. We study potential further > improvement in this issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3866) HttpFS POM should have property where to download tomcat from
[
https://issues.apache.org/jira/browse/HDFS-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448243#comment-13448243
]
Hudson commented on HDFS-3866:
--
Integrated in Hadoop-Mapreduce-trunk-Commit #2703 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2703/])
HDFS-3866. HttpFS POM should have property where to download tomcat from
(zero45 via tucu) (Revision 1380927)
Result = FAILURE
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1380927
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> HttpFS POM should have property where to download tomcat from
> -
>
> Key: HDFS-3866
> URL: https://issues.apache.org/jira/browse/HDFS-3866
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: build
>Affects Versions: 2.0.0-alpha
> Environment: CDH4 build on CentOS 6.2
>Reporter: Ryan Hennig
>Assignee: Plamen Jeliazkov
>Priority: Minor
> Fix For: 2.2.0-alpha
>
> Attachments: HDFS-3866.patch
>
>
> When trying to enable a build of CDH4 in Jenkins, I got a build error due to
> an attempt to download Tomcat from the internet directly instead of via Maven
> and thus our internal Maven repository.
> The problem is due to this line in
> src/hadoop-hdfs-project/hadoop-hdfs-httpfs/target/antrun/build-main.xml:
>src="http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.32/bin/apache-tomcat-6.0.32.tar.gz"/>
> This build.xml is generated from
> src/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml:
> src="http://archive.apache.org/dist/tomcat/tomcat-6/v${tomcat.version}/bin/apache-tomcat-${tomcat.version}.tar.gz";
> dest="downloads/tomcat.tar.gz" verbose="true" skipexisting="true"/>
> Instead of directly downloading from a hardcoded location, the Tomcat
> dependency should be managed by Maven. This would enable the use of a local
> repository for build machines without internet access.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3888) BlockPlacementPolicyDefault code cleanup
[ https://issues.apache.org/jira/browse/HDFS-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448233#comment-13448233 ] Hudson commented on HDFS-3888: -- Integrated in Hadoop-Common-trunk-Commit #2679 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2679/]) HDFS-3888. Clean up BlockPlacementPolicyDefault. Contributed by Jing Zhao (Revision 1380939) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1380939 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java > BlockPlacementPolicyDefault code cleanup > > > Key: HDFS-3888 > URL: https://issues.apache.org/jira/browse/HDFS-3888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Jing Zhao >Priority: Minor > Fix For: 2.2.0-alpha > > Attachments: HDFS-3888.patch, HDFS-3888.patch > > > BlockPlacementPolicyDefault#LOG should be removed as it hides LOG from the > base class BlockPlacementPolicy. Also, in > BlockPlacementPolicyDefault#chooseTarget method, the logic computing the > maxTargetPerLoc can be made a separate method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3888) BlockPlacementPolicyDefault code cleanup
[ https://issues.apache.org/jira/browse/HDFS-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-3888: - Resolution: Fixed Fix Version/s: 2.2.0-alpha Status: Resolved (was: Patch Available) I have committed this. Thanks, Jing! > BlockPlacementPolicyDefault code cleanup > > > Key: HDFS-3888 > URL: https://issues.apache.org/jira/browse/HDFS-3888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Jing Zhao >Priority: Minor > Fix For: 2.2.0-alpha > > Attachments: HDFS-3888.patch, HDFS-3888.patch > > > BlockPlacementPolicyDefault#LOG should be removed as it hides LOG from the > base class BlockPlacementPolicy. Also, in > BlockPlacementPolicyDefault#chooseTarget method, the logic computing the > maxTargetPerLoc can be made a separate method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3888) BlockPlacementPolicyDefault code cleanup
[ https://issues.apache.org/jira/browse/HDFS-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448224#comment-13448224 ] Hudson commented on HDFS-3888: -- Integrated in Hadoop-Hdfs-trunk-Commit #2742 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2742/]) HDFS-3888. Clean up BlockPlacementPolicyDefault. Contributed by Jing Zhao (Revision 1380939) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1380939 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java > BlockPlacementPolicyDefault code cleanup > > > Key: HDFS-3888 > URL: https://issues.apache.org/jira/browse/HDFS-3888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Jing Zhao >Priority: Minor > Attachments: HDFS-3888.patch, HDFS-3888.patch > > > BlockPlacementPolicyDefault#LOG should be removed as it hides LOG from the > base class BlockPlacementPolicy. Also, in > BlockPlacementPolicyDefault#chooseTarget method, the logic computing the > maxTargetPerLoc can be made a separate method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3887) Remove redundant chooseTarget methods in BlockPlacementPolicy.java
[ https://issues.apache.org/jira/browse/HDFS-3887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448218#comment-13448218 ] Hudson commented on HDFS-3887: -- Integrated in Hadoop-Common-trunk-Commit #2678 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2678/]) HDFS-3887. Remove redundant chooseTarget methods in BlockPlacementPolicy. Contributed by Jing Zhao (Revision 1380934) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1380934 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java > Remove redundant chooseTarget methods in BlockPlacementPolicy.java > -- > > Key: HDFS-3887 > URL: https://issues.apache.org/jira/browse/HDFS-3887 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Jing Zhao >Priority: Trivial > Fix For: 2.2.0-alpha > > Attachments: HDFS-3887.patch > > > BlockPlacementPolicy.java contains multiple chooseTarget() methods with > different parameter lists. It is difficult to follow and understand the code > since some chooseTarget methods only have minor differences and some of them > are only invoked by testing code. > In this patch, I try to remove some of the chooseTarget methods and only keep > three of them: two abstract methods and the third one using BlockCollection > as its parameter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3887) Remove redundant chooseTarget methods in BlockPlacementPolicy.java
[ https://issues.apache.org/jira/browse/HDFS-3887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448213#comment-13448213 ] Hudson commented on HDFS-3887: -- Integrated in Hadoop-Hdfs-trunk-Commit #2741 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2741/]) HDFS-3887. Remove redundant chooseTarget methods in BlockPlacementPolicy. Contributed by Jing Zhao (Revision 1380934) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1380934 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicy.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicyWithNodeGroup.java > Remove redundant chooseTarget methods in BlockPlacementPolicy.java > -- > > Key: HDFS-3887 > URL: https://issues.apache.org/jira/browse/HDFS-3887 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Jing Zhao >Priority: Trivial > Fix For: 2.2.0-alpha > > Attachments: HDFS-3887.patch > > > BlockPlacementPolicy.java contains multiple chooseTarget() methods with > different parameter lists. It is difficult to follow and understand the code > since some chooseTarget methods only have minor differences and some of them > are only invoked by testing code. > In this patch, I try to remove some of the chooseTarget methods and only keep > three of them: two abstract methods and the third one using BlockCollection > as its parameter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3887) Remove redundant chooseTarget methods in BlockPlacementPolicy.java
[ https://issues.apache.org/jira/browse/HDFS-3887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-3887: - Resolution: Fixed Fix Version/s: 2.2.0-alpha Status: Resolved (was: Patch Available) I have committed this. Thanks, Jing! > Remove redundant chooseTarget methods in BlockPlacementPolicy.java > -- > > Key: HDFS-3887 > URL: https://issues.apache.org/jira/browse/HDFS-3887 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Jing Zhao >Priority: Trivial > Fix For: 2.2.0-alpha > > Attachments: HDFS-3887.patch > > > BlockPlacementPolicy.java contains multiple chooseTarget() methods with > different parameter lists. It is difficult to follow and understand the code > since some chooseTarget methods only have minor differences and some of them > are only invoked by testing code. > In this patch, I try to remove some of the chooseTarget methods and only keep > three of them: two abstract methods and the third one using BlockCollection > as its parameter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3888) BlockPlacementPolicyDefault code cleanup
[ https://issues.apache.org/jira/browse/HDFS-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448209#comment-13448209 ] Hadoop QA commented on HDFS-3888: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12543755/HDFS-3888.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3143//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3143//console This message is automatically generated. > BlockPlacementPolicyDefault code cleanup > > > Key: HDFS-3888 > URL: https://issues.apache.org/jira/browse/HDFS-3888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Jing Zhao >Priority: Minor > Attachments: HDFS-3888.patch, HDFS-3888.patch > > > BlockPlacementPolicyDefault#LOG should be removed as it hides LOG from the > base class BlockPlacementPolicy. Also, in > BlockPlacementPolicyDefault#chooseTarget method, the logic computing the > maxTargetPerLoc can be made a separate method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3887) Remove redundant chooseTarget methods in BlockPlacementPolicy.java
[ https://issues.apache.org/jira/browse/HDFS-3887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-3887: - Component/s: name-node Hadoop Flags: Reviewed +1 patch looks good. > Remove redundant chooseTarget methods in BlockPlacementPolicy.java > -- > > Key: HDFS-3887 > URL: https://issues.apache.org/jira/browse/HDFS-3887 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Jing Zhao >Priority: Trivial > Attachments: HDFS-3887.patch > > > BlockPlacementPolicy.java contains multiple chooseTarget() methods with > different parameter lists. It is difficult to follow and understand the code > since some chooseTarget methods only have minor differences and some of them > are only invoked by testing code. > In this patch, I try to remove some of the chooseTarget methods and only keep > three of them: two abstract methods and the third one using BlockCollection > as its parameter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3888) BlockPlacementPolicyDefault code cleanup
[ https://issues.apache.org/jira/browse/HDFS-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-3888: - Status: Patch Available (was: Open) > BlockPlacementPolicyDefault code cleanup > > > Key: HDFS-3888 > URL: https://issues.apache.org/jira/browse/HDFS-3888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Jing Zhao >Priority: Minor > Attachments: HDFS-3888.patch, HDFS-3888.patch > > > BlockPlacementPolicyDefault#LOG should be removed as it hides LOG from the > base class BlockPlacementPolicy. Also, in > BlockPlacementPolicyDefault#chooseTarget method, the logic computing the > maxTargetPerLoc can be made a separate method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3888) BlockPlacementPolicyDefault code cleanup
[ https://issues.apache.org/jira/browse/HDFS-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-3888: - Hadoop Flags: Reviewed +1 patch looks good. > BlockPlacementPolicyDefault code cleanup > > > Key: HDFS-3888 > URL: https://issues.apache.org/jira/browse/HDFS-3888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Jing Zhao >Priority: Minor > Attachments: HDFS-3888.patch, HDFS-3888.patch > > > BlockPlacementPolicyDefault#LOG should be removed as it hides LOG from the > base class BlockPlacementPolicy. Also, in > BlockPlacementPolicyDefault#chooseTarget method, the logic computing the > maxTargetPerLoc can be made a separate method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3540) Further improvement on recovery mode and edit log toleration in branch-1
[
https://issues.apache.org/jira/browse/HDFS-3540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448185#comment-13448185
]
Tsz Wo (Nicholas), SZE commented on HDFS-3540:
--
{quote}
>Correct me if I am wrong: Recovery Mode without HDFS-3479 means the entire
> end-of-log is not checked and, therefore, the silent data loss length is not
> limited. It is even worst.
No, that is incorrect.
Recovery mode has always read up to the end of the log, and it always will. The
confusion arises because sometimes we are not very good at determining where
the "end of the log" is.
{quote}
I should be more specific: For Recovery Mode without HDFS-3479, if there is
corruption in the middle of the edit log and the user chooses "stop reading" in
recovery mode, then the remaining data of the edit log will not be checked. Is
it correct?
> Further improvement on recovery mode and edit log toleration in branch-1
>
>
> Key: HDFS-3540
> URL: https://issues.apache.org/jira/browse/HDFS-3540
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 1.2.0
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
>
> *Recovery Mode*: HDFS-3479 backported HDFS-3335 to branch-1. However, the
> recovery mode feature in branch-1 is dramatically different from the recovery
> mode in trunk since the edit log implementations in these two branch are
> different. For example, there is UNCHECKED_REGION_LENGTH in branch-1 but not
> in trunk.
> *Edit Log Toleration*: HDFS-3521 added this feature to branch-1 to remedy
> UNCHECKED_REGION_LENGTH and to tolerate edit log corruption.
> There are overlaps between these two features. We study potential further
> improvement in this issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3866) HttpFS POM should have property where to download tomcat from
[
https://issues.apache.org/jira/browse/HDFS-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448182#comment-13448182
]
Hudson commented on HDFS-3866:
--
Integrated in Hadoop-Hdfs-trunk-Commit #2740 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2740/])
HDFS-3866. HttpFS POM should have property where to download tomcat from
(zero45 via tucu) (Revision 1380927)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1380927
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> HttpFS POM should have property where to download tomcat from
> -
>
> Key: HDFS-3866
> URL: https://issues.apache.org/jira/browse/HDFS-3866
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: build
>Affects Versions: 2.0.0-alpha
> Environment: CDH4 build on CentOS 6.2
>Reporter: Ryan Hennig
>Assignee: Plamen Jeliazkov
>Priority: Minor
> Fix For: 2.2.0-alpha
>
> Attachments: HDFS-3866.patch
>
>
> When trying to enable a build of CDH4 in Jenkins, I got a build error due to
> an attempt to download Tomcat from the internet directly instead of via Maven
> and thus our internal Maven repository.
> The problem is due to this line in
> src/hadoop-hdfs-project/hadoop-hdfs-httpfs/target/antrun/build-main.xml:
>src="http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.32/bin/apache-tomcat-6.0.32.tar.gz"/>
> This build.xml is generated from
> src/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml:
> src="http://archive.apache.org/dist/tomcat/tomcat-6/v${tomcat.version}/bin/apache-tomcat-${tomcat.version}.tar.gz";
> dest="downloads/tomcat.tar.gz" verbose="true" skipexisting="true"/>
> Instead of directly downloading from a hardcoded location, the Tomcat
> dependency should be managed by Maven. This would enable the use of a local
> repository for build machines without internet access.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3866) HttpFS POM should have property where to download tomcat from
[
https://issues.apache.org/jira/browse/HDFS-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448171#comment-13448171
]
Hudson commented on HDFS-3866:
--
Integrated in Hadoop-Common-trunk-Commit #2677 (See
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2677/])
HDFS-3866. HttpFS POM should have property where to download tomcat from
(zero45 via tucu) (Revision 1380927)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1380927
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> HttpFS POM should have property where to download tomcat from
> -
>
> Key: HDFS-3866
> URL: https://issues.apache.org/jira/browse/HDFS-3866
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: build
>Affects Versions: 2.0.0-alpha
> Environment: CDH4 build on CentOS 6.2
>Reporter: Ryan Hennig
>Assignee: Plamen Jeliazkov
>Priority: Minor
> Fix For: 2.2.0-alpha
>
> Attachments: HDFS-3866.patch
>
>
> When trying to enable a build of CDH4 in Jenkins, I got a build error due to
> an attempt to download Tomcat from the internet directly instead of via Maven
> and thus our internal Maven repository.
> The problem is due to this line in
> src/hadoop-hdfs-project/hadoop-hdfs-httpfs/target/antrun/build-main.xml:
>src="http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.32/bin/apache-tomcat-6.0.32.tar.gz"/>
> This build.xml is generated from
> src/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml:
> src="http://archive.apache.org/dist/tomcat/tomcat-6/v${tomcat.version}/bin/apache-tomcat-${tomcat.version}.tar.gz";
> dest="downloads/tomcat.tar.gz" verbose="true" skipexisting="true"/>
> Instead of directly downloading from a hardcoded location, the Tomcat
> dependency should be managed by Maven. This would enable the use of a local
> repository for build machines without internet access.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3888) BlockPlacementPolicyDefault code cleanup
[ https://issues.apache.org/jira/browse/HDFS-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3888: Attachment: HDFS-3888.patch The code for computing the maxNodePerRack in chooseTarget() method is putting back because it may also change the value of numOfReplicas. > BlockPlacementPolicyDefault code cleanup > > > Key: HDFS-3888 > URL: https://issues.apache.org/jira/browse/HDFS-3888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Jing Zhao >Priority: Minor > Attachments: HDFS-3888.patch, HDFS-3888.patch > > > BlockPlacementPolicyDefault#LOG should be removed as it hides LOG from the > base class BlockPlacementPolicy. Also, in > BlockPlacementPolicyDefault#chooseTarget method, the logic computing the > maxTargetPerLoc can be made a separate method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3621) Add a main method to HdfsConfiguration, for debug purposes
[ https://issues.apache.org/jira/browse/HDFS-3621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448162#comment-13448162 ] Hadoop QA commented on HDFS-3621: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12543748/HDFS-3621.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3142//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3142//console This message is automatically generated. > Add a main method to HdfsConfiguration, for debug purposes > -- > > Key: HDFS-3621 > URL: https://issues.apache.org/jira/browse/HDFS-3621 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs client >Affects Versions: 2.0.0-alpha >Reporter: Harsh J >Assignee: Plamen Jeliazkov >Priority: Trivial > Labels: newbie > Attachments: HDFS-3621.patch > > > Just like Configuration has a main() func that dumps XML out for debug > purposes, we should have a similar function under the HdfsConfiguration class > that does the same. This is useful in testing out app classpath setups at > times. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3866) HttpFS POM should have property where to download tomcat from
[
https://issues.apache.org/jira/browse/HDFS-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated HDFS-3866:
-
Resolution: Fixed
Fix Version/s: 2.2.0-alpha
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks Plamen. Committed to trunk and branch-2.
> HttpFS POM should have property where to download tomcat from
> -
>
> Key: HDFS-3866
> URL: https://issues.apache.org/jira/browse/HDFS-3866
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: build
>Affects Versions: 2.0.0-alpha
> Environment: CDH4 build on CentOS 6.2
>Reporter: Ryan Hennig
>Assignee: Plamen Jeliazkov
>Priority: Minor
> Fix For: 2.2.0-alpha
>
> Attachments: HDFS-3866.patch
>
>
> When trying to enable a build of CDH4 in Jenkins, I got a build error due to
> an attempt to download Tomcat from the internet directly instead of via Maven
> and thus our internal Maven repository.
> The problem is due to this line in
> src/hadoop-hdfs-project/hadoop-hdfs-httpfs/target/antrun/build-main.xml:
>src="http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.32/bin/apache-tomcat-6.0.32.tar.gz"/>
> This build.xml is generated from
> src/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml:
> src="http://archive.apache.org/dist/tomcat/tomcat-6/v${tomcat.version}/bin/apache-tomcat-${tomcat.version}.tar.gz";
> dest="downloads/tomcat.tar.gz" verbose="true" skipexisting="true"/>
> Instead of directly downloading from a hardcoded location, the Tomcat
> dependency should be managed by Maven. This would enable the use of a local
> repository for build machines without internet access.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3866) HttpFS POM should have property where to download tomcat from
[
https://issues.apache.org/jira/browse/HDFS-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated HDFS-3866:
-
Summary: HttpFS POM should have property where to download tomcat from
(was: HttpFS build should download Tomcat via Maven instead of directly)
changing summary of JIRA to reflect what the patch does.
> HttpFS POM should have property where to download tomcat from
> -
>
> Key: HDFS-3866
> URL: https://issues.apache.org/jira/browse/HDFS-3866
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build
>Affects Versions: 2.0.0-alpha
> Environment: CDH4 build on CentOS 6.2
>Reporter: Ryan Hennig
>Assignee: Plamen Jeliazkov
>Priority: Minor
> Attachments: HDFS-3866.patch
>
>
> When trying to enable a build of CDH4 in Jenkins, I got a build error due to
> an attempt to download Tomcat from the internet directly instead of via Maven
> and thus our internal Maven repository.
> The problem is due to this line in
> src/hadoop-hdfs-project/hadoop-hdfs-httpfs/target/antrun/build-main.xml:
>src="http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.32/bin/apache-tomcat-6.0.32.tar.gz"/>
> This build.xml is generated from
> src/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml:
> src="http://archive.apache.org/dist/tomcat/tomcat-6/v${tomcat.version}/bin/apache-tomcat-${tomcat.version}.tar.gz";
> dest="downloads/tomcat.tar.gz" verbose="true" skipexisting="true"/>
> Instead of directly downloading from a hardcoded location, the Tomcat
> dependency should be managed by Maven. This would enable the use of a local
> repository for build machines without internet access.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3866) HttpFS POM should have property where to download tomcat from
[
https://issues.apache.org/jira/browse/HDFS-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated HDFS-3866:
-
Issue Type: Improvement (was: Bug)
> HttpFS POM should have property where to download tomcat from
> -
>
> Key: HDFS-3866
> URL: https://issues.apache.org/jira/browse/HDFS-3866
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: build
>Affects Versions: 2.0.0-alpha
> Environment: CDH4 build on CentOS 6.2
>Reporter: Ryan Hennig
>Assignee: Plamen Jeliazkov
>Priority: Minor
> Attachments: HDFS-3866.patch
>
>
> When trying to enable a build of CDH4 in Jenkins, I got a build error due to
> an attempt to download Tomcat from the internet directly instead of via Maven
> and thus our internal Maven repository.
> The problem is due to this line in
> src/hadoop-hdfs-project/hadoop-hdfs-httpfs/target/antrun/build-main.xml:
>src="http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.32/bin/apache-tomcat-6.0.32.tar.gz"/>
> This build.xml is generated from
> src/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml:
> src="http://archive.apache.org/dist/tomcat/tomcat-6/v${tomcat.version}/bin/apache-tomcat-${tomcat.version}.tar.gz";
> dest="downloads/tomcat.tar.gz" verbose="true" skipexisting="true"/>
> Instead of directly downloading from a hardcoded location, the Tomcat
> dependency should be managed by Maven. This would enable the use of a local
> repository for build machines without internet access.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3866) HttpFS build should download Tomcat via Maven instead of directly
[
https://issues.apache.org/jira/browse/HDFS-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448144#comment-13448144
]
Alejandro Abdelnur commented on HDFS-3866:
--
+1
> HttpFS build should download Tomcat via Maven instead of directly
> -
>
> Key: HDFS-3866
> URL: https://issues.apache.org/jira/browse/HDFS-3866
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build
>Affects Versions: 2.0.0-alpha
> Environment: CDH4 build on CentOS 6.2
>Reporter: Ryan Hennig
>Assignee: Plamen Jeliazkov
>Priority: Minor
> Attachments: HDFS-3866.patch
>
>
> When trying to enable a build of CDH4 in Jenkins, I got a build error due to
> an attempt to download Tomcat from the internet directly instead of via Maven
> and thus our internal Maven repository.
> The problem is due to this line in
> src/hadoop-hdfs-project/hadoop-hdfs-httpfs/target/antrun/build-main.xml:
>src="http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.32/bin/apache-tomcat-6.0.32.tar.gz"/>
> This build.xml is generated from
> src/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml:
> src="http://archive.apache.org/dist/tomcat/tomcat-6/v${tomcat.version}/bin/apache-tomcat-${tomcat.version}.tar.gz";
> dest="downloads/tomcat.tar.gz" verbose="true" skipexisting="true"/>
> Instead of directly downloading from a hardcoded location, the Tomcat
> dependency should be managed by Maven. This would enable the use of a local
> repository for build machines without internet access.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3888) BlockPlacementPolicyDefault code cleanup
[ https://issues.apache.org/jira/browse/HDFS-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3888: Status: Open (was: Patch Available) > BlockPlacementPolicyDefault code cleanup > > > Key: HDFS-3888 > URL: https://issues.apache.org/jira/browse/HDFS-3888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Jing Zhao >Priority: Minor > Attachments: HDFS-3888.patch > > > BlockPlacementPolicyDefault#LOG should be removed as it hides LOG from the > base class BlockPlacementPolicy. Also, in > BlockPlacementPolicyDefault#chooseTarget method, the logic computing the > maxTargetPerLoc can be made a separate method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3888) BlockPlacementPolicyDefault code cleanup
[ https://issues.apache.org/jira/browse/HDFS-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-3888: -- Summary: BlockPlacementPolicyDefault code cleanup (was: BlockPlacementPolicyDefault#LOG should be removed) > BlockPlacementPolicyDefault code cleanup > > > Key: HDFS-3888 > URL: https://issues.apache.org/jira/browse/HDFS-3888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Jing Zhao >Priority: Minor > Attachments: HDFS-3888.patch > > > BlockPlacementPolicyDefault#LOG should be removed as it hides LOG from the > base class BlockPlacementPolicy. Also, in > BlockPlacementPolicyDefault#chooseTarget method, the logic computing the > maxTargetPerLoc can be made a separate method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3888) BlockPlacementPolicyDefault#LOG should be removed
[ https://issues.apache.org/jira/browse/HDFS-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3888: Attachment: HDFS-3888.patch > BlockPlacementPolicyDefault#LOG should be removed > - > > Key: HDFS-3888 > URL: https://issues.apache.org/jira/browse/HDFS-3888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Jing Zhao >Priority: Minor > Attachments: HDFS-3888.patch > > > BlockPlacementPolicyDefault#LOG should be removed as it hides LOG from the > base class BlockPlacementPolicy. Also, in > BlockPlacementPolicyDefault#chooseTarget method, the logic computing the > maxTargetPerLoc can be made a separate method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3888) BlockPlacementPolicyDefault#LOG should be removed
[ https://issues.apache.org/jira/browse/HDFS-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-3888: Status: Patch Available (was: Open) > BlockPlacementPolicyDefault#LOG should be removed > - > > Key: HDFS-3888 > URL: https://issues.apache.org/jira/browse/HDFS-3888 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Jing Zhao >Priority: Minor > Attachments: HDFS-3888.patch > > > BlockPlacementPolicyDefault#LOG should be removed as it hides LOG from the > base class BlockPlacementPolicy. Also, in > BlockPlacementPolicyDefault#chooseTarget method, the logic computing the > maxTargetPerLoc can be made a separate method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3888) BlockPlacementPolicyDefault#LOG should be removed
Jing Zhao created HDFS-3888: --- Summary: BlockPlacementPolicyDefault#LOG should be removed Key: HDFS-3888 URL: https://issues.apache.org/jira/browse/HDFS-3888 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.0.0 Reporter: Jing Zhao Assignee: Jing Zhao Priority: Minor BlockPlacementPolicyDefault#LOG should be removed as it hides LOG from the base class BlockPlacementPolicy. Also, in BlockPlacementPolicyDefault#chooseTarget method, the logic computing the maxTargetPerLoc can be made a separate method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3866) HttpFS build should download Tomcat via Maven instead of directly
[
https://issues.apache.org/jira/browse/HDFS-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Plamen Jeliazkov updated HDFS-3866:
---
Status: Patch Available (was: Open)
> HttpFS build should download Tomcat via Maven instead of directly
> -
>
> Key: HDFS-3866
> URL: https://issues.apache.org/jira/browse/HDFS-3866
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build
>Affects Versions: 2.0.0-alpha
> Environment: CDH4 build on CentOS 6.2
>Reporter: Ryan Hennig
>Assignee: Plamen Jeliazkov
>Priority: Minor
> Attachments: HDFS-3866.patch
>
>
> When trying to enable a build of CDH4 in Jenkins, I got a build error due to
> an attempt to download Tomcat from the internet directly instead of via Maven
> and thus our internal Maven repository.
> The problem is due to this line in
> src/hadoop-hdfs-project/hadoop-hdfs-httpfs/target/antrun/build-main.xml:
>src="http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.32/bin/apache-tomcat-6.0.32.tar.gz"/>
> This build.xml is generated from
> src/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml:
> src="http://archive.apache.org/dist/tomcat/tomcat-6/v${tomcat.version}/bin/apache-tomcat-${tomcat.version}.tar.gz";
> dest="downloads/tomcat.tar.gz" verbose="true" skipexisting="true"/>
> Instead of directly downloading from a hardcoded location, the Tomcat
> dependency should be managed by Maven. This would enable the use of a local
> repository for build machines without internet access.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3866) HttpFS build should download Tomcat via Maven instead of directly
[
https://issues.apache.org/jira/browse/HDFS-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448106#comment-13448106
]
Plamen Jeliazkov commented on HDFS-3866:
Taking Alejandro's advice here by making it a POM property for easy overriding
with -D or editing.
> HttpFS build should download Tomcat via Maven instead of directly
> -
>
> Key: HDFS-3866
> URL: https://issues.apache.org/jira/browse/HDFS-3866
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build
>Affects Versions: 2.0.0-alpha
> Environment: CDH4 build on CentOS 6.2
>Reporter: Ryan Hennig
>Assignee: Plamen Jeliazkov
>Priority: Minor
> Attachments: HDFS-3866.patch
>
>
> When trying to enable a build of CDH4 in Jenkins, I got a build error due to
> an attempt to download Tomcat from the internet directly instead of via Maven
> and thus our internal Maven repository.
> The problem is due to this line in
> src/hadoop-hdfs-project/hadoop-hdfs-httpfs/target/antrun/build-main.xml:
>src="http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.32/bin/apache-tomcat-6.0.32.tar.gz"/>
> This build.xml is generated from
> src/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml:
> src="http://archive.apache.org/dist/tomcat/tomcat-6/v${tomcat.version}/bin/apache-tomcat-${tomcat.version}.tar.gz";
> dest="downloads/tomcat.tar.gz" verbose="true" skipexisting="true"/>
> Instead of directly downloading from a hardcoded location, the Tomcat
> dependency should be managed by Maven. This would enable the use of a local
> repository for build machines without internet access.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3866) HttpFS build should download Tomcat via Maven instead of directly
[
https://issues.apache.org/jira/browse/HDFS-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Plamen Jeliazkov updated HDFS-3866:
---
Attachment: HDFS-3866.patch
> HttpFS build should download Tomcat via Maven instead of directly
> -
>
> Key: HDFS-3866
> URL: https://issues.apache.org/jira/browse/HDFS-3866
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build
>Affects Versions: 2.0.0-alpha
> Environment: CDH4 build on CentOS 6.2
>Reporter: Ryan Hennig
>Assignee: Plamen Jeliazkov
>Priority: Minor
> Attachments: HDFS-3866.patch
>
>
> When trying to enable a build of CDH4 in Jenkins, I got a build error due to
> an attempt to download Tomcat from the internet directly instead of via Maven
> and thus our internal Maven repository.
> The problem is due to this line in
> src/hadoop-hdfs-project/hadoop-hdfs-httpfs/target/antrun/build-main.xml:
>src="http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.32/bin/apache-tomcat-6.0.32.tar.gz"/>
> This build.xml is generated from
> src/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml:
> src="http://archive.apache.org/dist/tomcat/tomcat-6/v${tomcat.version}/bin/apache-tomcat-${tomcat.version}.tar.gz";
> dest="downloads/tomcat.tar.gz" verbose="true" skipexisting="true"/>
> Instead of directly downloading from a hardcoded location, the Tomcat
> dependency should be managed by Maven. This would enable the use of a local
> repository for build machines without internet access.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-3866) HttpFS build should download Tomcat via Maven instead of directly
[
https://issues.apache.org/jira/browse/HDFS-3866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Plamen Jeliazkov reassigned HDFS-3866:
--
Assignee: Plamen Jeliazkov
> HttpFS build should download Tomcat via Maven instead of directly
> -
>
> Key: HDFS-3866
> URL: https://issues.apache.org/jira/browse/HDFS-3866
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: build
>Affects Versions: 2.0.0-alpha
> Environment: CDH4 build on CentOS 6.2
>Reporter: Ryan Hennig
>Assignee: Plamen Jeliazkov
>Priority: Minor
> Attachments: HDFS-3866.patch
>
>
> When trying to enable a build of CDH4 in Jenkins, I got a build error due to
> an attempt to download Tomcat from the internet directly instead of via Maven
> and thus our internal Maven repository.
> The problem is due to this line in
> src/hadoop-hdfs-project/hadoop-hdfs-httpfs/target/antrun/build-main.xml:
>src="http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.32/bin/apache-tomcat-6.0.32.tar.gz"/>
> This build.xml is generated from
> src/hadoop-hdfs-project/hadoop-hdfs-httpfs/pom.xml:
> src="http://archive.apache.org/dist/tomcat/tomcat-6/v${tomcat.version}/bin/apache-tomcat-${tomcat.version}.tar.gz";
> dest="downloads/tomcat.tar.gz" verbose="true" skipexisting="true"/>
> Instead of directly downloading from a hardcoded location, the Tomcat
> dependency should be managed by Maven. This would enable the use of a local
> repository for build machines without internet access.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2793) Add an admin command to trigger an edit log roll
[ https://issues.apache.org/jira/browse/HDFS-2793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448105#comment-13448105 ] Aaron T. Myers commented on HDFS-2793: -- bq. Is the 2NN's principal always considered a superuser? I think so, but just want to confirm. Pretty certain that's the case, yes. > Add an admin command to trigger an edit log roll > > > Key: HDFS-2793 > URL: https://issues.apache.org/jira/browse/HDFS-2793 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Affects Versions: 0.24.0 >Reporter: Aaron T. Myers >Assignee: Todd Lipcon > Attachments: hdfs-2793.txt > > > This seems like it would also be helpful outside of the context of HA, but > especially so given that the standby can currently only read finalized log > segments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2793) Add an admin command to trigger an edit log roll
[ https://issues.apache.org/jira/browse/HDFS-2793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448099#comment-13448099 ] Todd Lipcon commented on HDFS-2793: --- bq. It's certainly the case that arbitrary clients should be super users, but when would a non-super user node be calling this method? I think it should be safe to always check for super user privileges. Is the 2NN's principal always considered a superuser? I think so, but just want to confirm. bq. Should the rollEdits RPC be marked @Idempotent? I can't think of a case when calling it twice would be problematic. Makes sense. > Add an admin command to trigger an edit log roll > > > Key: HDFS-2793 > URL: https://issues.apache.org/jira/browse/HDFS-2793 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Affects Versions: 0.24.0 >Reporter: Aaron T. Myers >Assignee: Todd Lipcon > Attachments: hdfs-2793.txt > > > This seems like it would also be helpful outside of the context of HA, but > especially so given that the standby can currently only read finalized log > segments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3621) Add a main method to HdfsConfiguration, for debug purposes
[ https://issues.apache.org/jira/browse/HDFS-3621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-3621: --- Status: Patch Available (was: Open) > Add a main method to HdfsConfiguration, for debug purposes > -- > > Key: HDFS-3621 > URL: https://issues.apache.org/jira/browse/HDFS-3621 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs client >Affects Versions: 2.0.0-alpha >Reporter: Harsh J >Assignee: Plamen Jeliazkov >Priority: Trivial > Labels: newbie > Attachments: HDFS-3621.patch > > > Just like Configuration has a main() func that dumps XML out for debug > purposes, we should have a similar function under the HdfsConfiguration class > that does the same. This is useful in testing out app classpath setups at > times. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3621) Add a main method to HdfsConfiguration, for debug purposes
[ https://issues.apache.org/jira/browse/HDFS-3621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov updated HDFS-3621: --- Attachment: HDFS-3621.patch > Add a main method to HdfsConfiguration, for debug purposes > -- > > Key: HDFS-3621 > URL: https://issues.apache.org/jira/browse/HDFS-3621 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs client >Affects Versions: 2.0.0-alpha >Reporter: Harsh J >Assignee: Plamen Jeliazkov >Priority: Trivial > Labels: newbie > Attachments: HDFS-3621.patch > > > Just like Configuration has a main() func that dumps XML out for debug > purposes, we should have a similar function under the HdfsConfiguration class > that does the same. This is useful in testing out app classpath setups at > times. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-3621) Add a main method to HdfsConfiguration, for debug purposes
[ https://issues.apache.org/jira/browse/HDFS-3621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Plamen Jeliazkov reassigned HDFS-3621: -- Assignee: Plamen Jeliazkov (was: Linden Hillenbrand) > Add a main method to HdfsConfiguration, for debug purposes > -- > > Key: HDFS-3621 > URL: https://issues.apache.org/jira/browse/HDFS-3621 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs client >Affects Versions: 2.0.0-alpha >Reporter: Harsh J >Assignee: Plamen Jeliazkov >Priority: Trivial > Labels: newbie > Attachments: HDFS-3621.patch > > > Just like Configuration has a main() func that dumps XML out for debug > purposes, we should have a similar function under the HdfsConfiguration class > that does the same. This is useful in testing out app classpath setups at > times. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3383) libhdfs does not build on ARM because jni_md.h is not found
[ https://issues.apache.org/jira/browse/HDFS-3383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448078#comment-13448078 ] Trevor Robinson commented on HDFS-3383: --- Can this be committed to branch-1, since it has not been changed to use CMake? > libhdfs does not build on ARM because jni_md.h is not found > --- > > Key: HDFS-3383 > URL: https://issues.apache.org/jira/browse/HDFS-3383 > Project: Hadoop HDFS > Issue Type: Bug > Components: libhdfs >Affects Versions: 0.23.1 > Environment: Linux 3.2.0-1412-omap4 #16-Ubuntu SMP PREEMPT Tue Apr 17 > 19:38:42 UTC 2012 armv7l armv7l armv7l GNU/Linux > java version "1.7.0_04-ea" > Java(TM) SE Runtime Environment for Embedded (build 1.7.0_04-ea-b20, headless) > Java HotSpot(TM) Embedded Server VM (build 23.0-b21, mixed mode, experimental) >Reporter: Trevor Robinson > Attachments: HDFS-3383.patch > > > The wrong include directory is used for jni_md.h: > [INFO] --- make-maven-plugin:1.0-beta-1:make-install (compile) @ hadoop-hdfs > --- > [INFO] /bin/bash ./libtool --tag=CC --mode=compile gcc > -DPACKAGE_NAME=\"libhdfs\" -DPACKAGE_TARNAME=\"libhdfs\" > -DPACKAGE_VERSION=\"0.1.0\" -DPACKAGE_STRING=\"libhdfs\ 0.1.0\" > -DPACKAGE_BUGREPORT=\"omal...@apache.org\" -DPACKAGE_URL=\"\" > -DPACKAGE=\"libhdfs\" -DVERSION=\"0.1.0\" -DSTDC_HEADERS=1 > -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 > -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 > -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=\".libs/\" -DHAVE_STRDUP=1 > -DHAVE_STRERROR=1 -DHAVE_STRTOUL=1 -DHAVE_FCNTL_H=1 -DHAVE__BOOL=1 > -DHAVE_STDBOOL_H=1 -I. -g -O2 -DOS_LINUX -DDSO_DLFCN -DCPU=\"arm\" > -I/usr/lib/jvm/ejdk1.7.0_04/include -I/usr/lib/jvm/ejdk1.7.0_04/include/arm > -Wall -Wstrict-prototypes -MT hdfs.lo -MD -MP -MF .deps/hdfs.Tpo -c -o > hdfs.lo hdfs.c > [INFO] libtool: compile: gcc -DPACKAGE_NAME=\"libhdfs\" > -DPACKAGE_TARNAME=\"libhdfs\" -DPACKAGE_VERSION=\"0.1.0\" > "-DPACKAGE_STRING=\"libhdfs 0.1.0\"" > -DPACKAGE_BUGREPORT=\"omal...@apache.org\" -DPACKAGE_URL=\"\" > -DPACKAGE=\"libhdfs\" -DVERSION=\"0.1.0\" -DSTDC_HEADERS=1 > -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 > -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 > -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=\".libs/\" -DHAVE_STRDUP=1 > -DHAVE_STRERROR=1 -DHAVE_STRTOUL=1 -DHAVE_FCNTL_H=1 -DHAVE__BOOL=1 > -DHAVE_STDBOOL_H=1 -I. -g -O2 -DOS_LINUX -DDSO_DLFCN -DCPU=\"arm\" > -I/usr/lib/jvm/ejdk1.7.0_04/include -I/usr/lib/jvm/ejdk1.7.0_04/include/arm > -Wall -Wstrict-prototypes -MT hdfs.lo -MD -MP -MF .deps/hdfs.Tpo -c hdfs.c > -fPIC -DPIC -o .libs/hdfs.o > [INFO] In file included from hdfs.h:33:0, > [INFO] from hdfs.c:19: > [INFO] /usr/lib/jvm/ejdk1.7.0_04/include/jni.h:45:20: fatal error: jni_md.h: > No such file or directory > [INFO] compilation terminated. > [INFO] make: *** [hdfs.lo] Error 1 > The problem is caused by > hadoop-hdfs-project/hadoop-hdfs/src/main/native/m4/apsupport.m4 overriding > supported_os=arm when host_cpu=arm*; supported_os should remain "linux", > since it determines the jni_md.h include path. OpenJDK 6 and 7 (in Ubuntu > 12.04, at least) and Oracle EJDK put jni_md.h in include/linux. Not sure > if/why this ever worked before. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2793) Add an admin command to trigger an edit log roll
[
https://issues.apache.org/jira/browse/HDFS-2793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448076#comment-13448076
]
Aaron T. Myers commented on HDFS-2793:
--
Patch looks pretty good to me. Just a few little comments:
# I don't understand why this "if(...)" is necessary:
{code}
if (isFromClient) {
checkSuperuserPrivilege();
}
{code}
It's certainly the case that arbitrary clients should be super users, but when
would a non-super user node be calling this method? I think it should be safe
to always check for super user privileges.
# Should the rollEdits RPC be marked {{@Idempotent}}? I can't think of a case
when calling it twice would be problematic.
> Add an admin command to trigger an edit log roll
>
>
> Key: HDFS-2793
> URL: https://issues.apache.org/jira/browse/HDFS-2793
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: name-node
>Affects Versions: 0.24.0
>Reporter: Aaron T. Myers
>Assignee: Todd Lipcon
> Attachments: hdfs-2793.txt
>
>
> This seems like it would also be helpful outside of the context of HA, but
> especially so given that the standby can currently only read finalized log
> segments.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3077) Quorum-based protocol for reading and writing edit logs
[ https://issues.apache.org/jira/browse/HDFS-3077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13447950#comment-13447950 ] Suresh Srinivas commented on HDFS-3077: --- Hey Todd, I have not looked at the work in this branch in a while. One thing I wanted to ask you about is, why are we using journal daemons to decide on an epoch? Could zookeeper be used for doing the same? What are the advantages of using journal daemons instead of zk? Adding this information to the document might also be useful. > Quorum-based protocol for reading and writing edit logs > --- > > Key: HDFS-3077 > URL: https://issues.apache.org/jira/browse/HDFS-3077 > Project: Hadoop HDFS > Issue Type: New Feature > Components: ha, name-node >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: QuorumJournalManager (HDFS-3077) > > Attachments: hdfs-3077-partial.txt, hdfs-3077.txt, hdfs-3077.txt, > hdfs-3077.txt, hdfs-3077.txt, hdfs-3077.txt, hdfs-3077.txt, hdfs-3077.txt, > qjournal-design.pdf, qjournal-design.pdf > > > Currently, one of the weak points of the HA design is that it relies on > shared storage such as an NFS filer for the shared edit log. One alternative > that has been proposed is to depend on BookKeeper, a ZooKeeper subproject > which provides a highly available replicated edit log on commodity hardware. > This JIRA is to implement another alternative, based on a quorum commit > protocol, integrated more tightly in HDFS and with the requirements driven > only by HDFS's needs rather than more generic use cases. More details to > follow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3540) Further improvement on recovery mode and edit log toleration in branch-1
[
https://issues.apache.org/jira/browse/HDFS-3540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13447906#comment-13447906
]
Colin Patrick McCabe commented on HDFS-3540:
bq. Correct me if I am wrong: Recovery Mode without HDFS-3479 means the entire
end-of-log is not checked and, therefore, the silent data loss length is not
limited. It is even worst.
No, that is incorrect.
Recovery mode has always read up to the end of the log, and it always will.
The confusion arises because sometimes we are not very good at determining
where the "end of the log" is.
I filed and implemented HDFS-3479 because I noticed that in certain scenarios
we would decide that the edit log ended before it really did because we spotted
an {{OP_INVALID}}.
The unchecked region which we've been discussing only applied to HDFS-3479
corruption, not to any other type of corruption. Frankly, the unchecked region
was a mistake.
However, none of this has *anything* to do with recovery mode. HDFS-3004 and
HDFS-3479 were separate JIRAs, that implemented separate features.
> Further improvement on recovery mode and edit log toleration in branch-1
>
>
> Key: HDFS-3540
> URL: https://issues.apache.org/jira/browse/HDFS-3540
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 1.2.0
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
>
> *Recovery Mode*: HDFS-3479 backported HDFS-3335 to branch-1. However, the
> recovery mode feature in branch-1 is dramatically different from the recovery
> mode in trunk since the edit log implementations in these two branch are
> different. For example, there is UNCHECKED_REGION_LENGTH in branch-1 but not
> in trunk.
> *Edit Log Toleration*: HDFS-3521 added this feature to branch-1 to remedy
> UNCHECKED_REGION_LENGTH and to tolerate edit log corruption.
> There are overlaps between these two features. We study potential further
> improvement in this issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3703) Decrease the datanode failure detection time
[ https://issues.apache.org/jira/browse/HDFS-3703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13447878#comment-13447878 ] nkeywal commented on HDFS-3703: --- Hi Jing, Suresh, I'm currently testing the patch with HBase trunk. I rebased your patch on hdfs branch-2, as HBase does not yet build with hdfs trunk. I will keep you updated. > Decrease the datanode failure detection time > > > Key: HDFS-3703 > URL: https://issues.apache.org/jira/browse/HDFS-3703 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node, name-node >Affects Versions: 1.0.3, 2.0.0-alpha >Reporter: nkeywal >Assignee: Suresh Srinivas > Attachments: HDFS-3703.patch > > > By default, if a box dies, the datanode will be marked as dead by the > namenode after 10:30 minutes. In the meantime, this datanode will still be > proposed by the nanenode to write blocks or to read replicas. It happens as > well if the datanode crashes: there is no shutdown hooks to tell the nanemode > we're not there anymore. > It especially an issue with HBase. HBase regionserver timeout for production > is often 30s. So with these configs, when a box dies HBase starts to recover > after 30s and, while 10 minutes, the namenode will consider the blocks on the > same box as available. Beyond the write errors, this will trigger a lot of > missed reads: > - during the recovery, HBase needs to read the blocks used on the dead box > (the ones in the 'HBase Write-Ahead-Log') > - after the recovery, reading these data blocks (the 'HBase region') will > fail 33% of the time with the default number of replica, slowering the data > access, especially when the errors are socket timeout (i.e. around 60s most > of the time). > Globally, it would be ideal if HDFS settings could be under HBase settings. > As a side note, HBase relies on ZooKeeper to detect regionservers issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3886) Shutdown requests can possibly check for checkpoint issues (corrupted edits) and save a good namespace copy before closing down?
[ https://issues.apache.org/jira/browse/HDFS-3886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13447866#comment-13447866 ] Aaron T. Myers commented on HDFS-3886: -- bq. I don't think you could easily do much with init.d as that is initiated by the OS when it's doing a shutdown and it may be unrolling large parts of the system: fast shutdowns are always appreciated before the monitoring layers escalate. I wasn't suggesting we modify the existing behavior of `/etc/init.d/* stop', but rather that we add an extra, optional command along the lines of `/etc/init.d/* clean-stop'. This would indeed make an RPC to the NN to enter safemode, perform a save namespace, and then shut itself down. This wouldn't affect the behavior of an OS shutdown, since that would still just use the 'stop' command. Does that make sense? > Shutdown requests can possibly check for checkpoint issues (corrupted edits) > and save a good namespace copy before closing down? > > > Key: HDFS-3886 > URL: https://issues.apache.org/jira/browse/HDFS-3886 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 2.0.0-alpha >Reporter: Harsh J >Priority: Minor > > HDFS-3878 sorta gives me this idea. Aside of having a method to download it > to a different location, we can also lock up the namesystem (or deactivate > the client rpc server) and save the namesystem before we complete up the > shutdown. > The init.d/shutdown scripts would have to work with this somehow though, to > not kill -9 it when in-process. Also, the new image may be stored in a > shutdown.chkpt directory, to not interfere in the regular dirs, but still > allow easier recovery. > Obviously this will still not work if all directories are broken. So maybe we > could have some configs to tackle that as well? > I haven't thought this through, so let me know what part is wrong to do :) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3884) QJM: Journal format() should reset cached values
[ https://issues.apache.org/jira/browse/HDFS-3884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13447853#comment-13447853 ] Aaron T. Myers commented on HDFS-3884: -- +1 > QJM: Journal format() should reset cached values > > > Key: HDFS-3884 > URL: https://issues.apache.org/jira/browse/HDFS-3884 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: QuorumJournalManager (HDFS-3077) >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: QuorumJournalManager (HDFS-3077) > > Attachments: hdfs-3884.txt, hdfs-3884.txt > > > Simple bug in the JournalNode: it caches certain values (eg accepted epoch) > in memory, and the cached values aren't reset when the journal is formatted. > So, after a format, further calls to the same Journal will see the old value > for accepted epoch, writer epoch, etc, preventing the journal from being > re-used until the JN is restarted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3876) NN should not RPC to self to find trash defaults (causes deadlock)
[
https://issues.apache.org/jira/browse/HDFS-3876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13447830#comment-13447830
]
Todd Lipcon commented on HDFS-3876:
---
- When you clobber the trash interval in the configuration, you sohuld do it on
a copy, rather than modifying the config that the user passed in.
{code}
+ // If we can not determine that trash is enabled server side then
+ // bail rather than potentially deleting a file when trash is enabled.
+ System.err.println("Failed to determine server trash configuration: "
+ + e.getMessage());
+ return false;
{code}
This doesn't seem to be what happens. See the TODO in {{Delete.java}}:
{code}
// TODO: if the user wants the trash to be used but there is any
// problem (ie. creating the trash dir, moving the item to be deleted,
// etc), then the path will just be deleted because moveToTrash returns
// false and it falls thru to fs.delete. this doesn't seem right
{code}
if you actually want it to bail, it should probably throw an exception -- or
return false but remove this comment, and separately address the TODO mentioned
above.
{code}
+Configuration clientConf = new Configuration(serverConf);
+if (clientTrash) {
+ clientConf.setLong(FS_TRASH_INTERVAL_KEY, 200);
+}
{code}
Since you instantiate the client conf from {{serverConf}}, won't you end up
with client trash enabled even if {{clientTrash}} is false?
> NN should not RPC to self to find trash defaults (causes deadlock)
> --
>
> Key: HDFS-3876
> URL: https://issues.apache.org/jira/browse/HDFS-3876
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 2.2.0-alpha
>Reporter: Todd Lipcon
>Assignee: Eli Collins
>Priority: Blocker
> Attachments: hdfs-3876.txt, hdfs-3876.txt, hdfs-3876.txt
>
>
> When transitioning a SBN to active, I ran into the following situation:
> - the TrashPolicy first gets loaded by an IPC Server Handler thread. The
> {{initialize}} function then tries to make an RPC to the same node to find
> out the defaults.
> - This is happening inside the NN write lock (since it's part of the active
> initialization). Hence, all of the other handler threads are already blocked
> waiting to get the NN lock.
> - Since no handler threads are free, the RPC blocks forever and the NN never
> enters active state.
> We need to have a general policy that the NN should never make RPCs to itself
> for any reason, due to potential for deadlocks like this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1490) TransferFSImage should timeout
[ https://issues.apache.org/jira/browse/HDFS-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13447737#comment-13447737 ] Hadoop QA commented on HDFS-1490: - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12543666/HDFS-1490.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3141//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3141//console This message is automatically generated. > TransferFSImage should timeout > -- > > Key: HDFS-1490 > URL: https://issues.apache.org/jira/browse/HDFS-1490 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Reporter: Dmytro Molkov >Assignee: Dmytro Molkov >Priority: Minor > Attachments: HDFS-1490.patch, HDFS-1490.patch, HDFS-1490.patch, > HDFS-1490.patch > > > Sometimes when primary crashes during image transfer secondary namenode would > hang trying to read the image from HTTP connection forever. > It would be great to set timeouts on the connection so if something like that > happens there is no need to restart the secondary itself. > In our case restarting components is handled by the set of scripts and since > the Secondary as the process is running it would just stay hung until we get > an alarm saying the checkpointing doesn't happen. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1490) TransferFSImage should timeout
[ https://issues.apache.org/jira/browse/HDFS-1490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay updated HDFS-1490: Attachment: HDFS-1490.patch Fixed typo. Added @VisibleForTesting, since 'timeout' is used in test. We have tested this in cluster, when the active nn's n/w broken. GetImage call got timeout. > TransferFSImage should timeout > -- > > Key: HDFS-1490 > URL: https://issues.apache.org/jira/browse/HDFS-1490 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Reporter: Dmytro Molkov >Assignee: Dmytro Molkov >Priority: Minor > Attachments: HDFS-1490.patch, HDFS-1490.patch, HDFS-1490.patch, > HDFS-1490.patch > > > Sometimes when primary crashes during image transfer secondary namenode would > hang trying to read the image from HTTP connection forever. > It would be great to set timeouts on the connection so if something like that > happens there is no need to restart the secondary itself. > In our case restarting components is handled by the set of scripts and since > the Secondary as the process is running it would just stay hung until we get > an alarm saying the checkpointing doesn't happen. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3887) Remove redundant chooseTarget methods in BlockPlacementPolicy.java
[ https://issues.apache.org/jira/browse/HDFS-3887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13447521#comment-13447521 ] Jing Zhao commented on HDFS-3887: - Have rerun the two failed tests in my local machine and both tests passed. > Remove redundant chooseTarget methods in BlockPlacementPolicy.java > -- > > Key: HDFS-3887 > URL: https://issues.apache.org/jira/browse/HDFS-3887 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.0.0 >Reporter: Jing Zhao >Assignee: Jing Zhao >Priority: Trivial > Attachments: HDFS-3887.patch > > > BlockPlacementPolicy.java contains multiple chooseTarget() methods with > different parameter lists. It is difficult to follow and understand the code > since some chooseTarget methods only have minor differences and some of them > are only invoked by testing code. > In this patch, I try to remove some of the chooseTarget methods and only keep > three of them: two abstract methods and the third one using BlockCollection > as its parameter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2793) Add an admin command to trigger an edit log roll
[ https://issues.apache.org/jira/browse/HDFS-2793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13447518#comment-13447518 ] Hadoop QA commented on HDFS-2793: - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12543618/hdfs-2793.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3140//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3140//console This message is automatically generated. > Add an admin command to trigger an edit log roll > > > Key: HDFS-2793 > URL: https://issues.apache.org/jira/browse/HDFS-2793 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Affects Versions: 0.24.0 >Reporter: Aaron T. Myers >Assignee: Todd Lipcon > Attachments: hdfs-2793.txt > > > This seems like it would also be helpful outside of the context of HA, but > especially so given that the standby can currently only read finalized log > segments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
