[jira] [Updated] (HDFS-4739) NN can miscalculate the number of extra edit log segments to retain
[ https://issues.apache.org/jira/browse/HDFS-4739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-4739: - Resolution: Fixed Fix Version/s: 2.0.5-beta Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I've just committed this to trunk and branch-2. Thanks again for the reviews, Todd and Chris. > NN can miscalculate the number of extra edit log segments to retain > --- > > Key: HDFS-4739 > URL: https://issues.apache.org/jira/browse/HDFS-4739 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.0.4-alpha >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Fix For: 2.0.5-beta > > Attachments: HDFS-4739.patch, HDFS-4739.patch > > > The code in NNStorageRetentionManager#purgeOldStorage is intended to place a > cap on the number of _extra_ edit log segments retained beyond what is > strictly required to replay the FS history since the last fsimage. In fact > this code currently places a limit on the _total_ number of extra edit log > segments. If the number of required segments is greater than the configured > cap, there will be no data loss, but an ugly error will be thrown and the NN > will fail to start. > The fix is simple, and in the meantime a work-around is just to raise the > value of dfs.namenode.max.extra.edits.segments.retained and start the NN. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4739) NN can miscalculate the number of extra edit log segments to retain
[ https://issues.apache.org/jira/browse/HDFS-4739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-4739: - Attachment: HDFS-4739.patch Missed a bounds check. Updated patch should fix the test failures. > NN can miscalculate the number of extra edit log segments to retain > --- > > Key: HDFS-4739 > URL: https://issues.apache.org/jira/browse/HDFS-4739 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.0.4-alpha >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: HDFS-4739.patch, HDFS-4739.patch > > > The code in NNStorageRetentionManager#purgeOldStorage is intended to place a > cap on the number of _extra_ edit log segments retained beyond what is > strictly required to replay the FS history since the last fsimage. In fact > this code currently places a limit on the _total_ number of extra edit log > segments. If the number of required segments is greater than the configured > cap, there will be no data loss, but an ugly error will be thrown and the NN > will fail to start. > The fix is simple, and in the meantime a work-around is just to raise the > value of dfs.namenode.max.extra.edits.segments.retained and start the NN. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4739) NN can miscalculate the number of extra edit log segments to retain
[ https://issues.apache.org/jira/browse/HDFS-4739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-4739: - Attachment: HDFS-4739.patch Attached patch fixes the issue by: # Discounting truly required segments from the calculation of which segments to purge. # Fixing an OBO error which made the original test case appear to work properly. > NN can miscalculate the number of extra edit log segments to retain > --- > > Key: HDFS-4739 > URL: https://issues.apache.org/jira/browse/HDFS-4739 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.0.4-alpha >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: HDFS-4739.patch > > > The code in NNStorageRetentionManager#purgeOldStorage is intended to place a > cap on the number of _extra_ edit log segments retained beyond what is > strictly required to replay the FS history since the last fsimage. In fact > this code currently places a limit on the _total_ number of extra edit log > segments. If the number of required segments is greater than the configured > cap, there will be no data loss, but an ugly error will be thrown and the NN > will fail to start. > The fix is simple, and in the meantime a work-around is just to raise the > value of dfs.namenode.max.extra.edits.segments.retained and start the NN. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4739) NN can miscalculate the number of extra edit log segments to retain
[ https://issues.apache.org/jira/browse/HDFS-4739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-4739: - Status: Patch Available (was: Open) > NN can miscalculate the number of extra edit log segments to retain > --- > > Key: HDFS-4739 > URL: https://issues.apache.org/jira/browse/HDFS-4739 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.0.4-alpha >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: HDFS-4739.patch > > > The code in NNStorageRetentionManager#purgeOldStorage is intended to place a > cap on the number of _extra_ edit log segments retained beyond what is > strictly required to replay the FS history since the last fsimage. In fact > this code currently places a limit on the _total_ number of extra edit log > segments. If the number of required segments is greater than the configured > cap, there will be no data loss, but an ugly error will be thrown and the NN > will fail to start. > The fix is simple, and in the meantime a work-around is just to raise the > value of dfs.namenode.max.extra.edits.segments.retained and start the NN. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
