[jira] [Updated] (HDFS-5099) Namenode#copyEditLogSegmentsToSharedDir should close EditLogInputStreams upon finishing
[ https://issues.apache.org/jira/browse/HDFS-5099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HDFS-5099: Resolution: Fixed Fix Version/s: 2.1.1-beta 3.0.0 Status: Resolved (was: Patch Available) I've committed this to trunk, branch-2, and branch-2.1-beta. Thank you to Chuan for contributing this fix. > Namenode#copyEditLogSegmentsToSharedDir should close EditLogInputStreams upon > finishing > --- > > Key: HDFS-5099 > URL: https://issues.apache.org/jira/browse/HDFS-5099 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 3.0.0, 2.1.1-beta >Reporter: Chuan Liu >Assignee: Chuan Liu > Fix For: 3.0.0, 2.1.1-beta > > Attachments: HDFS-5099-trunk.patch > > > In {{Namenode#copyEditLogSegmentsToSharedDir()}} method, we open a collection > of EditLogInputStreams to read and apply to shareEditlog. In {{readOpt()}} > method, we will open the underlying log file on disk. After applying all the > opts, we do not close the collection of streams currently. This lead to a > file handle leak on Windows as later we would fail to delete those files. > This happens in TestInitializeSharedEdits test case, where we explicitly > called {{Namenode# initializeSharedEdits()}}, where > {{copyEditLogSegmentsToSharedDir()}} is used. Later we fail to create new > MiniDFSCluster with the following exception. > {noformat} > java.io.IOException: Could not fully delete > C:\hdc\hadoop-hdfs-project\hadoop-hdfs\target\test\data\dfs\name1 > at > org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:759) > at > org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:644) > at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:334) > at > org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:316) > at > org.apache.hadoop.hdfs.server.namenode.ha.TestInitializeSharedEdits.setupCluster(TestInitializeSharedEdits.java:68) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > … > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5099) Namenode#copyEditLogSegmentsToSharedDir should close EditLogInputStreams upon finishing
[ https://issues.apache.org/jira/browse/HDFS-5099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HDFS-5099: Component/s: namenode Target Version/s: 3.0.0, 2.1.1-beta Affects Version/s: (was: 2.3.0) 2.1.1-beta Hadoop Flags: Reviewed +1 for the patch. Nice find, Chuan! This is applicable to branch-2.1-beta too, so I'll commit and merge it down to there. > Namenode#copyEditLogSegmentsToSharedDir should close EditLogInputStreams upon > finishing > --- > > Key: HDFS-5099 > URL: https://issues.apache.org/jira/browse/HDFS-5099 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 3.0.0, 2.1.1-beta >Reporter: Chuan Liu >Assignee: Chuan Liu > Attachments: HDFS-5099-trunk.patch > > > In {{Namenode#copyEditLogSegmentsToSharedDir()}} method, we open a collection > of EditLogInputStreams to read and apply to shareEditlog. In {{readOpt()}} > method, we will open the underlying log file on disk. After applying all the > opts, we do not close the collection of streams currently. This lead to a > file handle leak on Windows as later we would fail to delete those files. > This happens in TestInitializeSharedEdits test case, where we explicitly > called {{Namenode# initializeSharedEdits()}}, where > {{copyEditLogSegmentsToSharedDir()}} is used. Later we fail to create new > MiniDFSCluster with the following exception. > {noformat} > java.io.IOException: Could not fully delete > C:\hdc\hadoop-hdfs-project\hadoop-hdfs\target\test\data\dfs\name1 > at > org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:759) > at > org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:644) > at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:334) > at > org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:316) > at > org.apache.hadoop.hdfs.server.namenode.ha.TestInitializeSharedEdits.setupCluster(TestInitializeSharedEdits.java:68) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > … > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5099) Namenode#copyEditLogSegmentsToSharedDir should close EditLogInputStreams upon finishing
[ https://issues.apache.org/jira/browse/HDFS-5099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chuan Liu updated HDFS-5099: Attachment: HDFS-5099-trunk.patch Attaching the patch that closes all the streams in the "finally" clause. All other code changes are just indentation for the new "try" clause. > Namenode#copyEditLogSegmentsToSharedDir should close EditLogInputStreams upon > finishing > --- > > Key: HDFS-5099 > URL: https://issues.apache.org/jira/browse/HDFS-5099 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0, 2.3.0 >Reporter: Chuan Liu >Assignee: Chuan Liu > Attachments: HDFS-5099-trunk.patch > > > In {{Namenode#copyEditLogSegmentsToSharedDir()}} method, we open a collection > of EditLogInputStreams to read and apply to shareEditlog. In {{readOpt()}} > method, we will open the underlying log file on disk. After applying all the > opts, we do not close the collection of streams currently. This lead to a > file handle leak on Windows as later we would fail to delete those files. > This happens in TestInitializeSharedEdits test case, where we explicitly > called {{Namenode# initializeSharedEdits()}}, where > {{copyEditLogSegmentsToSharedDir()}} is used. Later we fail to create new > MiniDFSCluster with the following exception. > {noformat} > java.io.IOException: Could not fully delete > C:\hdc\hadoop-hdfs-project\hadoop-hdfs\target\test\data\dfs\name1 > at > org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:759) > at > org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:644) > at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:334) > at > org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:316) > at > org.apache.hadoop.hdfs.server.namenode.ha.TestInitializeSharedEdits.setupCluster(TestInitializeSharedEdits.java:68) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > … > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-5099) Namenode#copyEditLogSegmentsToSharedDir should close EditLogInputStreams upon finishing
[ https://issues.apache.org/jira/browse/HDFS-5099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chuan Liu updated HDFS-5099: Status: Patch Available (was: Open) > Namenode#copyEditLogSegmentsToSharedDir should close EditLogInputStreams upon > finishing > --- > > Key: HDFS-5099 > URL: https://issues.apache.org/jira/browse/HDFS-5099 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0, 2.3.0 >Reporter: Chuan Liu >Assignee: Chuan Liu > Attachments: HDFS-5099-trunk.patch > > > In {{Namenode#copyEditLogSegmentsToSharedDir()}} method, we open a collection > of EditLogInputStreams to read and apply to shareEditlog. In {{readOpt()}} > method, we will open the underlying log file on disk. After applying all the > opts, we do not close the collection of streams currently. This lead to a > file handle leak on Windows as later we would fail to delete those files. > This happens in TestInitializeSharedEdits test case, where we explicitly > called {{Namenode# initializeSharedEdits()}}, where > {{copyEditLogSegmentsToSharedDir()}} is used. Later we fail to create new > MiniDFSCluster with the following exception. > {noformat} > java.io.IOException: Could not fully delete > C:\hdc\hadoop-hdfs-project\hadoop-hdfs\target\test\data\dfs\name1 > at > org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:759) > at > org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:644) > at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:334) > at > org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:316) > at > org.apache.hadoop.hdfs.server.namenode.ha.TestInitializeSharedEdits.setupCluster(TestInitializeSharedEdits.java:68) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > … > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira