[ https://issues.apache.org/jira/browse/HDFS-4504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Colin Patrick McCabe updated HDFS-4504: --------------------------------------- Attachment: HDFS-4504.010.patch * {{TestDFSClientRetries#testNotYetReplicatedErrors}} was assuming that close didn't actually call complete() on the NameNode when there was an I/O error. I fixed this assumption by adding a complete() method which returned true on the mock NameNode object used in the test. * {{DFSOutputStream#close}} needed to set {{lastException}} to {{null}} after successfully stopping the I/O threads. The reason is that IOExceptions are handled in the throw clause already, so there is no need to rethrow the next time {{DFSOutputStream#close}} is called. Only when {{complete}} fails do we want to keep the exception around. > DFSOutputStream#close doesn't always release resources (such as leases) > ----------------------------------------------------------------------- > > Key: HDFS-4504 > URL: https://issues.apache.org/jira/browse/HDFS-4504 > Project: Hadoop HDFS > Issue Type: Bug > Reporter: Colin Patrick McCabe > Assignee: Colin Patrick McCabe > Attachments: HDFS-4504.001.patch, HDFS-4504.002.patch, > HDFS-4504.007.patch, HDFS-4504.008.patch, HDFS-4504.009.patch, > HDFS-4504.010.patch > > > {{DFSOutputStream#close}} can throw an {{IOException}} in some cases. One > example is if there is a pipeline error and then pipeline recovery fails. > Unfortunately, in this case, some of the resources used by the > {{DFSOutputStream}} are leaked. One particularly important resource is file > leases. > So it's possible for a long-lived HDFS client, such as Flume, to write many > blocks to a file, but then fail to close it. Unfortunately, the > {{LeaseRenewerThread}} inside the client will continue to renew the lease for > the "undead" file. Future attempts to close the file will just rethrow the > previous exception, and no progress can be made by the client. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira