[ https://issues.apache.org/jira/browse/MAPREDUCE-1914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dick King updated MAPREDUCE-1914: --------------------------------- Attachment: MAPREDUCE-1914--2010-07-30--1336.patch This patch adds two advantages to the code in MAPREDUCE-1538: * it tests the deletion of the directories * It improve that loop of potentially 10K iterations that takes place with a lock held [ see MAPREDUCE-1909 ] * reordering the {{if}} clause in the loop * using {{cachedArchives.values()}} The full regression test is in progress. I don't expect any problems, since I've done the relevant unit tests. > TrackerDistributedCacheManager never cleans its input directories > ----------------------------------------------------------------- > > Key: MAPREDUCE-1914 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1914 > Project: Hadoop Map/Reduce > Issue Type: Bug > Reporter: Dick King > Assignee: Dick King > Attachments: MAPREDUCE-1914--2010-07-30--1336.patch > > > When we localize a file into a node's cache, it's installed in a directory > whose subroot is a random {{long}} . These {{long}} s all sit in a single > flat directory [per disk, per cluster node]. When the cached file is no > longer needed, its reference count becomes zero in a tracking data structure. > The file then becomes eligible for deletion when the total amount of space > occupied by cached files exceeds 10G [by default] or the total number of such > files exceeds 10K. > However, when we delete a cached file, we don't delete the directory that > contains it; this importantly includes the elements of the flat directory, > which then accumulate until they reach a system limit, 32K in some cases, and > then the node stops working. > We need to delete the flat directory when we delete the localized cache file > it contains. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.