[jira] Updated: (MAPREDUCE-1538) TrackerDistributedCacheManager can fail because the number of subdirectories reaches system limit
[ https://issues.apache.org/jira/browse/MAPREDUCE-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated MAPREDUCE-1538: Status: Resolved (was: Patch Available) Fix Version/s: 0.22.0 (was: 0.21.0) Resolution: Fixed I just committed this. Thanks Scott! > TrackerDistributedCacheManager can fail because the number of subdirectories > reaches system limit > - > > Key: MAPREDUCE-1538 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1538 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1538-v2.txt, MAPREDUCE-1538.patch > > > TrackerDistributedCacheManager deletes the cached files when the size goes up > to a configured number. > But there is no such limit for the number of subdirectories. Therefore the > number of subdirectories may grow large and exceed system limit. > This will make TT cannot create directory when getLocalCache and fails the > tasks. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-1538) TrackerDistributedCacheManager can fail because the number of subdirectories reaches system limit
[ https://issues.apache.org/jira/browse/MAPREDUCE-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Chen updated MAPREDUCE-1538: -- Status: Patch Available (was: Open) > TrackerDistributedCacheManager can fail because the number of subdirectories > reaches system limit > - > > Key: MAPREDUCE-1538 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1538 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.21.0 > > Attachments: MAPREDUCE-1538-v2.txt, MAPREDUCE-1538.patch > > > TrackerDistributedCacheManager deletes the cached files when the size goes up > to a configured number. > But there is no such limit for the number of subdirectories. Therefore the > number of subdirectories may grow large and exceed system limit. > This will make TT cannot create directory when getLocalCache and fails the > tasks. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-1538) TrackerDistributedCacheManager can fail because the number of subdirectories reaches system limit
[ https://issues.apache.org/jira/browse/MAPREDUCE-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Chen updated MAPREDUCE-1538: -- Status: Open (was: Patch Available) > TrackerDistributedCacheManager can fail because the number of subdirectories > reaches system limit > - > > Key: MAPREDUCE-1538 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1538 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.21.0 > > Attachments: MAPREDUCE-1538-v2.txt, MAPREDUCE-1538.patch > > > TrackerDistributedCacheManager deletes the cached files when the size goes up > to a configured number. > But there is no such limit for the number of subdirectories. Therefore the > number of subdirectories may grow large and exceed system limit. > This will make TT cannot create directory when getLocalCache and fails the > tasks. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-1538) TrackerDistributedCacheManager can fail because the number of subdirectories reaches system limit
[ https://issues.apache.org/jira/browse/MAPREDUCE-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Chen updated MAPREDUCE-1538: -- Status: Patch Available (was: Open) > TrackerDistributedCacheManager can fail because the number of subdirectories > reaches system limit > - > > Key: MAPREDUCE-1538 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1538 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.21.0 > > Attachments: MAPREDUCE-1538-v2.txt, MAPREDUCE-1538.patch > > > TrackerDistributedCacheManager deletes the cached files when the size goes up > to a configured number. > But there is no such limit for the number of subdirectories. Therefore the > number of subdirectories may grow large and exceed system limit. > This will make TT cannot create directory when getLocalCache and fails the > tasks. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-1538) TrackerDistributedCacheManager can fail because the number of subdirectories reaches system limit
[ https://issues.apache.org/jira/browse/MAPREDUCE-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Chen updated MAPREDUCE-1538: -- Attachment: MAPREDUCE-1538-v2.txt > TrackerDistributedCacheManager can fail because the number of subdirectories > reaches system limit > - > > Key: MAPREDUCE-1538 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1538 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.21.0 > > Attachments: MAPREDUCE-1538-v2.txt, MAPREDUCE-1538.patch > > > TrackerDistributedCacheManager deletes the cached files when the size goes up > to a configured number. > But there is no such limit for the number of subdirectories. Therefore the > number of subdirectories may grow large and exceed system limit. > This will make TT cannot create directory when getLocalCache and fails the > tasks. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-1538) TrackerDistributedCacheManager can fail because the number of subdirectories reaches system limit
[ https://issues.apache.org/jira/browse/MAPREDUCE-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Chen updated MAPREDUCE-1538: -- Attachment: (was: MAPREDUCE-1538-v2.txt) > TrackerDistributedCacheManager can fail because the number of subdirectories > reaches system limit > - > > Key: MAPREDUCE-1538 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1538 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.21.0 > > Attachments: MAPREDUCE-1538-v2.txt, MAPREDUCE-1538.patch > > > TrackerDistributedCacheManager deletes the cached files when the size goes up > to a configured number. > But there is no such limit for the number of subdirectories. Therefore the > number of subdirectories may grow large and exceed system limit. > This will make TT cannot create directory when getLocalCache and fails the > tasks. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-1538) TrackerDistributedCacheManager can fail because the number of subdirectories reaches system limit
[ https://issues.apache.org/jira/browse/MAPREDUCE-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Chen updated MAPREDUCE-1538: -- Status: Open (was: Patch Available) > TrackerDistributedCacheManager can fail because the number of subdirectories > reaches system limit > - > > Key: MAPREDUCE-1538 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1538 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.21.0 > > Attachments: MAPREDUCE-1538-v2.txt, MAPREDUCE-1538.patch > > > TrackerDistributedCacheManager deletes the cached files when the size goes up > to a configured number. > But there is no such limit for the number of subdirectories. Therefore the > number of subdirectories may grow large and exceed system limit. > This will make TT cannot create directory when getLocalCache and fails the > tasks. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-1538) TrackerDistributedCacheManager can fail because the number of subdirectories reaches system limit
[ https://issues.apache.org/jira/browse/MAPREDUCE-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Chen updated MAPREDUCE-1538: -- Status: Patch Available (was: Open) > TrackerDistributedCacheManager can fail because the number of subdirectories > reaches system limit > - > > Key: MAPREDUCE-1538 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1538 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.21.0 > > Attachments: MAPREDUCE-1538-v2.txt, MAPREDUCE-1538.patch > > > TrackerDistributedCacheManager deletes the cached files when the size goes up > to a configured number. > But there is no such limit for the number of subdirectories. Therefore the > number of subdirectories may grow large and exceed system limit. > This will make TT cannot create directory when getLocalCache and fails the > tasks. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-1538) TrackerDistributedCacheManager can fail because the number of subdirectories reaches system limit
[ https://issues.apache.org/jira/browse/MAPREDUCE-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Chen updated MAPREDUCE-1538: -- Attachment: (was: MAPREDUCE-1538-v2.txt) > TrackerDistributedCacheManager can fail because the number of subdirectories > reaches system limit > - > > Key: MAPREDUCE-1538 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1538 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.21.0 > > Attachments: MAPREDUCE-1538-v2.txt, MAPREDUCE-1538.patch > > > TrackerDistributedCacheManager deletes the cached files when the size goes up > to a configured number. > But there is no such limit for the number of subdirectories. Therefore the > number of subdirectories may grow large and exceed system limit. > This will make TT cannot create directory when getLocalCache and fails the > tasks. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-1538) TrackerDistributedCacheManager can fail because the number of subdirectories reaches system limit
[ https://issues.apache.org/jira/browse/MAPREDUCE-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Chen updated MAPREDUCE-1538: -- Attachment: MAPREDUCE-1538-v2.txt > TrackerDistributedCacheManager can fail because the number of subdirectories > reaches system limit > - > > Key: MAPREDUCE-1538 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1538 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.21.0 > > Attachments: MAPREDUCE-1538-v2.txt, MAPREDUCE-1538.patch > > > TrackerDistributedCacheManager deletes the cached files when the size goes up > to a configured number. > But there is no such limit for the number of subdirectories. Therefore the > number of subdirectories may grow large and exceed system limit. > This will make TT cannot create directory when getLocalCache and fails the > tasks. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-1538) TrackerDistributedCacheManager can fail because the number of subdirectories reaches system limit
[ https://issues.apache.org/jira/browse/MAPREDUCE-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Chen updated MAPREDUCE-1538: -- Attachment: MAPREDUCE-1538-v2.txt 1. Move reading config to the constructor 2. Use a class to store cache directory properties > TrackerDistributedCacheManager can fail because the number of subdirectories > reaches system limit > - > > Key: MAPREDUCE-1538 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1538 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.21.0 > > Attachments: MAPREDUCE-1538-v2.txt, MAPREDUCE-1538.patch > > > TrackerDistributedCacheManager deletes the cached files when the size goes up > to a configured number. > But there is no such limit for the number of subdirectories. Therefore the > number of subdirectories may grow large and exceed system limit. > This will make TT cannot create directory when getLocalCache and fails the > tasks. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-1538) TrackerDistributedCacheManager can fail because the number of subdirectories reaches system limit
[
https://issues.apache.org/jira/browse/MAPREDUCE-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-1538:
-
Status: Open (was: Patch Available)
One comment:
{noformat}
+ long allowedNumberSubDir = conf.getLong(
+ TTConfig.TT_LOCAL_CACHE_SUBDIRS_LIMIT, DEFAULT_CACHE_SUBDIR_LIMIT);
{noformat}
We should save the variable from the conf rather than read it from the conf
each time.
> TrackerDistributedCacheManager can fail because the number of subdirectories
> reaches system limit
> -
>
> Key: MAPREDUCE-1538
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1538
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
>Affects Versions: 0.22.0
>Reporter: Scott Chen
>Assignee: Scott Chen
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-1538.patch
>
>
> TrackerDistributedCacheManager deletes the cached files when the size goes up
> to a configured number.
> But there is no such limit for the number of subdirectories. Therefore the
> number of subdirectories may grow large and exceed system limit.
> This will make TT cannot create directory when getLocalCache and fails the
> tasks.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-1538) TrackerDistributedCacheManager can fail because the number of subdirectories reaches system limit
[ https://issues.apache.org/jira/browse/MAPREDUCE-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Chen updated MAPREDUCE-1538: -- Status: Patch Available (was: In Progress) > TrackerDistributedCacheManager can fail because the number of subdirectories > reaches system limit > - > > Key: MAPREDUCE-1538 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1538 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.21.0 > > Attachments: MAPREDUCE-1538.patch > > > TrackerDistributedCacheManager deletes the cached files when the size goes up > to a configured number. > But there is no such limit for the number of subdirectories. Therefore the > number of subdirectories may grow large and exceed system limit. > This will make TT cannot create directory when getLocalCache and fails the > tasks. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-1538) TrackerDistributedCacheManager can fail because the number of subdirectories reaches system limit
[ https://issues.apache.org/jira/browse/MAPREDUCE-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Chen updated MAPREDUCE-1538: -- Status: Open (was: Patch Available) > TrackerDistributedCacheManager can fail because the number of subdirectories > reaches system limit > - > > Key: MAPREDUCE-1538 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1538 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.21.0 > > Attachments: MAPREDUCE-1538.patch > > > TrackerDistributedCacheManager deletes the cached files when the size goes up > to a configured number. > But there is no such limit for the number of subdirectories. Therefore the > number of subdirectories may grow large and exceed system limit. > This will make TT cannot create directory when getLocalCache and fails the > tasks. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-1538) TrackerDistributedCacheManager can fail because the number of subdirectories reaches system limit
[ https://issues.apache.org/jira/browse/MAPREDUCE-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated MAPREDUCE-1538: Status: Open (was: Patch Available) resubmit for HadoopQA > TrackerDistributedCacheManager can fail because the number of subdirectories > reaches system limit > - > > Key: MAPREDUCE-1538 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1538 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1538.patch > > > TrackerDistributedCacheManager deletes the cached files when the size goes up > to a configured number. > But there is no such limit for the number of subdirectories. Therefore the > number of subdirectories may grow large and exceed system limit. > This will make TT cannot create directory when getLocalCache and fails the > tasks. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-1538) TrackerDistributedCacheManager can fail because the number of subdirectories reaches system limit
[ https://issues.apache.org/jira/browse/MAPREDUCE-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated MAPREDUCE-1538: Status: Patch Available (was: Open) Fix Version/s: 0.21.0 (was: 0.22.0) > TrackerDistributedCacheManager can fail because the number of subdirectories > reaches system limit > - > > Key: MAPREDUCE-1538 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1538 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.21.0 > > Attachments: MAPREDUCE-1538.patch > > > TrackerDistributedCacheManager deletes the cached files when the size goes up > to a configured number. > But there is no such limit for the number of subdirectories. Therefore the > number of subdirectories may grow large and exceed system limit. > This will make TT cannot create directory when getLocalCache and fails the > tasks. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-1538) TrackerDistributedCacheManager can fail because the number of subdirectories reaches system limit
[ https://issues.apache.org/jira/browse/MAPREDUCE-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Chen updated MAPREDUCE-1538: -- Status: Patch Available (was: Open) > TrackerDistributedCacheManager can fail because the number of subdirectories > reaches system limit > - > > Key: MAPREDUCE-1538 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1538 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1538.patch > > > TrackerDistributedCacheManager deletes the cached files when the size goes up > to a configured number. > But there is no such limit for the number of subdirectories. Therefore the > number of subdirectories may grow large and exceed system limit. > This will make TT cannot create directory when getLocalCache and fails the > tasks. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1538) TrackerDistributedCacheManager can fail because the number of subdirectories reaches system limit
[ https://issues.apache.org/jira/browse/MAPREDUCE-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Chen updated MAPREDUCE-1538: -- Attachment: MAPREDUCE-1538.patch > TrackerDistributedCacheManager can fail because the number of subdirectories > reaches system limit > - > > Key: MAPREDUCE-1538 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1538 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1538.patch > > > TrackerDistributedCacheManager deletes the cached files when the size goes up > to a configured number. > But there is no such limit for the number of subdirectories. Therefore the > number of subdirectories may grow large and exceed system limit. > This will make TT cannot create directory when getLocalCache and fails the > tasks. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
