[
https://issues.apache.org/jira/browse/MAPREDUCE-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12839129#action_12839129
]
Scott Chen commented on MAPREDUCE-1538:
---------------------------------------
When this happens, the log will actually show the follwing.
2010-02-25 12:45:41,022 INFO org.apache.hadoop.mapred.TaskInProgress: Error
from attempt_201002230013_22452_m_003831_0 on
tracker_hadoop0143.snc3.facebook.com.:localhost.localdomain/127.0.0.1:37489:
java.io.FileNotFoundException:
/mnt/d3/SILVER/local/taskTracker/jobcache/job_201002230013_22452/attempt_201002230013_22452_m_003831_0/output/file.out
(No space left on device)
But if we do df on the machine, we found the space is not an issue. It is
because of the number of subdirectories are too high.
> TrackerDistributedCacheManager can fail because the number of subdirectories
> reaches system limit
> -------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1538
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1538
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
>
> TrackerDistributedCacheManager deletes the cached files when the size goes up
> to a configured number.
> But there is no such limit for the number of subdirectories. Therefore the
> number of subdirectories may grow large and exceed system limit.
> This will make TT cannot create directory when getLocalCache and fails the
> tasks.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
