[
https://issues.apache.org/jira/browse/MAPREDUCE-2389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13007689#comment-13007689
]
Todd Lipcon commented on MAPREDUCE-2389:
----------------------------------------
Improved my systemtap script to include calls to dup2
[12909] Wed Mar 16 20:59:27 2011: open
"/data/4/scratch/todd/haus-mrlocal/taskTracker/todd/jobcache/job_201103161358_0001/attempt_201103161358_0001_m_000339_0/output/file.out.index",
O_RDONLY 163
[12909] Wed Mar 16 20:59:27 2011 - dup2 66 ->
/data/4/scratch/todd/haus-mrlocal/taskTracker/todd/jobcache/job_201103161358_0001/attempt_201103161358_0001_m_000339_0/output/file.out.index
163
corresponds almost exactly to:
2011-03-16 13:59:27,994 WARN org.apache.hadoop.mapred.TaskTracker:
getMapOutput(attempt_201103161358_0001_m_000339_0,22) failed :
java.io.IOException: Invalid request Map Id =
attempt_201103161358_0001_m_000339_0 Reducer = 22 Index Info Length = 0
So almost certainly due to a JVM bug with NIO using dup2 in a racy way that
overwrites an open fd!
> Spurious EOFExceptions reading SpillRecord index files
> ------------------------------------------------------
>
> Key: MAPREDUCE-2389
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2389
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Affects Versions: 0.22.0
> Environment: Seen on RHEL 5.5, RHEL 6.0, local dirs on ext3, Java
> 6u20 and 6u24
> Reporter: Todd Lipcon
> Priority: Critical
> Attachments: stap-output.txt
>
>
> In large jobs, I see around 1 shuffle fetch out of every million fetches fail
> with an EOFException reading the SpillRecord index file. After lots of
> investigation, including systemtap, it looks like the read() syscall is
> actually returning a premature "0" result for no reason, so this is likely a
> kernel or filesystem bug which is exacerbated by some workload the TT does.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
