[ https://issues.apache.org/jira/browse/HADOOP-984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-984: --------------------------------- Status: Patch Available (was: Open) > Exception while retrieving a map output index file is not reported to > JobTracker > -------------------------------------------------------------------------------- > > Key: HADOOP-984 > URL: https://issues.apache.org/jira/browse/HADOOP-984 > Project: Hadoop > Issue Type: Bug > Components: mapred > Affects Versions: 0.11.0 > Reporter: Nigel Daley > Assigned To: Arun C Murthy > Priority: Critical > Fix For: 0.11.1 > > Attachments: HADOOP-984_20070207_1.patch > > > The index file of a map output reported a checksum error and was moved to > "bad_file" dir. Reduces, however, continued to attempt retrieval of the file > forever. It seem the checksum exception did *NOT* result in the map's > TaskTracker reporting to the JobTracker that the map output was lost. This > was likely introduced by HADOOP-331. > TaskTracker log: > 2007-02-06 09:06:19,200 WARN org.apache.hadoop.dfs.DistributedFileSystem: > Moving bad file > /foo/testbase/tmp/mapred/local2/task_0002_m_011745_0/file.out.index to > /foo/bad_files/file.out.index.-48633003 > 2007-02-06 09:06:19,203 INFO org.apache.hadoop.fs.DataInputStream: Found > checksum error: org.apache.hadoop.fs.ChecksumException: Checksum error: > /foo/testbase/tmp/mapred/local2/task_0002_m_011745_0/file.out.index at 363792 > at > org.apache.hadoop.fs.FSDataInputStream$Checker.verifySum(FSDataInputStream.java:167) > at > org.apache.hadoop.fs.FSDataInputStream$Checker.read(FSDataInputStream.java:125) > at > org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInputStream.java:218) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) > at java.io.BufferedInputStream.read(BufferedInputStream.java:235) > at > org.apache.hadoop.fs.FSDataInputStream$Buffer.read(FSDataInputStream.java:281) > at > org.apache.hadoop.fs.FSDataInputStream$Buffer.seek(FSDataInputStream.java:268) > at > org.apache.hadoop.fs.FSDataInputStream.seek(FSDataInputStream.java:331) > at > org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:1533) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:689) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) > at > org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427) > at > org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:475) > at > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:567) > at org.mortbay.http.HttpContext.handle(HttpContext.java:1565) > at > org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:635) > at org.mortbay.http.HttpContext.handle(HttpContext.java:1517) > at org.mortbay.http.HttpServer.service(HttpServer.java:954) > at org.mortbay.http.HttpConnection.service(HttpConnection.java:814) > at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981) > at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831) > at > org.mortbay.http.SocketListener.handleConnection(SocketListener.java:244) > at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357) > at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.