Yes, the stacktrace in my previous message is from the task tracker. It seems to happen when there is no data locality for the mapper and it needs to get it from some other datanode. The number of failures is the same as the number of rack-local mappers.
> Anything in TaskTracker logs ? > > On Jan 31, 2012, at 10:18 AM, Markus Jelsma wrote: > > In our case, which seems to be the same problem, the web UI does not show > > anything useful except the first line of the stack trace: > > > > 2012-01-03 21:16:27,256 WARN org.apache.hadoop.mapred.TaskLog: Failed to > > retrieve stdout log for task: attempt_201201031651_0008_m_000233_0 > > > > Only the task tracker log shows a full stack trace. This happened on > > 1.0.0 and 0.20.205.0 but not 0.20.203.0. > > > > 2012-01-03 21:16:27,256 WARN org.apache.hadoop.mapred.TaskLog: Failed to > > retrieve stdout log for task: attempt_201201031651_0008_m_000233_0 > > java.io.FileNotFoundException: > > /opt/hadoop/hadoop-0.20.205.0/libexec/../logs/userlogs/job_201201031651_0 > > 008/attempt_201201031651_0008_m_000233_0/log.index (No such file or > > directory) > > at java.io.FileInputStream.open(Native Method) > > at java.io.FileInputStream.(SecureIOUtils.java:102) > > at > > org.apache.hadoop.mapred.TaskLog.getAllLogsFileDetails(TaskLog.java:187) > > at org.apache.hadoop.mapred.TaskLog$Reader.(TaskLogServlet.java:81) > > at > > org.apache.hadoop.mapred.TaskLogServlet.doGet(TaskLogServlet.java:296) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) > > at > > org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) > > at > > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHand > > ler.java:1221) at > > org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer. > > java:835) at > > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHand > > ler.java:1212) at > > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) > > at > > org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:21 > > 6) at > > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) > > at > > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) > > at > > org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) > > at > > org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerC > > ollection.java:230) at > > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) > > at org.mortbay.jetty.Server.handle(Server.java:326) > > at > > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) > > at > > org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnec > > tion.java:928) > > > >> Actually, all that is telling you is that the task failed and the > >> job-client couldn't display the logs. > >> > >> Can you check the JT web-ui and see why the task failed ? > >> > >> If you don't see anything there, you can try see the TaskTracker logs on > >> the node on which the task ran. > >> > >> Arun > >> > >> On Jan 31, 2012, at 3:21 AM, Marcin Cylke wrote: > >>> Hi > >>> > >>> I've upgraded my hadoop cluster to version 1.0.0. The upgrade process > >>> went relatively smoothly but it rendered the cluster inoperable due to > >>> errors in jobtrackers operation: > >>> > >>> # in job output > >>> Error reading task > >>> outputhttp://hadoop4:50060/tasklog?plaintext=true&attemptid=attempt_201 > >>> 20 1311241_0003_m_000004_2&filter=stdout > >>> > >>> # in each of the jobtrackers' logs > >>> WARN org.apache.hadoop.mapred.TaskLog: Failed to retrieve stderr log > >>> for task: attempt_201201311241_0003_r_000000_1 > >>> java.io.FileNotFoundException: > >>> /usr/lib/hadoop-1.0.0/libexec/../logs/userlogs/job_201201311241_0003/at > >>> te mpt_201201311241_0003_r_000000_1/log.index (No such file or > >>> directory) > >>> > >>> at java.io.FileInputStream.open(Native Method) > >>> > >>> These errors seem related to this two problems: > >>> > >>> http://grokbase.com/t/hadoop.apache.org/mapreduce-user/2012/01/error-re > >>> ad > >>> ing-task-output-and-log-filenotfoundexceptions/03mjwctewcnxlgp2jkcrhvs > >>> gep 4e > >>> > >>> https://issues.apache.org/jira/browse/MAPREDUCE-2846 > >>> > >>> But I've looked into the source code and the fix from MAPREDUCE-2846 is > >>> there. Perhaps there is some other reason? > >>> > >>> Regards > >>> Marcin > >> > >> -- > >> Arun C. Murthy > >> Hortonworks Inc. > >> http://hortonworks.com/ > > -- > Arun C. Murthy > Hortonworks Inc. > http://hortonworks.com/
