[
https://issues.apache.org/jira/browse/MAPREDUCE-4003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Koji Noguchi updated MAPREDUCE-4003:
------------------------------------
Affects Version/s: 0.20.205.0
1.0.1
This also happens on our 0.20.205 clusters when users' jvm crashes at start up
due to invalid jvm param or JNI crashes. stderr/stdout files exist but
log.index does not.
WebUI shows
{noformat}
HTTP ERROR 410
Problem accessing /tasklog. Reason:
Failed to retrieve stdout log for task:
attempt_201202130706_50000_m_000004_0
{noformat}
TaskTracker log shows
{noformat}
2012-03-15 01:35:18,431 WARN org.apache.hadoop.mapred.TaskLog: Failed to
retrieve stdout log for task: attempt_201202130706_50000_m_000004_0
java.io.FileNotFoundException:
/grid/log/mapred/userlogs/job_201202130706_50000/attempt_201202130706_50000_m_000004_0/log.index
(No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:106)
at
org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:102)
at
org.apache.hadoop.mapred.TaskLog.getAllLogsFileDetails(TaskLog.java:187)
at org.apache.hadoop.mapred.TaskLog$Reader.<init>(TaskLog.java:422)
at
org.apache.hadoop.mapred.TaskLogServlet.printTaskLog(TaskLogServlet.java:81)
at
org.apache.hadoop.mapred.TaskLogServlet.doGet(TaskLogServlet.java:269)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at yjava.servlet.filter.BouncerFilter.doFilter(BouncerFilter.java:411)
at
com.yahoo.hadoop.HadoopBouncerFilter.doFilter(HadoopBouncerFilter.java:64)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2012-03-15 01:35:18,431 WARN org.mortbay.log: /tasklog: java.io.IOException:
Closed
{noformat}
Stdout/Stderr still exist with information that would help the users.
{noformat}
-bash-3.2$ ls -l
/grid/log/mapred/userlogs/job_201202130706_50000/attempt_201202130706_50000_m_000004_0/
total 4
-rw-r----- 1 knoguchi hadoop 137 Mar 14 22:43 stderr
-rw-r----- 1 knoguchi hadoop 0 Mar 14 22:43 stdout
-bash-3.2$ cat
/grid/log/mapred/userlogs/job_201202130706_50000/attempt_201202130706_50000_m_000004_0/stderr
Invalid maximum heap size: -Xmx10g
The specified size exceeds the maximum representable size.
Could not create the Java virtual machine.
-bash-3.2$
{noformat}
I thought MAPREDUCE-2366 fixed this but I guess it didn't cover this case.
Fixing this would save users' time as well as support(my) time.
> log.index (No such file or directory) AND Task process exit with nonzero
> status of 126
> --------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4003
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4003
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: task-controller, tasktracker
> Affects Versions: 0.20.205.0, 1.0.1
> Environment: hadoop version -------Hadoop 0.20.2-cdh3u3
> uname -a: Linux xxxx 2.6.18-194.17.4.0.1.el5PAE #1 SMP Tue Oct 26 20:15:18
> EDT 2010 i686 i686 i386 GNU/Linux
> core-site.xml:<configuration>
> <property>
> <name>fs.default.name</name>
> <value>hdfs://xxxxx:8020</value>
> </property>
> <property>
> <name>hadoop.tmp.dir</name>
> <value>/home/hadoop/tmp20/</value>
> </property>
> </configuration>
> mapred-site.xml:
> <configuration>
> <property>
> <name>mapred.job.tracker</name>
> <value>192.168.9.60:9001</value>
> </property>
> <property> <name>mapred.local.dir</name>
> <value>/var/tmp/mapred/local</value> </property>
> <property> <name>mapred.system.dir</name> <value>/mapred/system</value>
> </property>
> </configuration>
> Reporter: toughman
>
> hello，I have dwelled on this hadoop(cdhu3) problem for 2 days,I have tried
> every google method.This is the issue: when ran hadoop example "wordcount"
> ,the tasktracker's log in one slave node presented such errors
> 1.WARN org.apache.hadoop.mapred.DefaultTaskController: Task wrapper stderr:
> bash:
> /var/tmp/mapred/local/ttprivate/taskTracker/hdfs/jobcache/job_201203131751_0003/attempt_201203131751_0003_m_000006_0/taskjvm.sh:
> Permission denied
> 2.WARN org.apache.hadoop.mapred.TaskRunner:
> attempt_201203131751_0003_m_000006_0 : Child Error java.io.IOException: Task
> process exit with nonzero status of 126.
> 3.WARN org.apache.hadoop.mapred.TaskLog: Failed to retrieve stdout log for
> task: attempt_201203131751_0003_m_000003_0 java.io.FileNotFoundException:
> /usr/lib/hadoop-0.20/logs/userlogs/job_201203131751_0003/attempt_201203131751_0003_m_000003_0/log.index
> (No such file or directory)
> I could not find similar issues in google,just got some posts seem a little
> relevant ,which suggest: A. the ulimit of hadoop user----but my ulimit is set
> large enough for this bundled example;B. the memory used by jvm,but my jvm
> only use Xmx200m,too small to exceed the limit of my machine ;C.the privilege
> of the mapred.local.dir and logs dir----I set them by "chmod 777";D .the disk
> space is full----there are enough space for hadoop in my log directory and
> mapred.local.dir.
> Thanks for you all,I am really at my wit's end,I have spend days on it. I
> really appreciate any light!
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
