Hi, I'm using Hadoop v1.2.1 and it is running fine so long(3 months) without any issues.
Suddenly I got the below error in Jobtracker and jobs are failed to run. Is this issue in JT or TT or Jetty issue? 2014-04-12 02:13:57,618 ERROR org.mortbay.log: EXCEPTION java.lang.OutOfMemoryError: Java heap space at java.util.HashMap.newKeyIterator(HashMap.java:840) at java.util.HashMap$KeySet.iterator(HashMap.java:874) at java.util.HashSet.iterator(HashSet.java:153) at sun.nio.ch.SelectorImpl.processDeregisterQueue(SelectorImpl.java:127) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:69) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:498) at org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192) at org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124) at org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) 2014-04-12 02:26:02,093 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:root cause:java.io.IOException: java.lang.OutOfMemoryError: Java heap space 2014-04-12 02:25:47,963 ERROR org.mortbay.log: Error for /jobtracker.jsp java.lang.OutOfMemoryError: Java heap space at java.lang.String.toCharArray(String.java:2725) at sun.net.www.ParseUtil.encodePath(ParseUtil.java:86) at sun.misc.URLClassPath$JarLoader.checkResource(URLClassPath.java:674) at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:759) at sun.misc.URLClassPath.getResource(URLClassPath.java:169) at java.net.URLClassLoader$1.run(URLClassLoader.java:194) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) at org.apache.hadoop.mapred.jobtracker_jsp._jspService(jobtracker_jsp.java:508) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:914) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) 2014-04-12 02:25:47,963 ERROR org.apache.hadoop.mapred.JobHistory: Unable to move history file to DONE canonical subfolder. java.lang.OutOfMemoryError: Java heap space Please help as I'm facing in production. -- Regards, Viswa.J