Re: Exception in hadoop jobtracker OOM
unsubscribe 2014-04-12 23:02 GMT+08:00 Viswanathan J jayamviswanat...@gmail.com: Hi, I'm using Hadoop v1.2.1 and it is running fine so long(3 months) without any issues. Suddenly I got the below error in Jobtracker and jobs are failed to run. Is this issue in JT or TT or Jetty issue? 2014-04-12 02:13:57,618 ERROR org.mortbay.log: EXCEPTION java.lang.OutOfMemoryError: Java heap space at java.util.HashMap. newKeyIterator(HashMap.java:840) at java.util.HashMap$KeySet. iterator(HashMap.java:874) at java.util.HashSet.iterator( HashSet.java:153) at sun.nio.ch.SelectorImpl. processDeregisterQueue(SelectorImpl.java:127) at sun.nio.ch.EPollSelectorImpl. doSelect(EPollSelectorImpl.java:69) at sun.nio.ch.SelectorImpl. lockAndDoSelect(SelectorImpl.java:69) at sun.nio.ch.SelectorImpl. select(SelectorImpl.java:80) at org.mortbay.io.nio. SelectorManager$SelectSet. doSelect(SelectorManager.java:498) at org.mortbay.io.nio. SelectorManager.doSelect( SelectorManager.java:192) at org.mortbay.jetty.nio. SelectChannelConnector.accept( SelectChannelConnector.java:124) at org.mortbay.jetty. AbstractConnector$Acceptor. run(AbstractConnector.java:708) at org.mortbay.thread. QueuedThreadPool$PoolThread. run(QueuedThreadPool.java:582) 2014-04-12 02:26:02,093 ERROR org.apache.hadoop.security. UserGroupInformation: PriviledgedActionException as:root cause:java.io.IOException: java.lang.OutOfMemoryError: Java heap space 2014-04-12 02:25:47,963 ERROR org.mortbay.log: Error for /jobtracker.jsp java.lang.OutOfMemoryError: Java heap space at java.lang.String.toCharArray( String.java:2725) at sun.net.www.ParseUtil. encodePath(ParseUtil.java:86) at sun.misc.URLClassPath$ JarLoader.checkResource( URLClassPath.java:674) at sun.misc.URLClassPath$ JarLoader.getResource(URLClassPath.java:759) at sun.misc.URLClassPath. getResource(URLClassPath.java:169) at java.net.URLClassLoader$1.run( URLClassLoader.java:194) at java.security. AccessController.doPrivileged(Native Method) at java.net.URLClassLoader. findClass(URLClassLoader.java:190) at java.lang.ClassLoader. loadClass(ClassLoader.java:306) at sun.misc.Launcher$ AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader. loadClass(ClassLoader.java:247) at org.apache.hadoop.mapred. jobtracker_jsp._jspService( jobtracker_jsp.java:508) at org.apache.jasper.runtime. HttpJspBase.service(HttpJspBase.java:97) at javax.servlet.http.HttpServlet.service( HttpServlet.java:820) at org.mortbay.jetty.servlet. ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet. ServletHandler$CachedChain. doFilter(ServletHandler.java:1221) at org.apache.hadoop.http. HttpServer$QuotingInputFilter. doFilter(HttpServer.java:914) at org.mortbay.jetty.servlet. ServletHandler$CachedChain. doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet. ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security. SecurityHandler.handle( SecurityHandler.java:216) at org.mortbay.jetty.servlet. SessionHandler.handle( SessionHandler.java:182) at org.mortbay.jetty.handler. ContextHandler.handle( ContextHandler.java:766) at org.mortbay.jetty.webapp. WebAppContext.handle( WebAppContext.java:450) at org.mortbay.jetty.handler. ContextHandlerCollection.handle( ContextHandlerCollection.java:230) at org.mortbay.jetty.handler. HandlerWrapper.handle( HandlerWrapper.java:152) at org.mortbay.jetty.Server. handle(Server.java:326) at org.mortbay.jetty. HttpConnection.handleRequest( HttpConnection.java:542) at org.mortbay.jetty. HttpConnection$RequestHandler. headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser. parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser. parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle( HttpConnection.java:404) at org.mortbay.io.nio. SelectChannelEndPoint.run( SelectChannelEndPoint.java:410) 2014-04-12 02:25:47,963 ERROR org.apache.hadoop.mapred.JobHistory: Unable to move history file to DONE canonical subfolder. java.lang.OutOfMemoryError: Java heap space Please help as I'm facing in production.
Re: Exception in hadoop jobtracker OOM
This usually occurs when the task takes more memory and exceeds its heap space. You can increase the memory of the tasks by setting a property in mapred-site.xml Mapred.child.opts specify the -Xmx to a higher value. P.S the property name might not be correct, you may look up the mapred-default.xml for the exact property name. Sent from my phone. Thanks Divye Sheth On Apr 12, 2014 8:33 PM, Viswanathan J jayamviswanat...@gmail.com wrote: Hi, I'm using Hadoop v1.2.1 and it is running fine so long(3 months) without any issues. Suddenly I got the below error in Jobtracker and jobs are failed to run. Is this issue in JT or TT or Jetty issue? 2014-04-12 02:13:57,618 ERROR org.mortbay.log: EXCEPTION java.lang.OutOfMemoryError: Java heap space at java.util.HashMap. newKeyIterator(HashMap.java:840) at java.util.HashMap$KeySet. iterator(HashMap.java:874) at java.util.HashSet.iterator( HashSet.java:153) at sun.nio.ch.SelectorImpl. processDeregisterQueue(SelectorImpl.java:127) at sun.nio.ch.EPollSelectorImpl. doSelect(EPollSelectorImpl.java:69) at sun.nio.ch.SelectorImpl. lockAndDoSelect(SelectorImpl.java:69) at sun.nio.ch.SelectorImpl. select(SelectorImpl.java:80) at org.mortbay.io.nio. SelectorManager$SelectSet. doSelect(SelectorManager.java:498) at org.mortbay.io.nio. SelectorManager.doSelect( SelectorManager.java:192) at org.mortbay.jetty.nio. SelectChannelConnector.accept( SelectChannelConnector.java:124) at org.mortbay.jetty. AbstractConnector$Acceptor. run(AbstractConnector.java:708) at org.mortbay.thread. QueuedThreadPool$PoolThread. run(QueuedThreadPool.java:582) 2014-04-12 02:26:02,093 ERROR org.apache.hadoop.security. UserGroupInformation: PriviledgedActionException as:root cause:java.io.IOException: java.lang.OutOfMemoryError: Java heap space 2014-04-12 02:25:47,963 ERROR org.mortbay.log: Error for /jobtracker.jsp java.lang.OutOfMemoryError: Java heap space at java.lang.String.toCharArray( String.java:2725) at sun.net.www.ParseUtil. encodePath(ParseUtil.java:86) at sun.misc.URLClassPath$ JarLoader.checkResource( URLClassPath.java:674) at sun.misc.URLClassPath$ JarLoader.getResource(URLClassPath.java:759) at sun.misc.URLClassPath. getResource(URLClassPath.java:169) at java.net.URLClassLoader$1.run( URLClassLoader.java:194) at java.security. AccessController.doPrivileged(Native Method) at java.net.URLClassLoader. findClass(URLClassLoader.java:190) at java.lang.ClassLoader. loadClass(ClassLoader.java:306) at sun.misc.Launcher$ AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader. loadClass(ClassLoader.java:247) at org.apache.hadoop.mapred. jobtracker_jsp._jspService( jobtracker_jsp.java:508) at org.apache.jasper.runtime. HttpJspBase.service(HttpJspBase.java:97) at javax.servlet.http.HttpServlet.service( HttpServlet.java:820) at org.mortbay.jetty.servlet. ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet. ServletHandler$CachedChain. doFilter(ServletHandler.java:1221) at org.apache.hadoop.http. HttpServer$QuotingInputFilter. doFilter(HttpServer.java:914) at org.mortbay.jetty.servlet. ServletHandler$CachedChain. doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet. ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security. SecurityHandler.handle( SecurityHandler.java:216) at org.mortbay.jetty.servlet. SessionHandler.handle( SessionHandler.java:182) at org.mortbay.jetty.handler. ContextHandler.handle( ContextHandler.java:766) at org.mortbay.jetty.webapp. WebAppContext.handle( WebAppContext.java:450) at org.mortbay.jetty.handler. ContextHandlerCollection.handle( ContextHandlerCollection.java:230) at org.mortbay.jetty.handler. HandlerWrapper.handle( HandlerWrapper.java:152) at org.mortbay.jetty.Server. handle(Server.java:326) at org.mortbay.jetty. HttpConnection.handleRequest( HttpConnection.java:542) at org.mortbay.jetty. HttpConnection$RequestHandler. headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser. parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser. parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle( HttpConnection.java:404) at org.mortbay.io.nio. SelectChannelEndPoint.run( SelectChannelEndPoint.java:410) 2014-04-12 02:25:47,963 ERROR org.apache.hadoop.mapred.JobHistory: Unable to move history file to DONE canonical subfolder. java.lang.OutOfMemoryError: Java heap space Please help as I'm facing in production.
Re: Exception in hadoop jobtracker OOM
Sorry for the error. Did not have a proper look at the logs. This seems to be a JT issue. Ignore the previous email. Thanks Divye Sheth On Apr 14, 2014 6:06 PM, divye sheth divs.sh...@gmail.com wrote: This usually occurs when the task takes more memory and exceeds its heap space. You can increase the memory of the tasks by setting a property in mapred-site.xml Mapred.child.opts specify the -Xmx to a higher value. P.S the property name might not be correct, you may look up the mapred-default.xml for the exact property name. Sent from my phone. Thanks Divye Sheth On Apr 12, 2014 8:33 PM, Viswanathan J jayamviswanat...@gmail.com wrote: Hi, I'm using Hadoop v1.2.1 and it is running fine so long(3 months) without any issues. Suddenly I got the below error in Jobtracker and jobs are failed to run. Is this issue in JT or TT or Jetty issue? 2014-04-12 02:13:57,618 ERROR org.mortbay.log: EXCEPTION java.lang.OutOfMemoryError: Java heap space at java.util.HashMap. newKeyIterator(HashMap.java:840) at java.util.HashMap$KeySet. iterator(HashMap.java:874) at java.util.HashSet.iterator( HashSet.java:153) at sun.nio.ch.SelectorImpl. processDeregisterQueue(SelectorImpl.java:127) at sun.nio.ch.EPollSelectorImpl. doSelect(EPollSelectorImpl.java:69) at sun.nio.ch.SelectorImpl. lockAndDoSelect(SelectorImpl.java:69) at sun.nio.ch.SelectorImpl. select(SelectorImpl.java:80) at org.mortbay.io.nio. SelectorManager$SelectSet. doSelect(SelectorManager.java:498) at org.mortbay.io.nio. SelectorManager.doSelect( SelectorManager.java:192) at org.mortbay.jetty.nio. SelectChannelConnector.accept( SelectChannelConnector.java:124) at org.mortbay.jetty. AbstractConnector$Acceptor. run(AbstractConnector.java:708) at org.mortbay.thread. QueuedThreadPool$PoolThread. run(QueuedThreadPool.java:582) 2014-04-12 02:26:02,093 ERROR org.apache.hadoop.security. UserGroupInformation: PriviledgedActionException as:root cause:java.io.IOException: java.lang.OutOfMemoryError: Java heap space 2014-04-12 02:25:47,963 ERROR org.mortbay.log: Error for /jobtracker.jsp java.lang.OutOfMemoryError: Java heap space at java.lang.String.toCharArray( String.java:2725) at sun.net.www.ParseUtil. encodePath(ParseUtil.java:86) at sun.misc.URLClassPath$ JarLoader.checkResource( URLClassPath.java:674) at sun.misc.URLClassPath$ JarLoader.getResource(URLClassPath.java:759) at sun.misc.URLClassPath. getResource(URLClassPath.java:169) at java.net.URLClassLoader$1.run( URLClassLoader.java:194) at java.security. AccessController.doPrivileged(Native Method) at java.net.URLClassLoader. findClass(URLClassLoader.java:190) at java.lang.ClassLoader. loadClass(ClassLoader.java:306) at sun.misc.Launcher$ AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader. loadClass(ClassLoader.java:247) at org.apache.hadoop.mapred. jobtracker_jsp._jspService( jobtracker_jsp.java:508) at org.apache.jasper.runtime. HttpJspBase.service(HttpJspBase.java:97) at javax.servlet.http.HttpServlet.service( HttpServlet.java:820) at org.mortbay.jetty.servlet. ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet. ServletHandler$CachedChain. doFilter(ServletHandler.java:1221) at org.apache.hadoop.http. HttpServer$QuotingInputFilter. doFilter(HttpServer.java:914) at org.mortbay.jetty.servlet. ServletHandler$CachedChain. doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet. ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security. SecurityHandler.handle( SecurityHandler.java:216) at org.mortbay.jetty.servlet. SessionHandler.handle( SessionHandler.java:182) at org.mortbay.jetty.handler. ContextHandler.handle( ContextHandler.java:766) at org.mortbay.jetty.webapp. WebAppContext.handle( WebAppContext.java:450) at org.mortbay.jetty.handler. ContextHandlerCollection.handle( ContextHandlerCollection.java:230) at org.mortbay.jetty.handler. HandlerWrapper.handle( HandlerWrapper.java:152) at org.mortbay.jetty.Server. handle(Server.java:326) at org.mortbay.jetty. HttpConnection.handleRequest( HttpConnection.java:542) at org.mortbay.jetty. HttpConnection$RequestHandler. headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser. parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser. parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle( HttpConnection.java:404) at org.mortbay.io.nio. SelectChannelEndPoint.run( SelectChannelEndPoint.java:410) 2014-04-12 02:25:47,963 ERROR org.apache.hadoop.mapred.JobHistory: Unable to move history file to DONE canonical subfolder. java.lang.OutOfMemoryError: Java heap space Please help as I'm facing in production.
Exception in hadoop jobtracker OOM
Hi, I'm using Hadoop v1.2.1 and it is running fine so long(3 months) without any issues. Suddenly I got the below error in Jobtracker and jobs are failed to run. Is this issue in JT or TT or Jetty issue? 2014-04-12 02:13:57,618 ERROR org.mortbay.log: EXCEPTION java.lang.OutOfMemoryError: Java heap space at java.util.HashMap. newKeyIterator(HashMap.java:840) at java.util.HashMap$KeySet. iterator(HashMap.java:874) at java.util.HashSet.iterator( HashSet.java:153) at sun.nio.ch.SelectorImpl. processDeregisterQueue(SelectorImpl.java:127) at sun.nio.ch.EPollSelectorImpl. doSelect(EPollSelectorImpl.java:69) at sun.nio.ch.SelectorImpl. lockAndDoSelect(SelectorImpl.java:69) at sun.nio.ch.SelectorImpl. select(SelectorImpl.java:80) at org.mortbay.io.nio. SelectorManager$SelectSet. doSelect(SelectorManager.java:498) at org.mortbay.io.nio. SelectorManager.doSelect( SelectorManager.java:192) at org.mortbay.jetty.nio. SelectChannelConnector.accept( SelectChannelConnector.java:124) at org.mortbay.jetty. AbstractConnector$Acceptor. run(AbstractConnector.java:708) at org.mortbay.thread. QueuedThreadPool$PoolThread. run(QueuedThreadPool.java:582) 2014-04-12 02:26:02,093 ERROR org.apache.hadoop.security. UserGroupInformation: PriviledgedActionException as:root cause:java.io.IOException: java.lang.OutOfMemoryError: Java heap space 2014-04-12 02:25:47,963 ERROR org.mortbay.log: Error for /jobtracker.jsp java.lang.OutOfMemoryError: Java heap space at java.lang.String.toCharArray( String.java:2725) at sun.net.www.ParseUtil. encodePath(ParseUtil.java:86) at sun.misc.URLClassPath$ JarLoader.checkResource( URLClassPath.java:674) at sun.misc.URLClassPath$ JarLoader.getResource(URLClassPath.java:759) at sun.misc.URLClassPath. getResource(URLClassPath.java:169) at java.net.URLClassLoader$1.run( URLClassLoader.java:194) at java.security. AccessController.doPrivileged(Native Method) at java.net.URLClassLoader. findClass(URLClassLoader.java:190) at java.lang.ClassLoader. loadClass(ClassLoader.java:306) at sun.misc.Launcher$ AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader. loadClass(ClassLoader.java:247) at org.apache.hadoop.mapred. jobtracker_jsp._jspService( jobtracker_jsp.java:508) at org.apache.jasper.runtime. HttpJspBase.service(HttpJspBase.java:97) at javax.servlet.http.HttpServlet.service( HttpServlet.java:820) at org.mortbay.jetty.servlet. ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet. ServletHandler$CachedChain. doFilter(ServletHandler.java:1221) at org.apache.hadoop.http. HttpServer$QuotingInputFilter. doFilter(HttpServer.java:914) at org.mortbay.jetty.servlet. ServletHandler$CachedChain. doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet. ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security. SecurityHandler.handle( SecurityHandler.java:216) at org.mortbay.jetty.servlet. SessionHandler.handle( SessionHandler.java:182) at org.mortbay.jetty.handler. ContextHandler.handle( ContextHandler.java:766) at org.mortbay.jetty.webapp. WebAppContext.handle( WebAppContext.java:450) at org.mortbay.jetty.handler. ContextHandlerCollection.handle( ContextHandlerCollection.java:230) at org.mortbay.jetty.handler. HandlerWrapper.handle( HandlerWrapper.java:152) at org.mortbay.jetty.Server. handle(Server.java:326) at org.mortbay.jetty. HttpConnection.handleRequest( HttpConnection.java:542) at org.mortbay.jetty. HttpConnection$RequestHandler. headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser. parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser. parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle( HttpConnection.java:404) at org.mortbay.io.nio. SelectChannelEndPoint.run( SelectChannelEndPoint.java:410) 2014-04-12 02:25:47,963 ERROR org.apache.hadoop.mapred.JobHistory: Unable to move history file to DONE canonical subfolder. java.lang.OutOfMemoryError: Java heap space Please help as I'm facing in production.
