[
https://issues.apache.org/jira/browse/MAPREDUCE-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13399383#comment-13399383
]
Ravi Prakash commented on MAPREDUCE-4300:
-----------------------------------------
Sorry Bobby! I meant to write sooner.
- I wonder if there is a fault injection framework which we can use to make
sure that an OOM in every thread is handled properly.
- The other option ofcourse is to have 1 thread constantly monitor the headroom
we have in memory, and change the behavior of the daemon when there isn't
enough. That way we don't need to depend on every thread to do the right thing
(using try-catch or your proposed framework). Just another option (if we're
brain-storming)
> OOM in AM can turn it into a zombie.
> ------------------------------------
>
> Key: MAPREDUCE-4300
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4300
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
> Affects Versions: 0.23.3
> Reporter: Robert Joseph Evans
>
> It looks like 4 threads in the AM died with OOM but not the one pinging the
> RM.
> stderr for this AM
> {noformat}
> WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use
> org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
> May 30, 2012 4:49:55 AM
> com.google.inject.servlet.InternalServletModule$BackwardsCompatibleServletContextProvider
> get
> WARNING: You are attempting to use a deprecated API (specifically, attempting
> to @Inject ServletContext inside an eagerly created singleton. While we allow
> this for backwards compatibility, be warned that this MAY have unexpected
> behavior if you have more than one injector (with ServletModule) running in
> the same JVM. Please consult the Guice documentation at
> http://code.google.com/p/google-guice/wiki/Servlets for more information.
> May 30, 2012 4:49:55 AM
> com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
> INFO: Registering
> org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver as a provider
> class
> May 30, 2012 4:49:55 AM
> com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
> INFO: Registering org.apache.hadoop.yarn.webapp.GenericExceptionHandler as a
> provider class
> May 30, 2012 4:49:55 AM
> com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
> INFO: Registering org.apache.hadoop.mapreduce.v2.app.webapp.AMWebServices as
> a root resource class
> May 30, 2012 4:49:55 AM
> com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
> INFO: Initiating Jersey application, version 'Jersey: 1.8 06/24/2011 12:17 PM'
> May 30, 2012 4:49:55 AM
> com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
> getComponentProvider
> INFO: Binding org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver
> to GuiceManagedComponentProvider with the scope "Singleton"
> May 30, 2012 4:49:56 AM
> com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
> getComponentProvider
> INFO: Binding org.apache.hadoop.yarn.webapp.GenericExceptionHandler to
> GuiceManagedComponentProvider with the scope "Singleton"
> May 30, 2012 4:49:56 AM
> com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
> getComponentProvider
> INFO: Binding org.apache.hadoop.mapreduce.v2.app.webapp.AMWebServices to
> GuiceManagedComponentProvider with the scope "PerRequest"
> Exception in thread "ResponseProcessor for block
> BP-1114822160-<IP>-1322528669066:blk_-6528896407411719649_34227308"
> java.lang.OutOfMemoryError: Java heap space
> at com.google.protobuf.CodedInputStream.(CodedInputStream.java:538)
> at
> com.google.protobuf.CodedInputStream.newInstance(CodedInputStream.java:55)
> at
> com.google.protobuf.AbstractMessageLite$Builder.mergeFrom(AbstractMessageLite.java:201)
> at
> com.google.protobuf.AbstractMessage$Builder.mergeFrom(AbstractMessage.java:738)
> at
> org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos$PipelineAckProto.parseFrom(DataTransferProtos.java:7287)
> at
> org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:95)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:656)
> Exception in thread "DefaultSpeculator background processing"
> java.lang.OutOfMemoryError: Java heap space
> at java.util.HashMap.resize(HashMap.java:462)
> at java.util.HashMap.addEntry(HashMap.java:755)
> at java.util.HashMap.put(HashMap.java:385)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.getTasks(JobImpl.java:632)
> at
> org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator.maybeScheduleASpeculation(DefaultSpeculator.java:465)
> at
> org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator.maybeScheduleAMapSpeculation(DefaultSpeculator.java:433)
> at
> org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator.computeSpeculations(DefaultSpeculator.java:509)
> at
> org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator.access$100(DefaultSpeculator.java:56)
> at
> org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator$1.run(DefaultSpeculator.java:176)
> at java.lang.Thread.run(Thread.java:619)
> Exception in thread "Timer for 'MRAppMaster' metrics system"
> java.lang.OutOfMemoryError: Java heap space
> Exception in thread "Socket Reader #4 for port 50500"
> java.lang.OutOfMemoryError: Java heap space
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
