I have seen several instances of the Yarn nodemanager shutting down due to this
exception:
2016-02-16 18:04:24,510 WARN impl.MetricsSystemImpl
(MetricsSystemImpl.java:run(373)) - java.util.ConcurrentModificationException
2016-02-16 18:04:34,509 WARN impl.MetricsSystemImpl
(MetricsSystemImpl.java:run(373)) - java.util.ConcurrentModificationException
2016-02-16 18:04:44,510 WARN impl.MetricsSystemImpl
(MetricsSystemImpl.java:run(373)) - java.util.ConcurrentModificationException
2016-02-16 18:04:59,424 FATAL yarn.YarnUncaughtExceptionHandler
(YarnUncaughtExceptionHandler.java:uncaughtException(51)) - Thread
Thread[timeline,5,main] threw an Error. Shutting down now...
java.lang.OutOfMemoryError: Java heap space
at java.util.HashMap.resize(HashMap.java:703)
at java.util.HashMap.putVal(HashMap.java:662)
at java.util.HashMap.put(HashMap.java:611)
at
org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache$TimelineMetricHolder.put(TimelineMetricsCache.java:123)
at
org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache.putTimelineMetric(TimelineMetricsCache.java:154)
at
org.apache.hadoop.metrics2.sink.timeline.cache.TimelineMetricsCache.putTimelineMetric(TimelineMetricsCache.java:177)
at
org.apache.hadoop.metrics2.sink.timeline.HadoopTimelineMetricsSink.putMetrics(HadoopTimelineMetricsSink.java:195)
at
org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:186)
at
org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.consume(MetricsSinkAdapter.java:43)
at
org.apache.hadoop.metrics2.impl.SinkQueue.consumeAll(SinkQueue.java:87)
at
org.apache.hadoop.metrics2.impl.MetricsSinkAdapter.publishMetricsFromQueue(MetricsSinkAdapter.java:134)
at
org.apache.hadoop.metrics2.impl.MetricsSinkAdapter$1.run(MetricsSinkAdapter.java:88)
2016-02-16 18:04:59,433 INFO util.ExitUtil (ExitUtil.java:halt(147)) - Halt
with status -1 Message: HaltException
This code is located in the ambari-metrics project. Has anyone seen this
before or have advice on how to proceed? Running with HDP-2.3.0.0-2557, Ambari
Metrics 0.1.0, YARN 2.7.1.2.3. Thanks
-Adam L
________________________________
IMPORTANT: The information contained in this email and/or its attachments is
confidential. If you are not the intended recipient, please notify the sender
immediately by reply and immediately delete this message and all its
attachments. Any review, use, reproduction, disclosure or dissemination of this
message or any attachment by an unintended recipient is strictly prohibited.
Neither this message nor any attachment is intended as or should be construed
as an offer, solicitation or recommendation to buy or sell any security or
other financial instrument. Neither the sender, his or her employer nor any of
their respective affiliates makes any warranties as to the completeness or
accuracy of any of the information contained herein or that this message or any
of its attachments is free of viruses.
