[
https://issues.apache.org/jira/browse/MAPREDUCE-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14187472#comment-14187472
]
Sangjin Lee commented on MAPREDUCE-6139:
----------------------------------------
I think that call is there by design. The tail queue is an internal buffer and
that code is trying to write out the buffered log events. You'll also see
super.close() right below this for loop.
I think this is triggering the exception (not that it is the cause but that it
triggers the problem):
{noformat}
org.apache.commons.logging.impl.Log4JLogger.warn(Log4JLogger.java:197)
org.apache.hadoop.security.token.Token.getClassForIdentifier(Token.java:118)
{noformat}
I haven't thought much about this, but perhaps another way to solve this is to
modify append() not to use the tail queue if it is in the middle of a shutdown.
> ConcurrentModificationException in ContainerLogAppender
> -------------------------------------------------------
>
> Key: MAPREDUCE-6139
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6139
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Mohammad Kamrul Islam
> Assignee: Mohammad Kamrul Islam
> Priority: Critical
> Attachments: ContainerLogAppender.java
>
>
> Context:
> * Hadoop-2.3.0
> * Using Oozie 4.0.1
> * Pig version 0.11.x
> The job is submitted by Oozie to launch Pig script.
> The following exception traces were found on MR task log:
> In syslog:
> {noformat}
> 2014-10-24 20:37:29,317 WARN [Thread-5]
> org.apache.hadoop.util.ShutdownHookManager: ShutdownHook '' failed,
> java.util.ConcurrentModificationException
> java.util.ConcurrentModificationException
> at
> java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
> at java.util.LinkedList$ListItr.next(LinkedList.java:888)
> at
> org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94)
> at
> org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141)
> at org.apache.log4j.Category.removeAllAppenders(Category.java:891)
> at org.apache.log4j.Hierarchy.shutdown(Hierarchy.java:471)
> at org.apache.log4j.LogManager.shutdown(LogManager.java:267)
> at org.apache.hadoop.mapred.TaskLog.syncLogsShutdown(TaskLog.java:286)
> at org.apache.hadoop.mapred.TaskLog$2.run(TaskLog.java:339)
> at
> org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
> 2014-10-24 20:37:29,395 INFO [main]
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics
> system...
> {noformat}
> in stderr:
> {noformat}
> java.util.ConcurrentModificationException
> at
> java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
> at java.util.LinkedList$ListItr.next(LinkedList.java:888)
> at
> org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94)
> at
> org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141)
> at org.apache.log4j.Category.removeAllAppenders(Category.java:891)
> at
> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:759)
> at
> org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
> at
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
> at
> org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440)
> at org.apache.pig.Main.configureLog4J(Main.java:740)
> at org.apache.pig.Main.run(Main.java:384)
> at org.apache.pig.PigRunner.run(PigRunner.java:49)
> at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:283)
> at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:223)
> at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37)
> at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
