[
https://issues.apache.org/jira/browse/MAPREDUCE-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14187723#comment-14187723
]
Sangjin Lee commented on MAPREDUCE-6139:
----------------------------------------
Just off the top of my head, how about an approach where if the close() (or
shutdown) is in progress append() would simply call super.append() instead of
playing with the tail queue? Perhaps a boolean flag to indicate that a shutdown
is progress...
That might be better than blocking (?) append().
> ConcurrentModificationException in ContainerLogAppender
> -------------------------------------------------------
>
> Key: MAPREDUCE-6139
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6139
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Mohammad Kamrul Islam
> Assignee: Mohammad Kamrul Islam
> Priority: Critical
> Attachments: ContainerLogAppender.java
>
>
> Context:
> * Hadoop-2.3.0
> * Using Oozie 4.0.1
> * Pig version 0.11.x
> The job is submitted by Oozie to launch Pig script.
> The following exception traces were found on MR task log:
> In syslog:
> {noformat}
> 2014-10-24 20:37:29,317 WARN [Thread-5]
> org.apache.hadoop.util.ShutdownHookManager: ShutdownHook '' failed,
> java.util.ConcurrentModificationException
> java.util.ConcurrentModificationException
> at
> java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
> at java.util.LinkedList$ListItr.next(LinkedList.java:888)
> at
> org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94)
> at
> org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141)
> at org.apache.log4j.Category.removeAllAppenders(Category.java:891)
> at org.apache.log4j.Hierarchy.shutdown(Hierarchy.java:471)
> at org.apache.log4j.LogManager.shutdown(LogManager.java:267)
> at org.apache.hadoop.mapred.TaskLog.syncLogsShutdown(TaskLog.java:286)
> at org.apache.hadoop.mapred.TaskLog$2.run(TaskLog.java:339)
> at
> org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)
> 2014-10-24 20:37:29,395 INFO [main]
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics
> system...
> {noformat}
> in stderr:
> {noformat}
> java.util.ConcurrentModificationException
> at
> java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966)
> at java.util.LinkedList$ListItr.next(LinkedList.java:888)
> at
> org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94)
> at
> org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141)
> at org.apache.log4j.Category.removeAllAppenders(Category.java:891)
> at
> org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:759)
> at
> org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648)
> at
> org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514)
> at
> org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440)
> at org.apache.pig.Main.configureLog4J(Main.java:740)
> at org.apache.pig.Main.run(Main.java:384)
> at org.apache.pig.PigRunner.run(PigRunner.java:49)
> at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:283)
> at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:223)
> at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37)
> at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
