[jira] [Commented] (MAPREDUCE-6139) ConcurrentModificationException in ContainerLogAppender
[ https://issues.apache.org/jira/browse/MAPREDUCE-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14210024#comment-14210024 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-6139: This belongs to YARN. Moving.. ConcurrentModificationException in ContainerLogAppender --- Key: MAPREDUCE-6139 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6139 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Mohammad Kamrul Islam Assignee: Mohammad Kamrul Islam Priority: Critical Attachments: ContainerLogAppender.java, MAPREDUCE-6139-test.01.patch, MAPREDUCE-6139.1.patch, MAPREDUCE-6139.2.patch, MAPREDUCE-6139.3.patch Context: * Hadoop-2.3.0 * Using Oozie 4.0.1 * Pig version 0.11.x The job is submitted by Oozie to launch Pig script. The following exception traces were found on MR task log: In syslog: {noformat} 2014-10-24 20:37:29,317 WARN [Thread-5] org.apache.hadoop.util.ShutdownHookManager: ShutdownHook '' failed, java.util.ConcurrentModificationException java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.Hierarchy.shutdown(Hierarchy.java:471) at org.apache.log4j.LogManager.shutdown(LogManager.java:267) at org.apache.hadoop.mapred.TaskLog.syncLogsShutdown(TaskLog.java:286) at org.apache.hadoop.mapred.TaskLog$2.run(TaskLog.java:339) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) 2014-10-24 20:37:29,395 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system... {noformat} in stderr: {noformat} java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:759) at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514) at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440) at org.apache.pig.Main.configureLog4J(Main.java:740) at org.apache.pig.Main.run(Main.java:384) at org.apache.pig.PigRunner.run(PigRunner.java:49) at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:283) at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:223) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37) at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6139) ConcurrentModificationException in ContainerLogAppender
[ https://issues.apache.org/jira/browse/MAPREDUCE-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14208785#comment-14208785 ] Hadoop QA commented on MAPREDUCE-6139: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12681144/MAPREDUCE-6139-test.01.patch against trunk revision 782abbb. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common: org.apache.hadoop.yarn.TestContainerLogAppender {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5014//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5014//console This message is automatically generated. ConcurrentModificationException in ContainerLogAppender --- Key: MAPREDUCE-6139 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6139 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Mohammad Kamrul Islam Assignee: Mohammad Kamrul Islam Priority: Critical Attachments: ContainerLogAppender.java, MAPREDUCE-6139-test.01.patch, MAPREDUCE-6139.1.patch, MAPREDUCE-6139.2.patch, MAPREDUCE-6139.3.patch Context: * Hadoop-2.3.0 * Using Oozie 4.0.1 * Pig version 0.11.x The job is submitted by Oozie to launch Pig script. The following exception traces were found on MR task log: In syslog: {noformat} 2014-10-24 20:37:29,317 WARN [Thread-5] org.apache.hadoop.util.ShutdownHookManager: ShutdownHook '' failed, java.util.ConcurrentModificationException java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.Hierarchy.shutdown(Hierarchy.java:471) at org.apache.log4j.LogManager.shutdown(LogManager.java:267) at org.apache.hadoop.mapred.TaskLog.syncLogsShutdown(TaskLog.java:286) at org.apache.hadoop.mapred.TaskLog$2.run(TaskLog.java:339) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) 2014-10-24 20:37:29,395 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system... {noformat} in stderr: {noformat} java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:759) at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514) at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440) at org.apache.pig.Main.configureLog4J(Main.java:740) at org.apache.pig.Main.run(Main.java:384) at org.apache.pig.PigRunner.run(PigRunner.java:49) at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:283) at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:223) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37) at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[jira] [Commented] (MAPREDUCE-6139) ConcurrentModificationException in ContainerLogAppender
[ https://issues.apache.org/jira/browse/MAPREDUCE-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14209062#comment-14209062 ] Mohammad Kamrul Islam commented on MAPREDUCE-6139: -- Thanks [~jira.shegalov] for the test class. I will incorporate that in my next patch. ConcurrentModificationException in ContainerLogAppender --- Key: MAPREDUCE-6139 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6139 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Mohammad Kamrul Islam Assignee: Mohammad Kamrul Islam Priority: Critical Attachments: ContainerLogAppender.java, MAPREDUCE-6139-test.01.patch, MAPREDUCE-6139.1.patch, MAPREDUCE-6139.2.patch, MAPREDUCE-6139.3.patch Context: * Hadoop-2.3.0 * Using Oozie 4.0.1 * Pig version 0.11.x The job is submitted by Oozie to launch Pig script. The following exception traces were found on MR task log: In syslog: {noformat} 2014-10-24 20:37:29,317 WARN [Thread-5] org.apache.hadoop.util.ShutdownHookManager: ShutdownHook '' failed, java.util.ConcurrentModificationException java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.Hierarchy.shutdown(Hierarchy.java:471) at org.apache.log4j.LogManager.shutdown(LogManager.java:267) at org.apache.hadoop.mapred.TaskLog.syncLogsShutdown(TaskLog.java:286) at org.apache.hadoop.mapred.TaskLog$2.run(TaskLog.java:339) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) 2014-10-24 20:37:29,395 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system... {noformat} in stderr: {noformat} java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:759) at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514) at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440) at org.apache.pig.Main.configureLog4J(Main.java:740) at org.apache.pig.Main.run(Main.java:384) at org.apache.pig.PigRunner.run(PigRunner.java:49) at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:283) at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:223) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37) at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6139) ConcurrentModificationException in ContainerLogAppender
[ https://issues.apache.org/jira/browse/MAPREDUCE-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14207018#comment-14207018 ] Hadoop QA commented on MAPREDUCE-6139: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12680864/MAPREDUCE-6139.3.patch against trunk revision 061bc29. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5007//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5007//console This message is automatically generated. ConcurrentModificationException in ContainerLogAppender --- Key: MAPREDUCE-6139 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6139 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Mohammad Kamrul Islam Assignee: Mohammad Kamrul Islam Priority: Critical Attachments: ContainerLogAppender.java, MAPREDUCE-6139.1.patch, MAPREDUCE-6139.2.patch, MAPREDUCE-6139.3.patch Context: * Hadoop-2.3.0 * Using Oozie 4.0.1 * Pig version 0.11.x The job is submitted by Oozie to launch Pig script. The following exception traces were found on MR task log: In syslog: {noformat} 2014-10-24 20:37:29,317 WARN [Thread-5] org.apache.hadoop.util.ShutdownHookManager: ShutdownHook '' failed, java.util.ConcurrentModificationException java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.Hierarchy.shutdown(Hierarchy.java:471) at org.apache.log4j.LogManager.shutdown(LogManager.java:267) at org.apache.hadoop.mapred.TaskLog.syncLogsShutdown(TaskLog.java:286) at org.apache.hadoop.mapred.TaskLog$2.run(TaskLog.java:339) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) 2014-10-24 20:37:29,395 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system... {noformat} in stderr: {noformat} java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:759) at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514) at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440) at org.apache.pig.Main.configureLog4J(Main.java:740) at org.apache.pig.Main.run(Main.java:384) at org.apache.pig.PigRunner.run(PigRunner.java:49) at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:283) at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:223) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37) at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76)
[jira] [Commented] (MAPREDUCE-6139) ConcurrentModificationException in ContainerLogAppender
[ https://issues.apache.org/jira/browse/MAPREDUCE-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14207432#comment-14207432 ] Sangjin Lee commented on MAPREDUCE-6139: LGTM. Just a small nit: I think it's the standard style to have a space between // and the comment that follows it. It would be good if you could touch up that line in append(). [~jira.shegalov]? ConcurrentModificationException in ContainerLogAppender --- Key: MAPREDUCE-6139 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6139 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Mohammad Kamrul Islam Assignee: Mohammad Kamrul Islam Priority: Critical Attachments: ContainerLogAppender.java, MAPREDUCE-6139.1.patch, MAPREDUCE-6139.2.patch, MAPREDUCE-6139.3.patch Context: * Hadoop-2.3.0 * Using Oozie 4.0.1 * Pig version 0.11.x The job is submitted by Oozie to launch Pig script. The following exception traces were found on MR task log: In syslog: {noformat} 2014-10-24 20:37:29,317 WARN [Thread-5] org.apache.hadoop.util.ShutdownHookManager: ShutdownHook '' failed, java.util.ConcurrentModificationException java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.Hierarchy.shutdown(Hierarchy.java:471) at org.apache.log4j.LogManager.shutdown(LogManager.java:267) at org.apache.hadoop.mapred.TaskLog.syncLogsShutdown(TaskLog.java:286) at org.apache.hadoop.mapred.TaskLog$2.run(TaskLog.java:339) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) 2014-10-24 20:37:29,395 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system... {noformat} in stderr: {noformat} java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:759) at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514) at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440) at org.apache.pig.Main.configureLog4J(Main.java:740) at org.apache.pig.Main.run(Main.java:384) at org.apache.pig.PigRunner.run(PigRunner.java:49) at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:283) at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:223) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37) at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6139) ConcurrentModificationException in ContainerLogAppender
[ https://issues.apache.org/jira/browse/MAPREDUCE-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1420#comment-1420 ] Gera Shegalov commented on MAPREDUCE-6139: -- [~kamrul], MAPREDUCE-6139.3.patch looks good to me as well. I should have suggested this earlier but could you write a unit test for this bug? You can create a test with the logic similar to {{o.a.h.mapred.TestTaskLogAppender}} as something like {{o.a.h.yarn.TestContainerLogAppender}}. ./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn. Add tesContainerLogAppenderWithSizeLimit with the regression scenario: # Create a Logger {{claLog}} # Create a CLA appender {{claAppender}} with size limit # claLog.addAppender(claAppender) # claLog.info(message1) with {{Object message1}} where message1.toString() calls claLog.info(stringmessage). # claAppender.close Since you ended up fixing CLA in hadoop-yarn-common, this JIRA has to move to YARN project. ConcurrentModificationException in ContainerLogAppender --- Key: MAPREDUCE-6139 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6139 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Mohammad Kamrul Islam Assignee: Mohammad Kamrul Islam Priority: Critical Attachments: ContainerLogAppender.java, MAPREDUCE-6139.1.patch, MAPREDUCE-6139.2.patch, MAPREDUCE-6139.3.patch Context: * Hadoop-2.3.0 * Using Oozie 4.0.1 * Pig version 0.11.x The job is submitted by Oozie to launch Pig script. The following exception traces were found on MR task log: In syslog: {noformat} 2014-10-24 20:37:29,317 WARN [Thread-5] org.apache.hadoop.util.ShutdownHookManager: ShutdownHook '' failed, java.util.ConcurrentModificationException java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.Hierarchy.shutdown(Hierarchy.java:471) at org.apache.log4j.LogManager.shutdown(LogManager.java:267) at org.apache.hadoop.mapred.TaskLog.syncLogsShutdown(TaskLog.java:286) at org.apache.hadoop.mapred.TaskLog$2.run(TaskLog.java:339) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) 2014-10-24 20:37:29,395 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system... {noformat} in stderr: {noformat} java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:759) at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514) at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440) at org.apache.pig.Main.configureLog4J(Main.java:740) at org.apache.pig.Main.run(Main.java:384) at org.apache.pig.PigRunner.run(PigRunner.java:49) at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:283) at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:223) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37) at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at
[jira] [Commented] (MAPREDUCE-6139) ConcurrentModificationException in ContainerLogAppender
[ https://issues.apache.org/jira/browse/MAPREDUCE-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14201810#comment-14201810 ] Gera Shegalov commented on MAPREDUCE-6139: -- IMHO this complexity is not necessary or even correct just to be able to deal with accidental logging activity occurring when CLA.close is already in progress. ConcurrentModificationException in ContainerLogAppender --- Key: MAPREDUCE-6139 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6139 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Mohammad Kamrul Islam Assignee: Mohammad Kamrul Islam Priority: Critical Attachments: ContainerLogAppender.java, MAPREDUCE-6139.1.patch Context: * Hadoop-2.3.0 * Using Oozie 4.0.1 * Pig version 0.11.x The job is submitted by Oozie to launch Pig script. The following exception traces were found on MR task log: In syslog: {noformat} 2014-10-24 20:37:29,317 WARN [Thread-5] org.apache.hadoop.util.ShutdownHookManager: ShutdownHook '' failed, java.util.ConcurrentModificationException java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.Hierarchy.shutdown(Hierarchy.java:471) at org.apache.log4j.LogManager.shutdown(LogManager.java:267) at org.apache.hadoop.mapred.TaskLog.syncLogsShutdown(TaskLog.java:286) at org.apache.hadoop.mapred.TaskLog$2.run(TaskLog.java:339) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) 2014-10-24 20:37:29,395 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system... {noformat} in stderr: {noformat} java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:759) at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514) at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440) at org.apache.pig.Main.configureLog4J(Main.java:740) at org.apache.pig.Main.run(Main.java:384) at org.apache.pig.PigRunner.run(PigRunner.java:49) at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:283) at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:223) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37) at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6139) ConcurrentModificationException in ContainerLogAppender
[ https://issues.apache.org/jira/browse/MAPREDUCE-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14202507#comment-14202507 ] Mohammad Kamrul Islam commented on MAPREDUCE-6139: -- Thanks again [~jira.shegalov] for your quick and informative comment. Actually I tried the both options : 1. List based 2. using closed variable. Both indeed work. If you think the closed variable based approach is better I would go with that option. As soon as you reconfirm this, I will post a patch. ConcurrentModificationException in ContainerLogAppender --- Key: MAPREDUCE-6139 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6139 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Mohammad Kamrul Islam Assignee: Mohammad Kamrul Islam Priority: Critical Attachments: ContainerLogAppender.java, MAPREDUCE-6139.1.patch Context: * Hadoop-2.3.0 * Using Oozie 4.0.1 * Pig version 0.11.x The job is submitted by Oozie to launch Pig script. The following exception traces were found on MR task log: In syslog: {noformat} 2014-10-24 20:37:29,317 WARN [Thread-5] org.apache.hadoop.util.ShutdownHookManager: ShutdownHook '' failed, java.util.ConcurrentModificationException java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.Hierarchy.shutdown(Hierarchy.java:471) at org.apache.log4j.LogManager.shutdown(LogManager.java:267) at org.apache.hadoop.mapred.TaskLog.syncLogsShutdown(TaskLog.java:286) at org.apache.hadoop.mapred.TaskLog$2.run(TaskLog.java:339) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) 2014-10-24 20:37:29,395 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system... {noformat} in stderr: {noformat} java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:759) at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514) at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440) at org.apache.pig.Main.configureLog4J(Main.java:740) at org.apache.pig.Main.run(Main.java:384) at org.apache.pig.PigRunner.run(PigRunner.java:49) at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:283) at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:223) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37) at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6139) ConcurrentModificationException in ContainerLogAppender
[ https://issues.apache.org/jira/browse/MAPREDUCE-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14202537#comment-14202537 ] Gera Shegalov commented on MAPREDUCE-6139: -- [~kamrul], my comments are non-binding. But yes my preference is for Option 2 based on boolean closed. ConcurrentModificationException in ContainerLogAppender --- Key: MAPREDUCE-6139 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6139 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Mohammad Kamrul Islam Assignee: Mohammad Kamrul Islam Priority: Critical Attachments: ContainerLogAppender.java, MAPREDUCE-6139.1.patch Context: * Hadoop-2.3.0 * Using Oozie 4.0.1 * Pig version 0.11.x The job is submitted by Oozie to launch Pig script. The following exception traces were found on MR task log: In syslog: {noformat} 2014-10-24 20:37:29,317 WARN [Thread-5] org.apache.hadoop.util.ShutdownHookManager: ShutdownHook '' failed, java.util.ConcurrentModificationException java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.Hierarchy.shutdown(Hierarchy.java:471) at org.apache.log4j.LogManager.shutdown(LogManager.java:267) at org.apache.hadoop.mapred.TaskLog.syncLogsShutdown(TaskLog.java:286) at org.apache.hadoop.mapred.TaskLog$2.run(TaskLog.java:339) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) 2014-10-24 20:37:29,395 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system... {noformat} in stderr: {noformat} java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:759) at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514) at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440) at org.apache.pig.Main.configureLog4J(Main.java:740) at org.apache.pig.Main.run(Main.java:384) at org.apache.pig.PigRunner.run(PigRunner.java:49) at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:283) at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:223) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37) at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6139) ConcurrentModificationException in ContainerLogAppender
[ https://issues.apache.org/jira/browse/MAPREDUCE-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14202855#comment-14202855 ] Mohammad Kamrul Islam commented on MAPREDUCE-6139: -- Thanks [~jira.shegalov]. I plan to update the patch soon with option 2. ConcurrentModificationException in ContainerLogAppender --- Key: MAPREDUCE-6139 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6139 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Mohammad Kamrul Islam Assignee: Mohammad Kamrul Islam Priority: Critical Attachments: ContainerLogAppender.java, MAPREDUCE-6139.1.patch Context: * Hadoop-2.3.0 * Using Oozie 4.0.1 * Pig version 0.11.x The job is submitted by Oozie to launch Pig script. The following exception traces were found on MR task log: In syslog: {noformat} 2014-10-24 20:37:29,317 WARN [Thread-5] org.apache.hadoop.util.ShutdownHookManager: ShutdownHook '' failed, java.util.ConcurrentModificationException java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.Hierarchy.shutdown(Hierarchy.java:471) at org.apache.log4j.LogManager.shutdown(LogManager.java:267) at org.apache.hadoop.mapred.TaskLog.syncLogsShutdown(TaskLog.java:286) at org.apache.hadoop.mapred.TaskLog$2.run(TaskLog.java:339) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) 2014-10-24 20:37:29,395 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system... {noformat} in stderr: {noformat} java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:759) at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514) at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440) at org.apache.pig.Main.configureLog4J(Main.java:740) at org.apache.pig.Main.run(Main.java:384) at org.apache.pig.PigRunner.run(PigRunner.java:49) at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:283) at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:223) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37) at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6139) ConcurrentModificationException in ContainerLogAppender
[ https://issues.apache.org/jira/browse/MAPREDUCE-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14203121#comment-14203121 ] Hadoop QA commented on MAPREDUCE-6139: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12680342/MAPREDUCE-6139.2.patch against trunk revision 4a114dd. {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5000//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5000//console This message is automatically generated. ConcurrentModificationException in ContainerLogAppender --- Key: MAPREDUCE-6139 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6139 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Mohammad Kamrul Islam Assignee: Mohammad Kamrul Islam Priority: Critical Attachments: ContainerLogAppender.java, MAPREDUCE-6139.1.patch, MAPREDUCE-6139.2.patch Context: * Hadoop-2.3.0 * Using Oozie 4.0.1 * Pig version 0.11.x The job is submitted by Oozie to launch Pig script. The following exception traces were found on MR task log: In syslog: {noformat} 2014-10-24 20:37:29,317 WARN [Thread-5] org.apache.hadoop.util.ShutdownHookManager: ShutdownHook '' failed, java.util.ConcurrentModificationException java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.Hierarchy.shutdown(Hierarchy.java:471) at org.apache.log4j.LogManager.shutdown(LogManager.java:267) at org.apache.hadoop.mapred.TaskLog.syncLogsShutdown(TaskLog.java:286) at org.apache.hadoop.mapred.TaskLog$2.run(TaskLog.java:339) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) 2014-10-24 20:37:29,395 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system... {noformat} in stderr: {noformat} java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:759) at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514) at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440) at org.apache.pig.Main.configureLog4J(Main.java:740) at org.apache.pig.Main.run(Main.java:384) at org.apache.pig.PigRunner.run(PigRunner.java:49) at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:283) at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:223) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37) at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76) at
[jira] [Commented] (MAPREDUCE-6139) ConcurrentModificationException in ContainerLogAppender
[ https://issues.apache.org/jira/browse/MAPREDUCE-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14203289#comment-14203289 ] Gera Shegalov commented on MAPREDUCE-6139: -- Thanks for the patch [~kamrul]. Reopening CLA violates the javadoc for [Appender#close|https://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/Appender.html#close()] 1. Let us move {code}closing = true;{code} before {code}if (tail != null){code} 2. undo the change try/finally ConcurrentModificationException in ContainerLogAppender --- Key: MAPREDUCE-6139 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6139 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Mohammad Kamrul Islam Assignee: Mohammad Kamrul Islam Priority: Critical Attachments: ContainerLogAppender.java, MAPREDUCE-6139.1.patch, MAPREDUCE-6139.2.patch Context: * Hadoop-2.3.0 * Using Oozie 4.0.1 * Pig version 0.11.x The job is submitted by Oozie to launch Pig script. The following exception traces were found on MR task log: In syslog: {noformat} 2014-10-24 20:37:29,317 WARN [Thread-5] org.apache.hadoop.util.ShutdownHookManager: ShutdownHook '' failed, java.util.ConcurrentModificationException java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.Hierarchy.shutdown(Hierarchy.java:471) at org.apache.log4j.LogManager.shutdown(LogManager.java:267) at org.apache.hadoop.mapred.TaskLog.syncLogsShutdown(TaskLog.java:286) at org.apache.hadoop.mapred.TaskLog$2.run(TaskLog.java:339) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) 2014-10-24 20:37:29,395 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system... {noformat} in stderr: {noformat} java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:759) at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514) at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440) at org.apache.pig.Main.configureLog4J(Main.java:740) at org.apache.pig.Main.run(Main.java:384) at org.apache.pig.PigRunner.run(PigRunner.java:49) at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:283) at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:223) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37) at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6139) ConcurrentModificationException in ContainerLogAppender
[ https://issues.apache.org/jira/browse/MAPREDUCE-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14201187#comment-14201187 ] Mohammad Kamrul Islam commented on MAPREDUCE-6139: -- Thanks [~sjlee0] and [~jira.shegalov] for the active participation. I was considering a little different approach that patch I upload in this JIRA. Can you please review it? Basically, instead of using the iterator based access , I'm using index based. Therefore using LinkedList. I did some basic testing. As soon as, I get more support, I will do the full testing. ConcurrentModificationException in ContainerLogAppender --- Key: MAPREDUCE-6139 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6139 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Mohammad Kamrul Islam Assignee: Mohammad Kamrul Islam Priority: Critical Attachments: ContainerLogAppender.java Context: * Hadoop-2.3.0 * Using Oozie 4.0.1 * Pig version 0.11.x The job is submitted by Oozie to launch Pig script. The following exception traces were found on MR task log: In syslog: {noformat} 2014-10-24 20:37:29,317 WARN [Thread-5] org.apache.hadoop.util.ShutdownHookManager: ShutdownHook '' failed, java.util.ConcurrentModificationException java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.Hierarchy.shutdown(Hierarchy.java:471) at org.apache.log4j.LogManager.shutdown(LogManager.java:267) at org.apache.hadoop.mapred.TaskLog.syncLogsShutdown(TaskLog.java:286) at org.apache.hadoop.mapred.TaskLog$2.run(TaskLog.java:339) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) 2014-10-24 20:37:29,395 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system... {noformat} in stderr: {noformat} java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:759) at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514) at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440) at org.apache.pig.Main.configureLog4J(Main.java:740) at org.apache.pig.Main.run(Main.java:384) at org.apache.pig.PigRunner.run(PigRunner.java:49) at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:283) at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:223) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37) at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6139) ConcurrentModificationException in ContainerLogAppender
[ https://issues.apache.org/jira/browse/MAPREDUCE-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14201208#comment-14201208 ] Gera Shegalov commented on MAPREDUCE-6139: -- [~kamrul], I disagree with this approach for the following reasons: 1. it does not address the root cause, the CLA appender is never closed and keeps accepting appends although it won't be able to get rid of them. 2. LinkedList is not a random access data structure. Every time you call {{get}} you end up scanning half of the list on average ConcurrentModificationException in ContainerLogAppender --- Key: MAPREDUCE-6139 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6139 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Mohammad Kamrul Islam Assignee: Mohammad Kamrul Islam Priority: Critical Attachments: ContainerLogAppender.java, MAPREDUCE-6139.1.patch Context: * Hadoop-2.3.0 * Using Oozie 4.0.1 * Pig version 0.11.x The job is submitted by Oozie to launch Pig script. The following exception traces were found on MR task log: In syslog: {noformat} 2014-10-24 20:37:29,317 WARN [Thread-5] org.apache.hadoop.util.ShutdownHookManager: ShutdownHook '' failed, java.util.ConcurrentModificationException java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.Hierarchy.shutdown(Hierarchy.java:471) at org.apache.log4j.LogManager.shutdown(LogManager.java:267) at org.apache.hadoop.mapred.TaskLog.syncLogsShutdown(TaskLog.java:286) at org.apache.hadoop.mapred.TaskLog$2.run(TaskLog.java:339) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) 2014-10-24 20:37:29,395 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system... {noformat} in stderr: {noformat} java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:759) at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514) at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440) at org.apache.pig.Main.configureLog4J(Main.java:740) at org.apache.pig.Main.run(Main.java:384) at org.apache.pig.PigRunner.run(PigRunner.java:49) at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:283) at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:223) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37) at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6139) ConcurrentModificationException in ContainerLogAppender
[ https://issues.apache.org/jira/browse/MAPREDUCE-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14201360#comment-14201360 ] Mohammad Kamrul Islam commented on MAPREDUCE-6139: -- Thanks [~jira.shegalov] for you comments. My comments inline 1. it does not address the root cause, the CLA appender is never closed and keeps accepting appends although it won't be able to get rid of them. I think it will ultimately be able to close all. Because the same list structure is being appended and close is accessing to those new entries as well. In my testing, I found a case, where close() started with 36 entries in the list and ended up with 38 entries to dealt with. The extra 2 was appended by indirect call from super,append() in the close() method. In other words, list size is increasing at the end and close() will ultimately append them. 2. LinkedList is not a random access data structure. Every time you call get you end up scanning half of the list on average Agree with this comment. ArrayList or some other similar data structure could help in this regard. ConcurrentModificationException in ContainerLogAppender --- Key: MAPREDUCE-6139 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6139 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Mohammad Kamrul Islam Assignee: Mohammad Kamrul Islam Priority: Critical Attachments: ContainerLogAppender.java, MAPREDUCE-6139.1.patch Context: * Hadoop-2.3.0 * Using Oozie 4.0.1 * Pig version 0.11.x The job is submitted by Oozie to launch Pig script. The following exception traces were found on MR task log: In syslog: {noformat} 2014-10-24 20:37:29,317 WARN [Thread-5] org.apache.hadoop.util.ShutdownHookManager: ShutdownHook '' failed, java.util.ConcurrentModificationException java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.Hierarchy.shutdown(Hierarchy.java:471) at org.apache.log4j.LogManager.shutdown(LogManager.java:267) at org.apache.hadoop.mapred.TaskLog.syncLogsShutdown(TaskLog.java:286) at org.apache.hadoop.mapred.TaskLog$2.run(TaskLog.java:339) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) 2014-10-24 20:37:29,395 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system... {noformat} in stderr: {noformat} java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:759) at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514) at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440) at org.apache.pig.Main.configureLog4J(Main.java:740) at org.apache.pig.Main.run(Main.java:384) at org.apache.pig.PigRunner.run(PigRunner.java:49) at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:283) at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:223) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37) at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at
[jira] [Commented] (MAPREDUCE-6139) ConcurrentModificationException in ContainerLogAppender
[ https://issues.apache.org/jira/browse/MAPREDUCE-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14197178#comment-14197178 ] Gera Shegalov commented on MAPREDUCE-6139: -- Hi [~kamrul], thanks for reporting the issue. This occurs when there mapreduce.task.userlog.limit.kb 0. I recommend not to use the original log limit implementation as it has other serious problems such as the potential to lose the valuable diagnostic information in the tail and a long delay until one can see the log to begin with. MAPREDUCE-5672 provided an alternative based on RollingFileAppender that does not have these problems which I recommend to start using. Coming back to the problem in this scenario. One of the pending events in the Queue generated a new warn LogEvent via Token.toString and modified the tail queue while the iterator is still in progress in the {{close}} stack frame. I would fix this as follows: introduce instance boolean variable {{closed}} and set it to {{true}} immediately when {{close}} is invoked. Then we can make CLA's {{append}} a nop when {{closed}} is {{true}}. ConcurrentModificationException in ContainerLogAppender --- Key: MAPREDUCE-6139 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6139 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Mohammad Kamrul Islam Assignee: Mohammad Kamrul Islam Priority: Critical Attachments: ContainerLogAppender.java Context: * Hadoop-2.3.0 * Using Oozie 4.0.1 * Pig version 0.11.x The job is submitted by Oozie to launch Pig script. The following exception traces were found on MR task log: In syslog: {noformat} 2014-10-24 20:37:29,317 WARN [Thread-5] org.apache.hadoop.util.ShutdownHookManager: ShutdownHook '' failed, java.util.ConcurrentModificationException java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.Hierarchy.shutdown(Hierarchy.java:471) at org.apache.log4j.LogManager.shutdown(LogManager.java:267) at org.apache.hadoop.mapred.TaskLog.syncLogsShutdown(TaskLog.java:286) at org.apache.hadoop.mapred.TaskLog$2.run(TaskLog.java:339) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) 2014-10-24 20:37:29,395 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system... {noformat} in stderr: {noformat} java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:759) at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514) at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440) at org.apache.pig.Main.configureLog4J(Main.java:740) at org.apache.pig.Main.run(Main.java:384) at org.apache.pig.PigRunner.run(PigRunner.java:49) at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:283) at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:223) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37) at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at
[jira] [Commented] (MAPREDUCE-6139) ConcurrentModificationException in ContainerLogAppender
[ https://issues.apache.org/jira/browse/MAPREDUCE-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14187298#comment-14187298 ] Mohammad Kamrul Islam commented on MAPREDUCE-6139: -- Updates: I added more debugging to find out how close() call is invoking append() call in the same thread. I dump the call stacktrace for this scenarios. Btw, the is the root cause of ConcurrentModificationException. Case 1: When running with Pig + Oozie. Please note close() is calling append() which modifies the Queue. {noformat} Apache Pig version 0.11.1-li0 (rexported) compiled Jan 27 2014, 14:16:26 Run pig script using PigRunner.run() for Pig version 0.8+ ContainerLogAppender.close() STARTED tail.size() :1 Size :36 object org.apache.hadoop.yarn.ContainerLogAppender@458ad742 In close() :java.lang.Thread.getStackTrace(Thread.java:1552) org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:108) org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) org.apache.log4j.Category.removeAllAppenders(Category.java:891) org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:759) org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648) org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514) org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440) org.apache.pig.Main.configureLog4J(Main.java:740) org.apache.pig.Main.run(Main.java:384) org.apache.pig.PigRunner.run(PigRunner.java:49) org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:283) org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:223) org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37) org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:483) org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226) org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:422) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) ContainerLogAppender.append() STARTED main tail.size() :36 object org.apache.hadoop.yarn.ContainerLogAppender@458ad742 In append() :java.lang.Thread.getStackTrace(Thread.java:1552) org.apache.hadoop.yarn.ContainerLogAppender.append(ContainerLogAppender.java:75) org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251) org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66) org.apache.log4j.Category.callAppenders(Category.java:206) org.apache.log4j.Category.forcedLog(Category.java:391) org.apache.log4j.Category.log(Category.java:856) org.apache.commons.logging.impl.Log4JLogger.warn(Log4JLogger.java:197) org.apache.hadoop.security.token.Token.getClassForIdentifier(Token.java:118) org.apache.hadoop.security.token.Token.decodeIdentifier(Token.java:132) org.apache.hadoop.security.token.Token.identifierToString(Token.java:314) org.apache.hadoop.security.token.Token.toString(Token.java:334) org.apache.log4j.or.DefaultRenderer.doRender(DefaultRenderer.java:37) org.apache.log4j.or.RendererMap.findAndRender(RendererMap.java:80) org.apache.log4j.spi.LoggingEvent.getRenderedMessage(LoggingEvent.java:368) org.apache.log4j.helpers.PatternParser$BasicPatternConverter.convert(PatternParser.java:402) org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65) org.apache.log4j.PatternLayout.format(PatternLayout.java:506) org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:310) org.apache.log4j.WriterAppender.append(WriterAppender.java:162) org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:117) org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) org.apache.log4j.Category.removeAllAppenders(Category.java:891) org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:759) org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648) org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514) org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440) org.apache.pig.Main.configureLog4J(Main.java:740) org.apache.pig.Main.run(Main.java:384) org.apache.pig.PigRunner.run(PigRunner.java:49)
[jira] [Commented] (MAPREDUCE-6139) ConcurrentModificationException in ContainerLogAppender
[ https://issues.apache.org/jira/browse/MAPREDUCE-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14187311#comment-14187311 ] Mohammad Kamrul Islam commented on MAPREDUCE-6139: -- Why does the close() call the super.append()? I think it should call super.close() instead? {noformat} for (LoggingEvent event : tail) { super.append(event); } {noformat} ConcurrentModificationException in ContainerLogAppender --- Key: MAPREDUCE-6139 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6139 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Mohammad Kamrul Islam Assignee: Mohammad Kamrul Islam Priority: Critical Attachments: ContainerLogAppender.java Context: * Hadoop-2.3.0 * Using Oozie 4.0.1 * Pig version 0.11.x The job is submitted by Oozie to launch Pig script. The following exception traces were found on MR task log: In syslog: {noformat} 2014-10-24 20:37:29,317 WARN [Thread-5] org.apache.hadoop.util.ShutdownHookManager: ShutdownHook '' failed, java.util.ConcurrentModificationException java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.Hierarchy.shutdown(Hierarchy.java:471) at org.apache.log4j.LogManager.shutdown(LogManager.java:267) at org.apache.hadoop.mapred.TaskLog.syncLogsShutdown(TaskLog.java:286) at org.apache.hadoop.mapred.TaskLog$2.run(TaskLog.java:339) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) 2014-10-24 20:37:29,395 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system... {noformat} in stderr: {noformat} java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:759) at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514) at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440) at org.apache.pig.Main.configureLog4J(Main.java:740) at org.apache.pig.Main.run(Main.java:384) at org.apache.pig.PigRunner.run(PigRunner.java:49) at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:283) at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:223) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37) at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6139) ConcurrentModificationException in ContainerLogAppender
[ https://issues.apache.org/jira/browse/MAPREDUCE-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14187472#comment-14187472 ] Sangjin Lee commented on MAPREDUCE-6139: I think that call is there by design. The tail queue is an internal buffer and that code is trying to write out the buffered log events. You'll also see super.close() right below this for loop. I think this is triggering the exception (not that it is the cause but that it triggers the problem): {noformat} org.apache.commons.logging.impl.Log4JLogger.warn(Log4JLogger.java:197) org.apache.hadoop.security.token.Token.getClassForIdentifier(Token.java:118) {noformat} I haven't thought much about this, but perhaps another way to solve this is to modify append() not to use the tail queue if it is in the middle of a shutdown. ConcurrentModificationException in ContainerLogAppender --- Key: MAPREDUCE-6139 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6139 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Mohammad Kamrul Islam Assignee: Mohammad Kamrul Islam Priority: Critical Attachments: ContainerLogAppender.java Context: * Hadoop-2.3.0 * Using Oozie 4.0.1 * Pig version 0.11.x The job is submitted by Oozie to launch Pig script. The following exception traces were found on MR task log: In syslog: {noformat} 2014-10-24 20:37:29,317 WARN [Thread-5] org.apache.hadoop.util.ShutdownHookManager: ShutdownHook '' failed, java.util.ConcurrentModificationException java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.Hierarchy.shutdown(Hierarchy.java:471) at org.apache.log4j.LogManager.shutdown(LogManager.java:267) at org.apache.hadoop.mapred.TaskLog.syncLogsShutdown(TaskLog.java:286) at org.apache.hadoop.mapred.TaskLog$2.run(TaskLog.java:339) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) 2014-10-24 20:37:29,395 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system... {noformat} in stderr: {noformat} java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:759) at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514) at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440) at org.apache.pig.Main.configureLog4J(Main.java:740) at org.apache.pig.Main.run(Main.java:384) at org.apache.pig.PigRunner.run(PigRunner.java:49) at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:283) at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:223) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37) at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at
[jira] [Commented] (MAPREDUCE-6139) ConcurrentModificationException in ContainerLogAppender
[ https://issues.apache.org/jira/browse/MAPREDUCE-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14187550#comment-14187550 ] Mohammad Kamrul Islam commented on MAPREDUCE-6139: -- Thanks [~sjlee0] for the reply. I agree with your suggestion and already started the testing process by blocking append() when it is called from close(). Not sure if it is the cleaner approach. I will update with my testing result. ConcurrentModificationException in ContainerLogAppender --- Key: MAPREDUCE-6139 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6139 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Mohammad Kamrul Islam Assignee: Mohammad Kamrul Islam Priority: Critical Attachments: ContainerLogAppender.java Context: * Hadoop-2.3.0 * Using Oozie 4.0.1 * Pig version 0.11.x The job is submitted by Oozie to launch Pig script. The following exception traces were found on MR task log: In syslog: {noformat} 2014-10-24 20:37:29,317 WARN [Thread-5] org.apache.hadoop.util.ShutdownHookManager: ShutdownHook '' failed, java.util.ConcurrentModificationException java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.Hierarchy.shutdown(Hierarchy.java:471) at org.apache.log4j.LogManager.shutdown(LogManager.java:267) at org.apache.hadoop.mapred.TaskLog.syncLogsShutdown(TaskLog.java:286) at org.apache.hadoop.mapred.TaskLog$2.run(TaskLog.java:339) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) 2014-10-24 20:37:29,395 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system... {noformat} in stderr: {noformat} java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:759) at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514) at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440) at org.apache.pig.Main.configureLog4J(Main.java:740) at org.apache.pig.Main.run(Main.java:384) at org.apache.pig.PigRunner.run(PigRunner.java:49) at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:283) at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:223) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37) at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6139) ConcurrentModificationException in ContainerLogAppender
[ https://issues.apache.org/jira/browse/MAPREDUCE-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14187723#comment-14187723 ] Sangjin Lee commented on MAPREDUCE-6139: Just off the top of my head, how about an approach where if the close() (or shutdown) is in progress append() would simply call super.append() instead of playing with the tail queue? Perhaps a boolean flag to indicate that a shutdown is progress... That might be better than blocking (?) append(). ConcurrentModificationException in ContainerLogAppender --- Key: MAPREDUCE-6139 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6139 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Mohammad Kamrul Islam Assignee: Mohammad Kamrul Islam Priority: Critical Attachments: ContainerLogAppender.java Context: * Hadoop-2.3.0 * Using Oozie 4.0.1 * Pig version 0.11.x The job is submitted by Oozie to launch Pig script. The following exception traces were found on MR task log: In syslog: {noformat} 2014-10-24 20:37:29,317 WARN [Thread-5] org.apache.hadoop.util.ShutdownHookManager: ShutdownHook '' failed, java.util.ConcurrentModificationException java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.Hierarchy.shutdown(Hierarchy.java:471) at org.apache.log4j.LogManager.shutdown(LogManager.java:267) at org.apache.hadoop.mapred.TaskLog.syncLogsShutdown(TaskLog.java:286) at org.apache.hadoop.mapred.TaskLog$2.run(TaskLog.java:339) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) 2014-10-24 20:37:29,395 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system... {noformat} in stderr: {noformat} java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:759) at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514) at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440) at org.apache.pig.Main.configureLog4J(Main.java:740) at org.apache.pig.Main.run(Main.java:384) at org.apache.pig.PigRunner.run(PigRunner.java:49) at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:283) at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:223) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37) at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6139) ConcurrentModificationException in ContainerLogAppender
[ https://issues.apache.org/jira/browse/MAPREDUCE-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14187810#comment-14187810 ] Mohammad Kamrul Islam commented on MAPREDUCE-6139: -- Wandering if it will create recursive call. Because, in this case, super.append() is ultimately calling this append(). ConcurrentModificationException in ContainerLogAppender --- Key: MAPREDUCE-6139 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6139 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Mohammad Kamrul Islam Assignee: Mohammad Kamrul Islam Priority: Critical Attachments: ContainerLogAppender.java Context: * Hadoop-2.3.0 * Using Oozie 4.0.1 * Pig version 0.11.x The job is submitted by Oozie to launch Pig script. The following exception traces were found on MR task log: In syslog: {noformat} 2014-10-24 20:37:29,317 WARN [Thread-5] org.apache.hadoop.util.ShutdownHookManager: ShutdownHook '' failed, java.util.ConcurrentModificationException java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.Hierarchy.shutdown(Hierarchy.java:471) at org.apache.log4j.LogManager.shutdown(LogManager.java:267) at org.apache.hadoop.mapred.TaskLog.syncLogsShutdown(TaskLog.java:286) at org.apache.hadoop.mapred.TaskLog$2.run(TaskLog.java:339) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) 2014-10-24 20:37:29,395 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system... {noformat} in stderr: {noformat} java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:759) at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514) at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440) at org.apache.pig.Main.configureLog4J(Main.java:740) at org.apache.pig.Main.run(Main.java:384) at org.apache.pig.PigRunner.run(PigRunner.java:49) at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:283) at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:223) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37) at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6139) ConcurrentModificationException in ContainerLogAppender
[ https://issues.apache.org/jira/browse/MAPREDUCE-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14187818#comment-14187818 ] Mohammad Kamrul Islam commented on MAPREDUCE-6139: -- Forgot to add, in append(), super.append() is called only once when tail is null. ConcurrentModificationException in ContainerLogAppender --- Key: MAPREDUCE-6139 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6139 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Mohammad Kamrul Islam Assignee: Mohammad Kamrul Islam Priority: Critical Attachments: ContainerLogAppender.java Context: * Hadoop-2.3.0 * Using Oozie 4.0.1 * Pig version 0.11.x The job is submitted by Oozie to launch Pig script. The following exception traces were found on MR task log: In syslog: {noformat} 2014-10-24 20:37:29,317 WARN [Thread-5] org.apache.hadoop.util.ShutdownHookManager: ShutdownHook '' failed, java.util.ConcurrentModificationException java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.Hierarchy.shutdown(Hierarchy.java:471) at org.apache.log4j.LogManager.shutdown(LogManager.java:267) at org.apache.hadoop.mapred.TaskLog.syncLogsShutdown(TaskLog.java:286) at org.apache.hadoop.mapred.TaskLog$2.run(TaskLog.java:339) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) 2014-10-24 20:37:29,395 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system... {noformat} in stderr: {noformat} java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:759) at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514) at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440) at org.apache.pig.Main.configureLog4J(Main.java:740) at org.apache.pig.Main.run(Main.java:384) at org.apache.pig.PigRunner.run(PigRunner.java:49) at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:283) at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:223) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37) at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6139) ConcurrentModificationException in ContainerLogAppender
[ https://issues.apache.org/jira/browse/MAPREDUCE-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14183496#comment-14183496 ] Mohammad Kamrul Islam commented on MAPREDUCE-6139: -- I did some investigation/debugging and found the problem happened when ContainerLogAppender.close() is iterating, the ContainerLogAppender.append() was executing by other thread. So append() add an entry and then close() found the ConcurrentModificationException. The main question is : append() is a synchronized method and close() is within synchronized block. The question is : how it is possible to execute both such synchronized methods at the same time. Am I missing something? ConcurrentModificationException in ContainerLogAppender --- Key: MAPREDUCE-6139 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6139 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Mohammad Kamrul Islam Assignee: Mohammad Kamrul Islam Priority: Critical Context: * Hadoop-2.3.0 * Using Oozie 4.0.1 * Pig version 0.11.x The job is submitted by Oozie to launch Pig script. The following exception traces were found on MR task log: In syslog: {noformat} 2014-10-24 20:37:29,317 WARN [Thread-5] org.apache.hadoop.util.ShutdownHookManager: ShutdownHook '' failed, java.util.ConcurrentModificationException java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.Hierarchy.shutdown(Hierarchy.java:471) at org.apache.log4j.LogManager.shutdown(LogManager.java:267) at org.apache.hadoop.mapred.TaskLog.syncLogsShutdown(TaskLog.java:286) at org.apache.hadoop.mapred.TaskLog$2.run(TaskLog.java:339) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) 2014-10-24 20:37:29,395 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system... {noformat} in stderr: {noformat} java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:759) at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514) at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440) at org.apache.pig.Main.configureLog4J(Main.java:740) at org.apache.pig.Main.run(Main.java:384) at org.apache.pig.PigRunner.run(PigRunner.java:49) at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:283) at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:223) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37) at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6139) ConcurrentModificationException in ContainerLogAppender
[ https://issues.apache.org/jira/browse/MAPREDUCE-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14183612#comment-14183612 ] Sangjin Lee commented on MAPREDUCE-6139: [~kamrul], the line doesn't seem to match up with the 2.3.0 source. ContainerLogAppender.close() is l.79-86 in 2.3.0, but your stack trace indicates the exception occurs on line 94. Are there private changes to it? Could you share the source in its entirety? ConcurrentModificationException in ContainerLogAppender --- Key: MAPREDUCE-6139 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6139 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Mohammad Kamrul Islam Assignee: Mohammad Kamrul Islam Priority: Critical Context: * Hadoop-2.3.0 * Using Oozie 4.0.1 * Pig version 0.11.x The job is submitted by Oozie to launch Pig script. The following exception traces were found on MR task log: In syslog: {noformat} 2014-10-24 20:37:29,317 WARN [Thread-5] org.apache.hadoop.util.ShutdownHookManager: ShutdownHook '' failed, java.util.ConcurrentModificationException java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.Hierarchy.shutdown(Hierarchy.java:471) at org.apache.log4j.LogManager.shutdown(LogManager.java:267) at org.apache.hadoop.mapred.TaskLog.syncLogsShutdown(TaskLog.java:286) at org.apache.hadoop.mapred.TaskLog$2.run(TaskLog.java:339) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) 2014-10-24 20:37:29,395 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system... {noformat} in stderr: {noformat} java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:759) at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514) at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440) at org.apache.pig.Main.configureLog4J(Main.java:740) at org.apache.pig.Main.run(Main.java:384) at org.apache.pig.PigRunner.run(PigRunner.java:49) at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:283) at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:223) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37) at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6139) ConcurrentModificationException in ContainerLogAppender
[ https://issues.apache.org/jira/browse/MAPREDUCE-6139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14183679#comment-14183679 ] Mohammad Kamrul Islam commented on MAPREDUCE-6139: -- Thanks [~sjlee0] for quick answer and catching the missing piece. Yes. this is coming from my debugged code. The stack trace coming from the original code ran few days earlier: in syslog {noformat} 2014-10-10 01:46:53,188 WARN [main] org.apache.hadoop.util.ShutdownHookManager: ShutdownHook '' failed, java.util.ConcurrentModificationException java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:81) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.Hierarchy.shutdown(Hierarchy.java:471) at org.apache.log4j.LogManager.shutdown(LogManager.java:267) at org.apache.hadoop.mapred.TaskLog.syncLogsShutdown(TaskLog.java:286) at org.apache.hadoop.mapred.TaskLog$2.run(TaskLog.java:339) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) {noformat} in stderr {noformat} java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:81) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:759) at org.apache.log4j.PropertyConfigurator.configureRootCategory(PropertyConfigurator.java:648) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:514) at org.apache.log4j.PropertyConfigurator.configure(PropertyConfigurator.java:440) at org.apache.pig.Main.configureLog4J(Main.java:740) at org.apache.pig.Main.run(Main.java:384) at org.apache.pig.PigRunner.run(PigRunner.java:49) at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:283) at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:223) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37) at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) {noformat} ConcurrentModificationException in ContainerLogAppender --- Key: MAPREDUCE-6139 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6139 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Mohammad Kamrul Islam Assignee: Mohammad Kamrul Islam Priority: Critical Context: * Hadoop-2.3.0 * Using Oozie 4.0.1 * Pig version 0.11.x The job is submitted by Oozie to launch Pig script. The following exception traces were found on MR task log: In syslog: {noformat} 2014-10-24 20:37:29,317 WARN [Thread-5] org.apache.hadoop.util.ShutdownHookManager: ShutdownHook '' failed, java.util.ConcurrentModificationException java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) at java.util.LinkedList$ListItr.next(LinkedList.java:888) at org.apache.hadoop.yarn.ContainerLogAppender.close(ContainerLogAppender.java:94) at org.apache.log4j.helpers.AppenderAttachableImpl.removeAllAppenders(AppenderAttachableImpl.java:141) at org.apache.log4j.Category.removeAllAppenders(Category.java:891) at org.apache.log4j.Hierarchy.shutdown(Hierarchy.java:471) at org.apache.log4j.LogManager.shutdown(LogManager.java:267) at org.apache.hadoop.mapred.TaskLog.syncLogsShutdown(TaskLog.java:286) at