[jira] [Commented] (MAPREDUCE-4135) MRAppMaster throws IllegalStateException while shutting down
[
https://issues.apache.org/jira/browse/MAPREDUCE-4135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13264350#comment-13264350
]
Ravi Prakash commented on MAPREDUCE-4135:
-
Thanks Tucu! I hadn't seen HADOOP-8325. You probably meant MAPREDUCE-4157. I am
planning on letting this JIRA stay open until either / both of those JIRA make
the problem go away.
MRAppMaster throws IllegalStateException while shutting down
Key: MAPREDUCE-4135
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4135
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Ravi Prakash
Attachments: MAPREDUCE-4135.branch-0.23.patch, MAPREDUCE-4135.patch,
MAPREDUCE-4135.patch
Always MRAppMaster throws IllegalStateException in the stderr while shutting
down. It doesn't look good having this exception in the stderr file of
MRAppMaster container.
{code:xml}
WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use
org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
Exception in thread Thread-1 java.lang.IllegalStateException: Shutdown in
progress
at
java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:55)
at java.lang.Runtime.removeShutdownHook(Runtime.java:220)
at org.apache.hadoop.fs.FileSystem$Cache.remove(FileSystem.java:2148)
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2180)
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2157)
at org.apache.hadoop.fs.FileSystem.closeAll(FileSystem.java:361)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1014)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4135) MRAppMaster throws IllegalStateException while shutting down
[
https://issues.apache.org/jira/browse/MAPREDUCE-4135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13263969#comment-13263969
]
Alejandro Abdelnur commented on MAPREDUCE-4135:
---
HADOOP-8325 MAPREDUCE-4135 take care of this problem in a more generic way.
Please evaluate and close if so.
MRAppMaster throws IllegalStateException while shutting down
Key: MAPREDUCE-4135
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4135
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Ravi Prakash
Attachments: MAPREDUCE-4135.branch-0.23.patch, MAPREDUCE-4135.patch,
MAPREDUCE-4135.patch
Always MRAppMaster throws IllegalStateException in the stderr while shutting
down. It doesn't look good having this exception in the stderr file of
MRAppMaster container.
{code:xml}
WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use
org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
Exception in thread Thread-1 java.lang.IllegalStateException: Shutdown in
progress
at
java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:55)
at java.lang.Runtime.removeShutdownHook(Runtime.java:220)
at org.apache.hadoop.fs.FileSystem$Cache.remove(FileSystem.java:2148)
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2180)
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2157)
at org.apache.hadoop.fs.FileSystem.closeAll(FileSystem.java:361)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1014)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4135) MRAppMaster throws IllegalStateException while shutting down
[
https://issues.apache.org/jira/browse/MAPREDUCE-4135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13261823#comment-13261823
]
Ravi Prakash commented on MAPREDUCE-4135:
-
I'm sorry I was wrong about my earlier statement:
bq. This is causing the subsequent statement (FileSystem.closeAll() ) to be
skipped. So to fix this JIRA, I'll see what I can do to fix the
InterruptedException.
The InterruptedException is thrown and then caught. So its not that. Its
MAPREDUCE-4157. The SIGTERM is sent without any delay on
ContainerLaunch.java:337.
MRAppMaster throws IllegalStateException while shutting down
Key: MAPREDUCE-4135
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4135
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Ravi Prakash
Attachments: MAPREDUCE-4135.branch-0.23.patch, MAPREDUCE-4135.patch,
MAPREDUCE-4135.patch
Always MRAppMaster throws IllegalStateException in the stderr while shutting
down. It doesn't look good having this exception in the stderr file of
MRAppMaster container.
{code:xml}
WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use
org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
Exception in thread Thread-1 java.lang.IllegalStateException: Shutdown in
progress
at
java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:55)
at java.lang.Runtime.removeShutdownHook(Runtime.java:220)
at org.apache.hadoop.fs.FileSystem$Cache.remove(FileSystem.java:2148)
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2180)
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2157)
at org.apache.hadoop.fs.FileSystem.closeAll(FileSystem.java:361)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1014)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4135) MRAppMaster throws IllegalStateException while shutting down
[
https://issues.apache.org/jira/browse/MAPREDUCE-4135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13260614#comment-13260614
]
Ravi Prakash commented on MAPREDUCE-4135:
-
Hi Devaraj! I've assigned this JIRA to myself. I hope you don't mind.
AFTER applying the latest patch, inside MRAppMaster.stop(), super.stop() is
throwing an InterruptedException
{noformat}
2012-04-24 09:25:20,104 WARN [AsyncDispatcher event handler]
org.apache.hadoop.yarn.event.AsyncDispatcher: Interrupted Exception while
stopping
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1186)
at java.lang.Thread.join(Thread.java:1239)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.stop(AsyncDispatcher.java:104)
at
org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:99)
at
org.apache.hadoop.yarn.service.CompositeService.stop(CompositeService.java:89)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.stop(MRAppMaster.java:1016)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:418)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler.handle(MRAppMaster.java:386)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:125)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:74)
at java.lang.Thread.run(Thread.java:662)
{noformat}
This is causing the subsequent statement (FileSystem.closeAll() ) to be
skipped. So to fix this JIRA, I'll see what I can do to fix the
InterruptedException.
MRAppMaster throws IllegalStateException while shutting down
Key: MAPREDUCE-4135
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4135
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Ravi Prakash
Attachments: MAPREDUCE-4135.branch-0.23.patch, MAPREDUCE-4135.patch,
MAPREDUCE-4135.patch
Always MRAppMaster throws IllegalStateException in the stderr while shutting
down. It doesn't look good having this exception in the stderr file of
MRAppMaster container.
{code:xml}
WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use
org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
Exception in thread Thread-1 java.lang.IllegalStateException: Shutdown in
progress
at
java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:55)
at java.lang.Runtime.removeShutdownHook(Runtime.java:220)
at org.apache.hadoop.fs.FileSystem$Cache.remove(FileSystem.java:2148)
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2180)
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2157)
at org.apache.hadoop.fs.FileSystem.closeAll(FileSystem.java:361)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1014)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4135) MRAppMaster throws IllegalStateException while shutting down
[
https://issues.apache.org/jira/browse/MAPREDUCE-4135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254830#comment-13254830
]
Hadoop QA commented on MAPREDUCE-4135:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12522790/MAPREDUCE-4135.branch-0.23.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell
org.apache.hadoop.yarn.server.TestDiskFailures
org.apache.hadoop.yarn.server.TestContainerManagerSecurity
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
org.apache.hadoop.mapred.TestMiniMRClasspath
org.apache.hadoop.mapreduce.v2.TestMRJobs
org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers
org.apache.hadoop.mapred.TestMiniMRBringup
org.apache.hadoop.mapred.TestMiniMRChildTask
org.apache.hadoop.mapred.TestReduceFetch
org.apache.hadoop.mapred.TestClusterMRNotification
org.apache.hadoop.mapred.TestReduceFetchFromPartialMem
org.apache.hadoop.mapred.TestJobCounters
org.apache.hadoop.mapreduce.TestChild
org.apache.hadoop.mapred.TestMiniMRClientCluster
org.apache.hadoop.ipc.TestSocketFactory
org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution
org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter
org.apache.hadoop.mapred.TestClientRedirect
org.apache.hadoop.mapred.TestLazyOutput
org.apache.hadoop.mapred.TestJobCleanup
org.apache.hadoop.mapreduce.TestMapReduceLazyOutput
org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
org.apache.hadoop.mapreduce.v2.TestMRAppWithCombiner
org.apache.hadoop.conf.TestNoDefaultsJobConf
org.apache.hadoop.mapreduce.v2.TestRMNMInfo
org.apache.hadoop.mapred.TestClusterMapReduceTestCase
org.apache.hadoop.mapreduce.v2.TestNonExistentJob
org.apache.hadoop.mapred.TestJobSysDirWithDFS
org.apache.hadoop.mapreduce.v2.TestUberAM
org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser
org.apache.hadoop.mapred.TestJobName
org.apache.hadoop.mapreduce.security.TestJHSSecurity
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2225//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2225//console
This message is automatically generated.
MRAppMaster throws IllegalStateException while shutting down
Key: MAPREDUCE-4135
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4135
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Attachments: MAPREDUCE-4135.branch-0.23.patch, MAPREDUCE-4135.patch,
MAPREDUCE-4135.patch
Always MRAppMaster throws IllegalStateException in the stderr while shutting
down. It doesn't look good having this exception in the stderr file of
MRAppMaster container.
{code:xml}
WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use
[jira] [Commented] (MAPREDUCE-4135) MRAppMaster throws IllegalStateException while shutting down
[
https://issues.apache.org/jira/browse/MAPREDUCE-4135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13252253#comment-13252253
]
Alejandro Abdelnur commented on MAPREDUCE-4135:
---
+1
MRAppMaster throws IllegalStateException while shutting down
Key: MAPREDUCE-4135
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4135
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Attachments: MAPREDUCE-4135.patch
Always MRAppMaster throws IllegalStateException in the stderr while shutting
down. It doesn't look good having this exception in the stderr file of
MRAppMaster container.
{code:xml}
WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use
org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
Exception in thread Thread-1 java.lang.IllegalStateException: Shutdown in
progress
at
java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:55)
at java.lang.Runtime.removeShutdownHook(Runtime.java:220)
at org.apache.hadoop.fs.FileSystem$Cache.remove(FileSystem.java:2148)
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2180)
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2157)
at org.apache.hadoop.fs.FileSystem.closeAll(FileSystem.java:361)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1014)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4135) MRAppMaster throws IllegalStateException while shutting down
[
https://issues.apache.org/jira/browse/MAPREDUCE-4135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13252341#comment-13252341
]
Hadoop QA commented on MAPREDUCE-4135:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12522393/MAPREDUCE-4135.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell
org.apache.hadoop.yarn.server.TestDiskFailures
org.apache.hadoop.yarn.server.TestContainerManagerSecurity
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
org.apache.hadoop.mapred.TestMiniMRClasspath
org.apache.hadoop.mapreduce.v2.TestMRJobs
org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers
org.apache.hadoop.mapred.TestMiniMRBringup
org.apache.hadoop.mapred.TestMiniMRChildTask
org.apache.hadoop.mapred.TestReduceFetch
org.apache.hadoop.mapred.TestClusterMRNotification
org.apache.hadoop.mapred.TestReduceFetchFromPartialMem
org.apache.hadoop.mapred.TestJobCounters
org.apache.hadoop.mapreduce.TestChild
org.apache.hadoop.mapred.TestMiniMRClientCluster
org.apache.hadoop.ipc.TestSocketFactory
org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution
org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter
org.apache.hadoop.mapred.TestClientRedirect
org.apache.hadoop.mapred.TestLazyOutput
org.apache.hadoop.mapred.TestJobCleanup
org.apache.hadoop.mapreduce.TestMapReduceLazyOutput
org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
org.apache.hadoop.mapreduce.v2.TestMRAppWithCombiner
org.apache.hadoop.conf.TestNoDefaultsJobConf
org.apache.hadoop.mapreduce.v2.TestRMNMInfo
org.apache.hadoop.mapred.TestClusterMapReduceTestCase
org.apache.hadoop.mapreduce.v2.TestNonExistentJob
org.apache.hadoop.mapred.TestJobSysDirWithDFS
org.apache.hadoop.mapreduce.v2.TestUberAM
org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser
org.apache.hadoop.mapred.TestJobName
org.apache.hadoop.mapreduce.security.TestJHSSecurity
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2209//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2209//console
This message is automatically generated.
MRAppMaster throws IllegalStateException while shutting down
Key: MAPREDUCE-4135
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4135
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Attachments: MAPREDUCE-4135.patch, MAPREDUCE-4135.patch
Always MRAppMaster throws IllegalStateException in the stderr while shutting
down. It doesn't look good having this exception in the stderr file of
MRAppMaster container.
{code:xml}
WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use
org.apache.hadoop.log.metrics.EventCounter in all the
[jira] [Commented] (MAPREDUCE-4135) MRAppMaster throws IllegalStateException while shutting down
[
https://issues.apache.org/jira/browse/MAPREDUCE-4135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13252355#comment-13252355
]
Devaraj K commented on MAPREDUCE-4135:
--
These above test failures are not related to the patch. Seems these are random
failures with the reason java.net.BindException: Problem binding to
[0.0.0.0:8031] java.net.BindException: Address already in use. Same tests
failed for MAPREDUCE-4140 QA also.
MRAppMaster throws IllegalStateException while shutting down
Key: MAPREDUCE-4135
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4135
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Attachments: MAPREDUCE-4135.patch, MAPREDUCE-4135.patch
Always MRAppMaster throws IllegalStateException in the stderr while shutting
down. It doesn't look good having this exception in the stderr file of
MRAppMaster container.
{code:xml}
WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use
org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
Exception in thread Thread-1 java.lang.IllegalStateException: Shutdown in
progress
at
java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:55)
at java.lang.Runtime.removeShutdownHook(Runtime.java:220)
at org.apache.hadoop.fs.FileSystem$Cache.remove(FileSystem.java:2148)
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2180)
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2157)
at org.apache.hadoop.fs.FileSystem.closeAll(FileSystem.java:361)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1014)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4135) MRAppMaster throws IllegalStateException while shutting down
[
https://issues.apache.org/jira/browse/MAPREDUCE-4135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13252472#comment-13252472
]
Ravi Prakash commented on MAPREDUCE-4135:
-
I'd set automatic close to false to fix MAPREDUCE-3614.
MRAppMaster.java:1000-1003
{noformat}
// Do not automatically close FileSystem objects so that in case of
// SIGTERM I have a chance to write out the job history. I'll be closing
// the objects myself.
conf.setBoolean(fs.automatic.close, false);
{noformat}
We cannot let FileSystem close the handle before we write out JobHistory. We'll
have to find a way to still gracefully release all resources.
MRAppMaster throws IllegalStateException while shutting down
Key: MAPREDUCE-4135
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4135
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Attachments: MAPREDUCE-4135.patch, MAPREDUCE-4135.patch
Always MRAppMaster throws IllegalStateException in the stderr while shutting
down. It doesn't look good having this exception in the stderr file of
MRAppMaster container.
{code:xml}
WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use
org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
Exception in thread Thread-1 java.lang.IllegalStateException: Shutdown in
progress
at
java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:55)
at java.lang.Runtime.removeShutdownHook(Runtime.java:220)
at org.apache.hadoop.fs.FileSystem$Cache.remove(FileSystem.java:2148)
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2180)
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2157)
at org.apache.hadoop.fs.FileSystem.closeAll(FileSystem.java:361)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1014)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4135) MRAppMaster throws IllegalStateException while shutting down
[
https://issues.apache.org/jira/browse/MAPREDUCE-4135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13252608#comment-13252608
]
Ravi Prakash commented on MAPREDUCE-4135:
-
This is a particularly vexing problem. :( I can think of two ways out of this
1. Swallow IllegalStateExceptions thrown by removeShutdownHook in FileSystem.
The javadoc states:
{quote}public boolean removeShutdownHook(Thread hook)
Throws:
IllegalStateException - If the virtual machine is already in the process of
shutting down {quote}
So if we are getting this exception, it MEANS we are already in the process of
shutdown, so we CANNOT, try what we may, removeShutdownHook. If Runtime had a
method Runtime.isShutdownInProgress(), we could have checked for it before the
removeShutdownHook call. As it stands, there is no such method. In my opinion,
this would be a good patch regardless of the needs for this JIRA.
2. Not send SIGTERMs from the NM to the MR-AM in the first place. Rather we
should expose a mechanism for the NM to politely tell the AM its no longer
needed and should shutdown asap. Even after this, if an admin were to kill the
MRAppMaster with a SIGTERM, the JobHistory would be lost defeating the purpose
of 3614
MRAppMaster throws IllegalStateException while shutting down
Key: MAPREDUCE-4135
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4135
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Attachments: MAPREDUCE-4135.patch, MAPREDUCE-4135.patch
Always MRAppMaster throws IllegalStateException in the stderr while shutting
down. It doesn't look good having this exception in the stderr file of
MRAppMaster container.
{code:xml}
WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use
org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
Exception in thread Thread-1 java.lang.IllegalStateException: Shutdown in
progress
at
java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:55)
at java.lang.Runtime.removeShutdownHook(Runtime.java:220)
at org.apache.hadoop.fs.FileSystem$Cache.remove(FileSystem.java:2148)
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2180)
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2157)
at org.apache.hadoop.fs.FileSystem.closeAll(FileSystem.java:361)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1014)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4135) MRAppMaster throws IllegalStateException while shutting down
[
https://issues.apache.org/jira/browse/MAPREDUCE-4135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13251441#comment-13251441
]
Steve Loughran commented on MAPREDUCE-4135:
---
The problem is that the FS shutdown process is being triggered by the app
master shutdown hook -at which point it is too late to remove other shutdown
hooks.
{{FileSystem.Cache.remove()}} should just catch any {{IllegalStateException}}
and not worry so much about shutdown hook state failures as the system is
usually in an odd state at this point.
MRAppMaster throws IllegalStateException while shutting down
Key: MAPREDUCE-4135
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4135
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 3.0.0
Reporter: Devaraj K
Always MRAppMaster throws IllegalStateException in the stderr while shutting
down. It doesn't look good having this exception in the stderr file of
MRAppMaster container.
{code:xml}
WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use
org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
Exception in thread Thread-1 java.lang.IllegalStateException: Shutdown in
progress
at
java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:55)
at java.lang.Runtime.removeShutdownHook(Runtime.java:220)
at org.apache.hadoop.fs.FileSystem$Cache.remove(FileSystem.java:2148)
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2180)
at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2157)
at org.apache.hadoop.fs.FileSystem.closeAll(FileSystem.java:361)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1014)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
