[jira] [Commented] (MAPREDUCE-5723) MR AM container log empty if exception occurs
[
https://issues.apache.org/jira/browse/MAPREDUCE-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13878693#comment-13878693
]
Jason Lowe commented on MAPREDUCE-5723:
---
Agree with Karthik that a test case would be nice because unit tests are useful
for regression tests and not just for new functionality. However this will be
a bit tricky to test via spy since LogManager.shutdown is a static function,
and MRAppMaster is not invoking it via an object that can be modified for
testing. For this patch I'm OK if it doesn't have a test since it's relatively
small and straightforward, but in general we want unit tests even if it's just
a bugfix.
As for the patch itself, we need to call LogManager.shutdown() after
super.stop(), otherwise we end up shutting down the logging system before
shutting down all the services registered within the MRAppMaster service. We
want to see the logs of those sub-services shutting down.
> MR AM container log empty if exception occurs
> -
>
> Key: MAPREDUCE-5723
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5723
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
>Affects Versions: 2.2.0
>Reporter: Mohammad Kamrul Islam
>Assignee: Mohammad Kamrul Islam
> Fix For: trunk, 2.2.0
>
> Attachments: MAPREDUCE-5723.1.patch, MAPREDUCE-5723.2.patch,
> MAPREDUCE-5723.3.patch
>
>
> It occurs when the property "mapreduce.task.userlog.limit.kb" is set non-zero
> in mapped-site.xml.
> AM container syslog remains empty if any exception occurs.
> Bug details:
> In MRAppMaster.java, the following code snippets show the bug.
> {code}
> } catch (Throwable t) {
>LOG.fatal("Error starting MRAppMaster", t);
> System.exit(1);
> }finally {
>LogManager.shutdown();
> }
> {code}
> In the catch block, we are exiting the JVM. So finally block (therefore
> LogManager.shutdown()) is never executed.
> Possible fix:
> Make sure LogManager.shutdown() is executed in all cases.
>
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (MAPREDUCE-5723) MR AM container log empty if exception occurs
[
https://issues.apache.org/jira/browse/MAPREDUCE-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13873130#comment-13873130
]
Mohammad Kamrul Islam commented on MAPREDUCE-5723:
--
* No new functionality is added. Therefore,no new test case is required. But I
tested it in a cluster setup.
Failed test case: It is unrelated to this change. There is a separate JIRA to
address this issue:
https://issues.apache.org/jira/browse/MAPREDUCE-5719
> MR AM container log empty if exception occurs
> -
>
> Key: MAPREDUCE-5723
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5723
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
>Affects Versions: 2.2.0
>Reporter: Mohammad Kamrul Islam
>Assignee: Mohammad Kamrul Islam
> Fix For: trunk, 2.2.0
>
> Attachments: MAPREDUCE-5723.1.patch, MAPREDUCE-5723.2.patch,
> MAPREDUCE-5723.3.patch
>
>
> It occurs when the property "mapreduce.task.userlog.limit.kb" is set non-zero
> in mapped-site.xml.
> AM container syslog remains empty if any exception occurs.
> Bug details:
> In MRAppMaster.java, the following code snippets show the bug.
> {code}
> } catch (Throwable t) {
>LOG.fatal("Error starting MRAppMaster", t);
> System.exit(1);
> }finally {
>LogManager.shutdown();
> }
> {code}
> In the catch block, we are exiting the JVM. So finally block (therefore
> LogManager.shutdown()) is never executed.
> Possible fix:
> Make sure LogManager.shutdown() is executed in all cases.
>
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (MAPREDUCE-5723) MR AM container log empty if exception occurs
[
https://issues.apache.org/jira/browse/MAPREDUCE-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13873044#comment-13873044
]
Hadoop QA commented on MAPREDUCE-5723:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12623305/MAPREDUCE-5723.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app:
org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4323//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4323//console
This message is automatically generated.
> MR AM container log empty if exception occurs
> -
>
> Key: MAPREDUCE-5723
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5723
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
>Affects Versions: 2.2.0
>Reporter: Mohammad Kamrul Islam
>Assignee: Mohammad Kamrul Islam
> Fix For: trunk, 2.2.0
>
> Attachments: MAPREDUCE-5723.1.patch, MAPREDUCE-5723.2.patch,
> MAPREDUCE-5723.3.patch
>
>
> It occurs when the property "mapreduce.task.userlog.limit.kb" is set non-zero
> in mapped-site.xml.
> AM container syslog remains empty if any exception occurs.
> Bug details:
> In MRAppMaster.java, the following code snippets show the bug.
> {code}
> } catch (Throwable t) {
>LOG.fatal("Error starting MRAppMaster", t);
> System.exit(1);
> }finally {
>LogManager.shutdown();
> }
> {code}
> In the catch block, we are exiting the JVM. So finally block (therefore
> LogManager.shutdown()) is never executed.
> Possible fix:
> Make sure LogManager.shutdown() is executed in all cases.
>
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (MAPREDUCE-5723) MR AM container log empty if exception occurs
[
https://issues.apache.org/jira/browse/MAPREDUCE-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13873038#comment-13873038
]
Karthik Kambatla commented on MAPREDUCE-5723:
-
Thanks [~kamrul]. The patch definitely looks better now. Should have mentioned
this earlier, but it would be nice to add a unit test. How about something that
just verifies LogManager.shutdown is called - may be using spy?
I ll have to verify why TestRMContainerAllocator is failing, but we can ignore
the failure if it is also failing on trunk without the patch.
> MR AM container log empty if exception occurs
> -
>
> Key: MAPREDUCE-5723
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5723
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
>Affects Versions: 2.2.0
>Reporter: Mohammad Kamrul Islam
>Assignee: Mohammad Kamrul Islam
> Fix For: trunk, 2.2.0
>
> Attachments: MAPREDUCE-5723.1.patch, MAPREDUCE-5723.2.patch,
> MAPREDUCE-5723.3.patch
>
>
> It occurs when the property "mapreduce.task.userlog.limit.kb" is set non-zero
> in mapped-site.xml.
> AM container syslog remains empty if any exception occurs.
> Bug details:
> In MRAppMaster.java, the following code snippets show the bug.
> {code}
> } catch (Throwable t) {
>LOG.fatal("Error starting MRAppMaster", t);
> System.exit(1);
> }finally {
>LogManager.shutdown();
> }
> {code}
> In the catch block, we are exiting the JVM. So finally block (therefore
> LogManager.shutdown()) is never executed.
> Possible fix:
> Make sure LogManager.shutdown() is executed in all cases.
>
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (MAPREDUCE-5723) MR AM container log empty if exception occurs
[
https://issues.apache.org/jira/browse/MAPREDUCE-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872967#comment-13872967
]
Mohammad Kamrul Islam commented on MAPREDUCE-5723:
--
Thanks [~kkambatl] for your proposal. I will put a patch based on that.
In the mean time, do you have any idea why the test case is failing? If I run
it myself w/o any code changes the test case still fails.
> MR AM container log empty if exception occurs
> -
>
> Key: MAPREDUCE-5723
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5723
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
>Affects Versions: 2.2.0
>Reporter: Mohammad Kamrul Islam
>Assignee: Mohammad Kamrul Islam
> Fix For: trunk, 2.2.0
>
> Attachments: MAPREDUCE-5723.1.patch, MAPREDUCE-5723.2.patch
>
>
> It occurs when the property "mapreduce.task.userlog.limit.kb" is set non-zero
> in mapped-site.xml.
> AM container syslog remains empty if any exception occurs.
> Bug details:
> In MRAppMaster.java, the following code snippets show the bug.
> {code}
> } catch (Throwable t) {
>LOG.fatal("Error starting MRAppMaster", t);
> System.exit(1);
> }finally {
>LogManager.shutdown();
> }
> {code}
> In the catch block, we are exiting the JVM. So finally block (therefore
> LogManager.shutdown()) is never executed.
> Possible fix:
> Make sure LogManager.shutdown() is executed in all cases.
>
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (MAPREDUCE-5723) MR AM container log empty if exception occurs
[
https://issues.apache.org/jira/browse/MAPREDUCE-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872562#comment-13872562
]
Karthik Kambatla commented on MAPREDUCE-5723:
-
I think the clean way, though slightly more involved, would be to have
MRAppMaster implement serviceStop() and call LogManager.shutdown() there.
While at it, I also think the {{MRAppMasterShutdownHook}} should only call
appMaster.stop() and rest of the items should move to serviceStop().
{code}
public void run() {
LOG.info("MRAppMaster received a signal. Signaling RMCommunicator and "
+ "JobHistoryEventHandler.");
// Notify the JHEH and RMCommunicator that a SIGTERM has been received so
// that they don't take too long in shutting down
if(appMaster.containerAllocator instanceof ContainerAllocatorRouter) {
((ContainerAllocatorRouter) appMaster.containerAllocator)
.setSignalled(true);
}
appMaster.notifyIsLastAMRetry(appMaster.isLastAMRetry);
appMaster.stop();
}
{code}
Further, instead of calling System.exit(1), we should call
ExitUtil.terminate(1, t):
{code}
} catch (Throwable t) {
LOG.fatal("Error starting MRAppMaster", t);
System.exit(1);
{code}
> MR AM container log empty if exception occurs
> -
>
> Key: MAPREDUCE-5723
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5723
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
>Affects Versions: 2.2.0
>Reporter: Mohammad Kamrul Islam
>Assignee: Mohammad Kamrul Islam
> Fix For: trunk, 2.2.0
>
> Attachments: MAPREDUCE-5723.1.patch, MAPREDUCE-5723.2.patch
>
>
> It occurs when the property "mapreduce.task.userlog.limit.kb" is set non-zero
> in mapped-site.xml.
> AM container syslog remains empty if any exception occurs.
> Bug details:
> In MRAppMaster.java, the following code snippets show the bug.
> } catch (Throwable t) {
>LOG.fatal("Error starting MRAppMaster", t);
> System.exit(1);
> }finally {
>LogManager.shutdown();
> }
> In the catch block, we are exiting the JVM. So finally block (therefore
> LogManager.shutdown()) is never executed.
> Possible fix:
> Make sure LogManager.shutdown() is executed in all cases.
>
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (MAPREDUCE-5723) MR AM container log empty if exception occurs
[
https://issues.apache.org/jira/browse/MAPREDUCE-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872561#comment-13872561
]
Hadoop QA commented on MAPREDUCE-5723:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12623211/MAPREDUCE-5723.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app:
org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4319//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4319//console
This message is automatically generated.
> MR AM container log empty if exception occurs
> -
>
> Key: MAPREDUCE-5723
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5723
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
>Affects Versions: 2.2.0
>Reporter: Mohammad Kamrul Islam
>Assignee: Mohammad Kamrul Islam
> Fix For: trunk, 2.2.0
>
> Attachments: MAPREDUCE-5723.1.patch, MAPREDUCE-5723.2.patch
>
>
> It occurs when the property "mapreduce.task.userlog.limit.kb" is set non-zero
> in mapped-site.xml.
> AM container syslog remains empty if any exception occurs.
> Bug details:
> In MRAppMaster.java, the following code snippets show the bug.
> } catch (Throwable t) {
>LOG.fatal("Error starting MRAppMaster", t);
> System.exit(1);
> }finally {
>LogManager.shutdown();
> }
> In the catch block, we are exiting the JVM. So finally block (therefore
> LogManager.shutdown()) is never executed.
> Possible fix:
> Make sure LogManager.shutdown() is executed in all cases.
>
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (MAPREDUCE-5723) MR AM container log empty if exception occurs
[
https://issues.apache.org/jira/browse/MAPREDUCE-5723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13871402#comment-13871402
]
Hadoop QA commented on MAPREDUCE-5723:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12623004/MAPREDUCE-5723.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app:
org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4318//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4318//console
This message is automatically generated.
> MR AM container log empty if exception occurs
> -
>
> Key: MAPREDUCE-5723
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5723
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
>Affects Versions: 2.2.0
>Reporter: Mohammad Kamrul Islam
>Assignee: Mohammad Kamrul Islam
> Fix For: trunk, 2.2.0
>
> Attachments: MAPREDUCE-5723.1.patch
>
>
> It occurs when the property "mapreduce.task.userlog.limit.kb" is set non-zero
> in mapped-site.xml.
> AM container syslog remains empty if any exception occurs.
> Bug details:
> In MRAppMaster.java, the following code snippets show the bug.
> } catch (Throwable t) {
>LOG.fatal("Error starting MRAppMaster", t);
> System.exit(1);
> }finally {
>LogManager.shutdown();
> }
> In the catch block, we are exiting the JVM. So finally block (therefore
> LogManager.shutdown()) is never executed.
> Possible fix:
> Make sure LogManager.shutdown() is executed in all cases.
>
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
