[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14256718#comment-14256718 ] sam liu commented on MAPREDUCE-6204: Hi Tsuyoshi, I attached new patch MAPREDUCE-6204-1.patch and could you please help take a review? Thanks! > TestJobCounters should use new properties instead > JobConf.MAPRED_TASK_JAVA_OPTS > --- > > Key: MAPREDUCE-6204 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Affects Versions: 2.6.0 >Reporter: sam liu >Assignee: sam liu >Priority: Minor > Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated MAPREDUCE-6204: --- Status: Patch Available (was: Open) > TestJobCounters should use new properties instead > JobConf.MAPRED_TASK_JAVA_OPTS > --- > > Key: MAPREDUCE-6204 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Affects Versions: 2.6.0 >Reporter: sam liu >Assignee: sam liu >Priority: Minor > Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated MAPREDUCE-6204: --- Status: Open (was: Patch Available) > TestJobCounters should use new properties instead > JobConf.MAPRED_TASK_JAVA_OPTS > --- > > Key: MAPREDUCE-6204 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Affects Versions: 2.6.0 >Reporter: sam liu >Assignee: sam liu >Priority: Minor > Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated MAPREDUCE-6204: --- Attachment: MAPREDUCE-6204-1.patch Attached new patch. Thanks! > TestJobCounters should use new properties instead > JobConf.MAPRED_TASK_JAVA_OPTS > --- > > Key: MAPREDUCE-6204 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Affects Versions: 2.6.0 >Reporter: sam liu >Assignee: sam liu >Priority: Minor > Attachments: MAPREDUCE-6204-1.patch, MAPREDUCE-6204.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14256687#comment-14256687 ] sam liu commented on MAPREDUCE-6204: Gera, I did a test: 1. Without the patch, current test TestJobCounters#MemoryLoaderMapper and MemoryLoaderReducer will get value 'null' for properties MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS during MR job execution: MAPRED_TASK_JAVA_OPTS=-Xms32m -Xmx1G MAPRED_MAP_TASK_JAVA_OPTS=null MAPRED_REDUCE_TASK_JAVA_OPTS=null 2. With the patch, current test TestJobCounters#MemoryLoaderMapper and MemoryLoaderReducer will get correct value for all the properties: MAPRED_TASK_JAVA_OPTS=-Xms32m -Xmx1G MAPRED_MAP_TASK_JAVA_OPTS=-Xms32m -Xmx1G MAPRED_REDUCE_TASK_JAVA_OPTS=-Xms32m -Xmx1G > TestJobCounters should use new properties instead > JobConf.MAPRED_TASK_JAVA_OPTS > --- > > Key: MAPREDUCE-6204 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Affects Versions: 2.6.0 >Reporter: sam liu >Assignee: sam liu >Priority: Minor > Attachments: MAPREDUCE-6204.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[
https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14256604#comment-14256604
]
Gera Shegalov commented on MAPREDUCE-6204:
--
I am not sure it helps with crashes because memory settings have not changed
with this patch if you look at
{{MapReduceChildJVM#getChildJavaOpts}}
> TestJobCounters should use new properties instead
> JobConf.MAPRED_TASK_JAVA_OPTS
> ---
>
> Key: MAPREDUCE-6204
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: sam liu
>Assignee: sam liu
>Priority: Minor
> Attachments: MAPREDUCE-6204.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14256594#comment-14256594 ] Tsuyoshi OZAWA commented on MAPREDUCE-6204: --- [~sam liu], the fix itself good to me, but the patch is not valid one. Could you generate the patch by using git diff --no-prefix command? > TestJobCounters should use new properties instead > JobConf.MAPRED_TASK_JAVA_OPTS > --- > > Key: MAPREDUCE-6204 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Affects Versions: 2.6.0 >Reporter: sam liu >Assignee: sam liu >Priority: Minor > Attachments: MAPREDUCE-6204.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[
https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14256579#comment-14256579
]
Hadoop QA commented on MAPREDUCE-6204:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12688787/MAPREDUCE-6204.patch
against trunk revision fdf042d.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5088//console
This message is automatically generated.
> TestJobCounters should use new properties instead
> JobConf.MAPRED_TASK_JAVA_OPTS
> ---
>
> Key: MAPREDUCE-6204
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: test
>Affects Versions: 2.6.0
>Reporter: sam liu
>Assignee: sam liu
>Priority: Minor
> Attachments: MAPREDUCE-6204.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated MAPREDUCE-6204: --- Attachment: MAPREDUCE-6204.patch The solution is to replace the deprecated property MAPRED_TASK_JAVA_OPTS with its new values MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS in the test. After applying the patch, current test could pass on PPC64 platform all the time. > TestJobCounters should use new properties instead > JobConf.MAPRED_TASK_JAVA_OPTS > --- > > Key: MAPREDUCE-6204 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Affects Versions: 2.6.0 >Reporter: sam liu >Assignee: sam liu >Priority: Minor > Attachments: MAPREDUCE-6204.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu updated MAPREDUCE-6204: --- Status: Patch Available (was: Open) > TestJobCounters should use new properties instead > JobConf.MAPRED_TASK_JAVA_OPTS > --- > > Key: MAPREDUCE-6204 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Affects Versions: 2.6.0 >Reporter: sam liu >Assignee: sam liu >Priority: Minor > Attachments: MAPREDUCE-6204.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14256575#comment-14256575 ] sam liu commented on MAPREDUCE-6204: In Hadoop 2.x, the property JobConf.MAPRED_TASK_JAVA_OPTS is deprecated, and hadoop will ignore this property directly without replacing its new values(MAPRED_MAP_TASK_JAVA_OPTS and MAPRED_REDUCE_TASK_JAVA_OPTS). This issue will cause test TestJobCounters fail on PPC64 platform sometimes, as PPC64 JVM sometimes comsumes more memory when executing map/reduce tasks and might encounter JVM crashing issue once the default heap size is exhausted. > TestJobCounters should use new properties instead > JobConf.MAPRED_TASK_JAVA_OPTS > --- > > Key: MAPREDUCE-6204 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Affects Versions: 2.6.0 >Reporter: sam liu >Assignee: sam liu >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sam liu reassigned MAPREDUCE-6204: -- Assignee: sam liu > TestJobCounters should use new properties instead > JobConf.MAPRED_TASK_JAVA_OPTS > --- > > Key: MAPREDUCE-6204 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Affects Versions: 2.6.0 >Reporter: sam liu >Assignee: sam liu >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6204) TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS
sam liu created MAPREDUCE-6204: -- Summary: TestJobCounters should use new properties instead JobConf.MAPRED_TASK_JAVA_OPTS Key: MAPREDUCE-6204 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6204 Project: Hadoop Map/Reduce Issue Type: Test Components: test Affects Versions: 2.6.0 Reporter: sam liu Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-6202) TestMRTimelineEventHandling fails on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhijie Shen resolved MAPREDUCE-6202.
Resolution: Duplicate
Close it as the duplicate of MAPREDUCE-6189
> TestMRTimelineEventHandling fails on trunk
> --
>
> Key: MAPREDUCE-6202
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6202
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Robert Kanter
>
> Currently, {{TestMRTimelineEventHandling}} is failing on trunk:
> {noformat}
> Running org.apache.hadoop.mapred.TestMRTimelineEventHandling
> Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 181.186 sec
> <<< FAILURE! - in org.apache.hadoop.mapred.TestMRTimelineEventHandling
> testMRTimelineEventHandling(org.apache.hadoop.mapred.TestMRTimelineEventHandling)
> Time elapsed: 97.149 sec <<< ERROR!
> java.io.IOException: Job didn't finish in 30 seconds
> at
> org.apache.hadoop.mapred.UtilsForTests.runJobSucceed(UtilsForTests.java:622)
> at
> org.apache.hadoop.mapred.TestMRTimelineEventHandling.testMRTimelineEventHandling(TestMRTimelineEventHandling.java:53)
> testMapreduceJobTimelineServiceEnabled(org.apache.hadoop.mapred.TestMRTimelineEventHandling)
> Time elapsed: 83.884 sec <<< ERROR!
> java.io.IOException: Job didn't finish in 30 seconds
> at
> org.apache.hadoop.mapred.UtilsForTests.runJobSucceed(UtilsForTests.java:622)
> at
> org.apache.hadoop.mapred.TestMRTimelineEventHandling.testMapreduceJobTimelineServiceEnabled(TestMRTimelineEventHandling.java:107)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6149) Document override log4j.properties in MR job
[
https://issues.apache.org/jira/browse/MAPREDUCE-6149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14255749#comment-14255749
]
Hadoop QA commented on MAPREDUCE-6149:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12688649/MAPREDUCE-6149-v3.patch
against trunk revision a696fbb.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 13 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5087//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5087//artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5087//console
This message is automatically generated.
> Document override log4j.properties in MR job
>
>
> Key: MAPREDUCE-6149
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6149
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: documentation
>Reporter: Junping Du
>Assignee: Junping Du
> Attachments: MAPREDUCE-6149-v2.patch, MAPREDUCE-6149-v3.patch,
> MAPREDUCE-6149.patch
>
>
> This new feature comes from MAPREDUCE-6052, some documentation requirements
> from Vinod below:
> Document the new config in mapred-default.xml
> Mention in that documentation that if no-scheme is given in the path, it
> defaults to a log4j file on the local FS.
> Modify the documentation of log-level configs to say that if you override
> to have your own log4j.properties file, the log-level configs may not work.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6149) Document override log4j.properties in MR job
[ https://issues.apache.org/jira/browse/MAPREDUCE-6149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14255714#comment-14255714 ] Junping Du commented on MAPREDUCE-6149: --- Thanks [~qwertymaniac] for review and comments! v3 patch should address your comments above, please help to review it again. Thx! > Document override log4j.properties in MR job > > > Key: MAPREDUCE-6149 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6149 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: documentation >Reporter: Junping Du >Assignee: Junping Du > Attachments: MAPREDUCE-6149-v2.patch, MAPREDUCE-6149-v3.patch, > MAPREDUCE-6149.patch > > > This new feature comes from MAPREDUCE-6052, some documentation requirements > from Vinod below: > Document the new config in mapred-default.xml > Mention in that documentation that if no-scheme is given in the path, it > defaults to a log4j file on the local FS. > Modify the documentation of log-level configs to say that if you override > to have your own log4j.properties file, the log-level configs may not work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6149) Document override log4j.properties in MR job
[ https://issues.apache.org/jira/browse/MAPREDUCE-6149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-6149: -- Attachment: MAPREDUCE-6149-v3.patch > Document override log4j.properties in MR job > > > Key: MAPREDUCE-6149 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6149 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: documentation >Reporter: Junping Du >Assignee: Junping Du > Attachments: MAPREDUCE-6149-v2.patch, MAPREDUCE-6149-v3.patch, > MAPREDUCE-6149.patch > > > This new feature comes from MAPREDUCE-6052, some documentation requirements > from Vinod below: > Document the new config in mapred-default.xml > Mention in that documentation that if no-scheme is given in the path, it > defaults to a log4j file on the local FS. > Modify the documentation of log-level configs to say that if you override > to have your own log4j.properties file, the log-level configs may not work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6203) if log-aggregation is enable and run some MR job, the AM log will be aggregated.then disable it,then in MR JobHistoryServer and YARN RM UI link,the AM log cannot be v
huangyitian created MAPREDUCE-6203: -- Summary: if log-aggregation is enable and run some MR job, the AM log will be aggregated.then disable it,then in MR JobHistoryServer and YARN RM UI link,the AM log cannot be visible Key: MAPREDUCE-6203 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6203 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 2.4.1 Reporter: huangyitian Priority: Minor as the Summary description, I think for the MR JobHistory Server, should not let “yarn.log-aggregation-enable” affect the history job,even “yarn.log-aggregation-enable” is false. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
