[jira] [Commented] (MAPREDUCE-6515) Update Application priority in AM side from AM-RM heartbeat
[
https://issues.apache.org/jira/browse/MAPREDUCE-6515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14974980#comment-14974980
]
Jason Lowe commented on MAPREDUCE-6515:
---
Thanks for updating the patch, Sunil! I, too, cannot reproduce the test
failures. However the findbugs errors flagged above are relevant. The report
is blank, but there are 3 warnings introduced by the patch. Unfortunately the
report generated is bogus and empty (see HADOOP-12312), but here's the relevant
findings if I run it manually on the patch:
{noformat}
{noformat}
The new get/set priority methods in JobReportPBImpl need to be synchronized.
Otherwise latest patch looks good.
> Update Application priority in AM side from AM-RM heartbeat
> ---
>
> Key: MAPREDUCE-6515
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6515
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: 0001-MAPREDUCE-6515.patch, 0002-MAPREDUCE-6515.patch,
> 0003-MAPREDUCE-6515.patch, 0004-MAPREDUCE-6515.patch
>
>
> After YARN-4170, Application Priority is available via heartbeat call. Update
> this information in AM sothat client can fetch this information via JobStatus
> (JobReport) call.
> This is as per the discussion happened in MAPREDUCE-5870.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6513) MR job got hanged forever when one NM unstable for some time
[ https://issues.apache.org/jira/browse/MAPREDUCE-6513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated MAPREDUCE-6513: Target Version/s: 2.7.2 > MR job got hanged forever when one NM unstable for some time > > > Key: MAPREDUCE-6513 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6513 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster, resourcemanager >Affects Versions: 2.7.0 >Reporter: Bob >Assignee: Varun Saxena >Priority: Critical > > when job is in-progress which is having more tasks,one node became unstable > due to some OS issue.After the node became unstable, the map on this node > status changed to KILLED state. > Currently maps which were running on unstable node are rescheduled, and all > are in scheduled state and wait for RM assign container.Seen ask requests for > map till Node is good (all those failed), there are no ask request after > this. But AM keeps on preempting the reducers (it's recycling). > Finally reducers are waiting for complete mappers and mappers did n't get > container.. > My Question Is: > > why map requests did not sent AM ,once after node recovery.? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5485) Allow repeating job commit by extending OutputCommitter API
[
https://issues.apache.org/jira/browse/MAPREDUCE-5485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14974888#comment-14974888
]
Hadoop QA commented on MAPREDUCE-5485:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 5s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 8m 2s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 26s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 10s | The applied patch generated 2
new checkstyle issues (total was 566, now 567). |
| {color:red}-1{color} | whitespace | 0m 1s | The patch has 3 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 36s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | mapreduce tests | 9m 41s | Tests failed in
hadoop-mapreduce-client-app. |
| {color:green}+1{color} | mapreduce tests | 1m 55s | Tests passed in
hadoop-mapreduce-client-core. |
| | | 54m 31s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.mapreduce.v2.app.TestMRAppMaster |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12768071/MAPREDUCE-5485-demo.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 123b3db |
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6089/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6089/artifact/patchprocess/whitespace.txt
|
| hadoop-mapreduce-client-app test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6089/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6089/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6089/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6089/console |
This message was automatically generated.
> Allow repeating job commit by extending OutputCommitter API
> ---
>
> Key: MAPREDUCE-5485
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5485
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Affects Versions: 2.1.0-beta
>Reporter: Nemon Lou
>Assignee: Junping Du
> Attachments: MAPREDUCE-5485-demo.patch
>
>
> There are chances MRAppMaster crush during job committing,or NodeManager
> restart cause the committing AM exit due to container expire.In these cases
> ,the job will fail.
> However,some jobs can redo commit so failing the job becomes unnecessary.
> Let clients tell AM to allow redo commit or not is a better choice.
> This idea comes from Jason Lowe's comments in MAPREDUCE-4819
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6514) Update ask to indicate to RM that it need not allocate for ramped down reducers
[
https://issues.apache.org/jira/browse/MAPREDUCE-6514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Saxena updated MAPREDUCE-6514:
Target Version/s: 2.7.2
> Update ask to indicate to RM that it need not allocate for ramped down
> reducers
> ---
>
> Key: MAPREDUCE-6514
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6514
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
>Affects Versions: 2.7.1
>Reporter: Varun Saxena
>Assignee: Varun Saxena
>Priority: Critical
>
> In RMContainerAllocator#preemptReducesIfNeeded, we simply clear the scheduled
> reduces map and put these reducers to pending. This is not updated in ask. So
> RM keeps on assigning and AM is not able to assign as no reducer is
> scheduled(check logs below the code).
> If this is updated immediately, RM will be able to schedule mappers
> immediately which anyways is the intention when we ramp down reducers.
> This if not handled can lead to map starvation as pointed out in
> MAPREDUCE-6513
> {code}
> LOG.info("Ramping down all scheduled reduces:"
> + scheduledRequests.reduces.size());
> for (ContainerRequest req : scheduledRequests.reduces.values()) {
> pendingReduces.add(req);
> }
> scheduledRequests.reduces.clear();
> {code}
> {noformat}
> 2015-10-13 04:55:04,912 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Container not
> assigned : container_1437451211867_1485_01_000215
> 2015-10-13 04:55:04,912 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Cannot assign
> container Container: [ContainerId: container_1437451211867_1485_01_000216,
> NodeId: hdszzdcxdat6g06u04p:26009, NodeHttpAddress:
> hdszzdcxdat6g06u04p:26010, Resource: , Priority: 10,
> Token: Token { kind: ContainerToken, service: 10.2.33.236:26009 }, ] for a
> reduce as either container memory less than required 4096 or no pending
> reduce tasks - reduces.isEmpty=true
> 2015-10-13 04:55:04,912 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Container not
> assigned : container_1437451211867_1485_01_000216
> 2015-10-13 04:55:04,912 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Cannot assign
> container Container: [ContainerId: container_1437451211867_1485_01_000217,
> NodeId: hdszzdcxdat6g06u06p:26009, NodeHttpAddress:
> hdszzdcxdat6g06u06p:26010, Resource: , Priority: 10,
> Token: Token { kind: ContainerToken, service: 10.2.33.239:26009 }, ] for a
> reduce as either container memory less than required 4096 or no pending
> reduce tasks - reduces.isEmpty=true
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6524) Fix intermittent test failure of TestMRJobsWithHistoryService.testJobHistoryData
[ https://issues.apache.org/jira/browse/MAPREDUCE-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated MAPREDUCE-6524: Component/s: test > Fix intermittent test failure of > TestMRJobsWithHistoryService.testJobHistoryData > > > Key: MAPREDUCE-6524 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6524 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Reporter: Masatake Iwasaki >Assignee: Masatake Iwasaki >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6525) Fix test failure of TestMiniMRClientCluster.testRestart
Masatake Iwasaki created MAPREDUCE-6525: --- Summary: Fix test failure of TestMiniMRClientCluster.testRestart Key: MAPREDUCE-6525 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6525 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Reporter: Masatake Iwasaki Assignee: Masatake Iwasaki Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6524) Fix intermittent test failure of TestMRJobsWithHistoryService.testJobHistoryData
Masatake Iwasaki created MAPREDUCE-6524: --- Summary: Fix intermittent test failure of TestMRJobsWithHistoryService.testJobHistoryData Key: MAPREDUCE-6524 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6524 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Masatake Iwasaki Assignee: Masatake Iwasaki Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6525) Fix test failure of TestMiniMRClientCluster.testRestart
[
https://issues.apache.org/jira/browse/MAPREDUCE-6525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14975543#comment-14975543
]
Masatake Iwasaki commented on MAPREDUCE-6525:
-
{noformat}
Running org.apache.hadoop.mapred.TestMiniMRClientCluster
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 71.73 sec <<<
FAILURE! - in org.apache.hadoop.mapred.TestMiniMRClientCluster
testRestart(org.apache.hadoop.mapred.TestMiniMRClientCluster) Time elapsed:
28.472 sec <<< FAILURE!
org.junit.ComparisonFailure: Address before restart: centos7:0 is different
from new address: localhost:37880 expected:<[centos7:]0> but
was:<[localhost:3788]0>
at org.junit.Assert.assertEquals(Assert.java:115)
at
org.apache.hadoop.mapred.TestMiniMRClientCluster.testRestart(TestMiniMRClientCluster.java:146)
{noformat}
> Fix test failure of TestMiniMRClientCluster.testRestart
> ---
>
> Key: MAPREDUCE-6525
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6525
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
>Priority: Minor
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6521) MiniMRYarnCluster should not create /tmp/hadoop-yarn/staging on local filesystem in unit test
[
https://issues.apache.org/jira/browse/MAPREDUCE-6521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated MAPREDUCE-6521:
Status: Patch Available (was: Open)
> MiniMRYarnCluster should not create /tmp/hadoop-yarn/staging on local
> filesystem in unit test
> -
>
> Key: MAPREDUCE-6521
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6521
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
> Attachments: MAPREDUCE-6521.001.patch
>
>
> MiniMRYarnCluster create /tmp/hadoop-yarn/staging/history/done by default. It
> should be under {{testWorkDir}} if the file system is localFs in order to
> make it to be removed by {{mvn clean}}. It would also avoid issues under
> parallel unit testing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6521) MiniMRYarnCluster should not create /tmp/hadoop-yarn/staging on local filesystem in unit test
[
https://issues.apache.org/jira/browse/MAPREDUCE-6521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Masatake Iwasaki updated MAPREDUCE-6521:
Attachment: MAPREDUCE-6521.001.patch
I attached the patch. I verified that /tmp/hadoop-yarn is not created by
manually running some unit tests which uses MiniMRYarnCluster with local
filesystem.
> MiniMRYarnCluster should not create /tmp/hadoop-yarn/staging on local
> filesystem in unit test
> -
>
> Key: MAPREDUCE-6521
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6521
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
> Attachments: MAPREDUCE-6521.001.patch
>
>
> MiniMRYarnCluster create /tmp/hadoop-yarn/staging/history/done by default. It
> should be under {{testWorkDir}} if the file system is localFs in order to
> make it to be removed by {{mvn clean}}. It would also avoid issues under
> parallel unit testing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6521) MiniMRYarnCluster should not create /tmp/hadoop-yarn/staging on local filesystem in unit test
[
https://issues.apache.org/jira/browse/MAPREDUCE-6521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14974408#comment-14974408
]
Hadoop QA commented on MAPREDUCE-6521:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 6m 38s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 8m 16s | There were no new javac warning
messages. |
| {color:green}+1{color} | release audit | 0m 21s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 37s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 39s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 0m 53s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | mapreduce tests | 102m 32s | Tests failed in
hadoop-mapreduce-client-jobclient. |
| | | 121m 33s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.mapreduce.v2.TestMRJobsWithHistoryService |
| | hadoop.mapred.TestMiniMRClientCluster |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12768718/MAPREDUCE-6521.001.patch
|
| Optional Tests | javac unit findbugs checkstyle |
| git revision | trunk / 5acdde4 |
| hadoop-mapreduce-client-jobclient test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6088/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6088/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6088/console |
This message was automatically generated.
> MiniMRYarnCluster should not create /tmp/hadoop-yarn/staging on local
> filesystem in unit test
> -
>
> Key: MAPREDUCE-6521
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6521
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Masatake Iwasaki
>Assignee: Masatake Iwasaki
> Attachments: MAPREDUCE-6521.001.patch
>
>
> MiniMRYarnCluster create /tmp/hadoop-yarn/staging/history/done by default. It
> should be under {{testWorkDir}} if the file system is localFs in order to
> make it to be removed by {{mvn clean}}. It would also avoid issues under
> parallel unit testing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5485) Allow repeating job commit by extending OutputCommitter API
[ https://issues.apache.org/jira/browse/MAPREDUCE-5485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-5485: -- Status: Patch Available (was: Open) > Allow repeating job commit by extending OutputCommitter API > --- > > Key: MAPREDUCE-5485 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5485 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 2.1.0-beta >Reporter: Nemon Lou >Assignee: Junping Du > Attachments: MAPREDUCE-5485-demo.patch > > > There are chances MRAppMaster crush during job committing,or NodeManager > restart cause the committing AM exit due to container expire.In these cases > ,the job will fail. > However,some jobs can redo commit so failing the job becomes unnecessary. > Let clients tell AM to allow redo commit or not is a better choice. > This idea comes from Jason Lowe's comments in MAPREDUCE-4819 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
