[jira] [Updated] (MAPREDUCE-6624) Include TeraChecksum in Hadoop mapreduce examples driver
[ https://issues.apache.org/jira/browse/MAPREDUCE-6624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Albert Chu updated MAPREDUCE-6624: -- Status: Patch Available (was: Open) Fix a checkstyle error from prior patch. Atleast 1 checkstyle error remains. It appears a style issue exists in ExampleDriver.java which is increased when staying consistent to the style in that file. > Include TeraChecksum in Hadoop mapreduce examples driver > > > Key: MAPREDUCE-6624 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6624 > Project: Hadoop Map/Reduce > Issue Type: Wish > Components: examples >Affects Versions: 2.7.1 >Reporter: Albert Chu >Priority: Minor > Attachments: MAPREDUCE-6624-branch-2.7.1.001.patch, > MAPREDUCE-6624-branch-2.7.1.002.patch, MAPREDUCE-6624.001.patch, > MAPREDUCE-6624.002.patch > > > In the mapreduce examples driver, you can run TeraGen, TeraSort, and > TeraValidate. To my surprise, TeraChecksum wasn't included in the examples > driver. I think it'd be nice to include. It can be used to compare the > checksum of the input & output. It's worth noting that TeraValidate > calculates the checksum as well. > Patch to be posted shortly. I tested against 2.7.1 branch but couldn't > against master. I am including patches for both. Patch also includes > TeraChecksum test case addition. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6621) Memory Leak in JobClient#submitJobInternal()
[
https://issues.apache.org/jira/browse/MAPREDUCE-6621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126861#comment-15126861
]
Jason Lowe commented on MAPREDUCE-6621:
---
Patch looks mostly good, just a nit that the comment is misleading. Setting
'prev' to null doesn't accomplish much -- it's just a local variable that's
just about to go out of scope anyway. The comment implies it's important that
it is set to null, but it has nothing to do with the issue. The null setting
from MAPREDUCE-6618 was important since those were instance variables that
survive the current method.
We can skip the unnecessary null, and the comment should just say it's
important to close the previous cluster instance to cleanup resources. IMHO
the debug log is overkill, but I'm OK if you want to leave it.
> Memory Leak in JobClient#submitJobInternal()
>
>
> Key: MAPREDUCE-6621
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6621
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: MAPREDUCE-6621.1.patch
>
>
> In JobClient:
> {code}
> public RunningJob submitJobInternal(final JobConf conf)
> throws FileNotFoundException, IOException {
> try {
> conf.setBooleanIfUnset("mapred.mapper.new-api", false);
> conf.setBooleanIfUnset("mapred.reducer.new-api", false);
> Job job = clientUgi.doAs(new PrivilegedExceptionAction () {
> @Override
> public Job run() throws IOException, ClassNotFoundException,
> InterruptedException {
> Job job = Job.getInstance(conf);
> job.submit();
> return job;
> }
> });
> // update our Cluster instance with the one created by Job for
> submission
> // (we can't pass our Cluster instance to Job, since Job wraps the
> config
> // instance, and the two configs would then diverge)
> cluster = job.getCluster();
> return new NetworkedJob(job);
> } catch (InterruptedException ie) {
> throw new IOException("interrupted", ie);
> }
> }
> {code}
> We will replace the cluster object with the cluster object from Job, but the
> previous old cluster object would never be closed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6622) Add capability to set JHS job cache to a task-based limit
[ https://issues.apache.org/jira/browse/MAPREDUCE-6622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126860#comment-15126860 ] Ray Chiang commented on MAPREDUCE-6622: --- While I do have some doubts about general Guava library compatibility, I don't have a lot with their Cache library. While I was trying to figure out how to implement this, I wrote up some separate code using the latest (v19) and the same API calls worked fine in the upstream tree (v11). > Add capability to set JHS job cache to a task-based limit > - > > Key: MAPREDUCE-6622 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6622 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Affects Versions: 2.7.2 >Reporter: Ray Chiang >Assignee: Ray Chiang > Labels: supportability > Attachments: MAPREDUCE-6622.001.patch, MAPREDUCE-6622.002.patch > > > When setting the property mapreduce.jobhistory.loadedjobs.cache.size the jobs > can be of varying size. This is generally not a problem when the jobs sizes > are uniform or small, but when the job sizes can be very large (say greater > than 250k tasks), then the JHS heap size can grow tremendously. > In cases, where multiple jobs are very large, then the JHS can lock up and > spend all its time in GC. However, since the cache is holding on to all the > jobs, not much heap space can be freed up. > By setting a property that sets a cap on the number of tasks allowed in the > cache and since the total number of tasks loaded is directly proportional to > the amount of heap used, this should help prevent the JHS from locking up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6624) Include TeraChecksum in Hadoop mapreduce examples driver
[
https://issues.apache.org/jira/browse/MAPREDUCE-6624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126972#comment-15126972
]
Hadoop QA commented on MAPREDUCE-6624:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 49s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 2m 36s
{color} | {color:red} root in branch-2.7.1 failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s
{color} | {color:green} branch-2.7.1 passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 13s
{color} | {color:green} branch-2.7.1 passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s {color} | {color:green} branch-2.7.1 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 30s
{color} | {color:green} branch-2.7.1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} branch-2.7.1 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s {color} | {color:green} branch-2.7.1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s
{color} | {color:green} branch-2.7.1 passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s
{color} | {color:green} branch-2.7.1 passed with JDK v1.7.0_91 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 11s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 11s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 12s
{color} | {color:red} hadoop-mapreduce-project/hadoop-mapreduce-examples: patch
generated 1 new + 34 unchanged - 0 fixed = 35 total (was 34) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 2s
{color} | {color:red} The patch has 797 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 19s
{color} | {color:red} The patch has 97 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 31s
{color} | {color:green} hadoop-mapreduce-examples in the patch passed with JDK
v1.8.0_72. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 32s
{color} | {color:green} hadoop-mapreduce-examples in the patch passed with JDK
v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 44m 39s
{color} | {color:red} Patch generated 82 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 56m 47s {color}
| {color:black} {color} |
\\
\\
|| Subsystem ||
[jira] [Updated] (MAPREDUCE-6620) Jobs that did not start are shown as starting in 1969 in the JHS web UI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated MAPREDUCE-6620: -- Attachment: MAPREDUCE-6620.002.patch Manually tested with cluster > Jobs that did not start are shown as starting in 1969 in the JHS web UI > --- > > Key: MAPREDUCE-6620 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6620 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 2.7.2 >Reporter: Daniel Templeton >Assignee: Haibo Chen > Attachments: MAPREDUCE-6620.001.patch, MAPREDUCE-6620.002.patch, > mapreduce6620.001.patch > > > If a job fails, its start time is stored as -1. The RM UI correctly handles > negative start times. The JHS UI does not, blindly converting it into a date > in 1969. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6620) Jobs that did not start are shown as starting in 1969 in the JHS web UI
[
https://issues.apache.org/jira/browse/MAPREDUCE-6620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127485#comment-15127485
]
Hadoop QA commented on MAPREDUCE-6620:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
55s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 18s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 24s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
35s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 12s
{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs:
patch generated 1 new + 71 unchanged - 0 fixed = 72 total (was 71) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 5m 36s {color}
| {color:red} hadoop-mapreduce-client-hs in the patch failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 5m 57s {color}
| {color:red} hadoop-mapreduce-client-hs in the patch failed with JDK
v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 18s
{color} | {color:red} Patch generated 14 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 24s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesJobsQuery |
| | hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesJobs |
| JDK v1.7.0_91 Failed junit tests |
[jira] [Updated] (MAPREDUCE-6618) YarnClientProtocolProvider leaking the YarnClient thread.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated MAPREDUCE-6618:
--
Target Version/s: 2.6.4
> YarnClientProtocolProvider leaking the YarnClient thread.
> --
>
> Key: MAPREDUCE-6618
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6618
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Fix For: 2.7.3, 2.6.4
>
> Attachments: MAPREDUCE-6618.1.patch, MAPREDUCE-6618.2.patch,
> MAPREDUCE-6618.3.patch, MAPREDUCE-6618.4.patch, MAPREDUCE-6618.5.patch,
> MAPREDUCE-6618.6.patch
>
>
> YarnClientProtocolProvider creates YarnRunner which includes
> ResourceMgrDelegate. In ResourceMgrDelegate, we would initiate and start
> yarnclient. The yarnClient thread would be leaked due to
> {code}
> @Override
> public void close(ClientProtocol clientProtocol) throws IOException {
> // nothing to do
> }
> {code} in YarnClientProtocolProvider
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-6623) TestRMNMInfo and TestNetworkedJob fails in trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe resolved MAPREDUCE-6623.
---
Resolution: Duplicate
Resolving as a duplicate per the previous comment.
> TestRMNMInfo and TestNetworkedJob fails in trunk
>
>
> Key: MAPREDUCE-6623
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6623
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Xuan Gong
>Assignee: Eric Badger
>
> TestRMNMInfo:
> {code}
> Running org.apache.hadoop.mapreduce.v2.TestRMNMInfo
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 32.347 sec
> <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.TestRMNMInfo
> testRMNMInfo(org.apache.hadoop.mapreduce.v2.TestRMNMInfo) Time elapsed:
> 1.572 sec <<< FAILURE!
> java.lang.AssertionError: Unexpected number of live nodes: expected:<4> but
> was:<0>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.mapreduce.v2.TestRMNMInfo.testRMNMInfo(TestRMNMInfo.java:111)
> {code}
> TestNetworkedJob
> {code}
> testNetworkedJob:174 expected:<[[Thu Jan 28 22:41:20 + 2016] Application
> is Activated, waiting for resources to be assigned for AM. Details : AM
> Partition = ; Partition Resource = vCores:16> ; Queue's Absolute capacity = 100.0 % ; Queue's Absolute used
> capacity = 0.0 % ; Queue's Absolute max capacity = 100.0 % ; ]> but was:<[]>
> TestRMNMInfo.testRMNMInfo:111 Unexpected number of live nodes: expected:<4>
> but was:<0>
> {code}
> JDK version: JDK v1.8.0_66
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6618) YarnClientProtocolProvider leaking the YarnClient thread.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated MAPREDUCE-6618:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.6.4
2.7.3
Status: Resolved (was: Patch Available)
Thanks, Xuan! I committed this to trunk, branch-2, branch-2.8, branch-2.7, and
branch-2.6.
> YarnClientProtocolProvider leaking the YarnClient thread.
> --
>
> Key: MAPREDUCE-6618
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6618
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Fix For: 2.7.3, 2.6.4
>
> Attachments: MAPREDUCE-6618.1.patch, MAPREDUCE-6618.2.patch,
> MAPREDUCE-6618.3.patch, MAPREDUCE-6618.4.patch, MAPREDUCE-6618.5.patch,
> MAPREDUCE-6618.6.patch
>
>
> YarnClientProtocolProvider creates YarnRunner which includes
> ResourceMgrDelegate. In ResourceMgrDelegate, we would initiate and start
> yarnclient. The yarnClient thread would be leaked due to
> {code}
> @Override
> public void close(ClientProtocol clientProtocol) throws IOException {
> // nothing to do
> }
> {code} in YarnClientProtocolProvider
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6618) YarnClientProtocolProvider leaking the YarnClient thread.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126471#comment-15126471
]
Hudson commented on MAPREDUCE-6618:
---
FAILURE: Integrated in Hadoop-trunk-Commit #9215 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/9215/])
MAPREDUCE-6618. YarnClientProtocolProvider leaking the YarnClient (jlowe: rev
59a212b6e1265adfa9b55c71b65a22157dfccf77)
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientCache.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YarnClientProtocolProvider.java
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestYarnClientProtocolProvider.java
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java
> YarnClientProtocolProvider leaking the YarnClient thread.
> --
>
> Key: MAPREDUCE-6618
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6618
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Fix For: 2.7.3, 2.6.4
>
> Attachments: MAPREDUCE-6618.1.patch, MAPREDUCE-6618.2.patch,
> MAPREDUCE-6618.3.patch, MAPREDUCE-6618.4.patch, MAPREDUCE-6618.5.patch,
> MAPREDUCE-6618.6.patch
>
>
> YarnClientProtocolProvider creates YarnRunner which includes
> ResourceMgrDelegate. In ResourceMgrDelegate, we would initiate and start
> yarnclient. The yarnClient thread would be leaked due to
> {code}
> @Override
> public void close(ClientProtocol clientProtocol) throws IOException {
> // nothing to do
> }
> {code} in YarnClientProtocolProvider
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6621) Memory Leak in JobClient#submitJobInternal()
[
https://issues.apache.org/jira/browse/MAPREDUCE-6621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126572#comment-15126572
]
Xuan Gong commented on MAPREDUCE-6621:
--
[~jlowe] Could you review this please ?
> Memory Leak in JobClient#submitJobInternal()
>
>
> Key: MAPREDUCE-6621
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6621
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: MAPREDUCE-6621.1.patch
>
>
> In JobClient:
> {code}
> public RunningJob submitJobInternal(final JobConf conf)
> throws FileNotFoundException, IOException {
> try {
> conf.setBooleanIfUnset("mapred.mapper.new-api", false);
> conf.setBooleanIfUnset("mapred.reducer.new-api", false);
> Job job = clientUgi.doAs(new PrivilegedExceptionAction () {
> @Override
> public Job run() throws IOException, ClassNotFoundException,
> InterruptedException {
> Job job = Job.getInstance(conf);
> job.submit();
> return job;
> }
> });
> // update our Cluster instance with the one created by Job for
> submission
> // (we can't pass our Cluster instance to Job, since Job wraps the
> config
> // instance, and the two configs would then diverge)
> cluster = job.getCluster();
> return new NetworkedJob(job);
> } catch (InterruptedException ie) {
> throw new IOException("interrupted", ie);
> }
> }
> {code}
> We will replace the cluster object with the cluster object from Job, but the
> previous old cluster object would never be closed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (MAPREDUCE-6507) TestRMNMInfo fails intermittently
[
https://issues.apache.org/jira/browse/MAPREDUCE-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Badger reassigned MAPREDUCE-6507:
--
Assignee: Eric Badger
> TestRMNMInfo fails intermittently
> -
>
> Key: MAPREDUCE-6507
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6507
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Rohith Sharma K S
>Assignee: Eric Badger
>
> TestRMNMInfo fails intermittently. Below is trace for the failure
> {noformat}
> testRMNMInfo(org.apache.hadoop.mapreduce.v2.TestRMNMInfo) Time elapsed: 0.28
> sec <<< FAILURE!
> java.lang.AssertionError: Unexpected number of live nodes: expected:<4> but
> was:<3>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.mapreduce.v2.TestRMNMInfo.testRMNMInfo(TestRMNMInfo.java:111)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6624) Include TeraChecksum in Hadoop mapreduce examples driver
[ https://issues.apache.org/jira/browse/MAPREDUCE-6624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Albert Chu updated MAPREDUCE-6624: -- Attachment: MAPREDUCE-6624-branch-2.7.1.002.patch MAPREDUCE-6624.002.patch > Include TeraChecksum in Hadoop mapreduce examples driver > > > Key: MAPREDUCE-6624 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6624 > Project: Hadoop Map/Reduce > Issue Type: Wish > Components: examples >Affects Versions: 2.7.1 >Reporter: Albert Chu >Priority: Minor > Attachments: MAPREDUCE-6624-branch-2.7.1.001.patch, > MAPREDUCE-6624-branch-2.7.1.002.patch, MAPREDUCE-6624.001.patch, > MAPREDUCE-6624.002.patch > > > In the mapreduce examples driver, you can run TeraGen, TeraSort, and > TeraValidate. To my surprise, TeraChecksum wasn't included in the examples > driver. I think it'd be nice to include. It can be used to compare the > checksum of the input & output. It's worth noting that TeraValidate > calculates the checksum as well. > Patch to be posted shortly. I tested against 2.7.1 branch but couldn't > against master. I am including patches for both. Patch also includes > TeraChecksum test case addition. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6621) Memory Leak in JobClient#submitJobInternal()
[
https://issues.apache.org/jira/browse/MAPREDUCE-6621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127776#comment-15127776
]
Hadoop QA commented on MAPREDUCE-6621:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
44s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 21s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 26s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 32s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 48s {color}
| {color:red} hadoop-mapreduce-client-core in the patch failed with JDK
v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 8s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 9s {color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests | hadoop.mapreduce.tools.TestCLI |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
[jira] [Updated] (MAPREDUCE-6617) flushTimer in JobHistoryEventHandler should purge canceled flushTimerTask
[ https://issues.apache.org/jira/browse/MAPREDUCE-6617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6617: - Status: Open (was: Patch Available) Cancelling patch for further discussion. > flushTimer in JobHistoryEventHandler should purge canceled flushTimerTask > - > > Key: MAPREDUCE-6617 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6617 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 3.0.0 >Reporter: Kousuke Saruta >Assignee: Kousuke Saruta > Attachments: MAPREDUCE-6617.0.patch > > > In JobHistoryEventHandler, flushTask is not purged after it's canceled so GC > never sweep flushTask. > It can cause memory leak. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6621) Memory Leak in JobClient#submitJobInternal()
[
https://issues.apache.org/jira/browse/MAPREDUCE-6621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127745#comment-15127745
]
Xuan Gong commented on MAPREDUCE-6621:
--
Thanks for the comments. [~jlowe]
Attached a new patch to address the comments.
> Memory Leak in JobClient#submitJobInternal()
>
>
> Key: MAPREDUCE-6621
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6621
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: MAPREDUCE-6621.1.patch, MAPREDUCE-6621.2.patch
>
>
> In JobClient:
> {code}
> public RunningJob submitJobInternal(final JobConf conf)
> throws FileNotFoundException, IOException {
> try {
> conf.setBooleanIfUnset("mapred.mapper.new-api", false);
> conf.setBooleanIfUnset("mapred.reducer.new-api", false);
> Job job = clientUgi.doAs(new PrivilegedExceptionAction () {
> @Override
> public Job run() throws IOException, ClassNotFoundException,
> InterruptedException {
> Job job = Job.getInstance(conf);
> job.submit();
> return job;
> }
> });
> // update our Cluster instance with the one created by Job for
> submission
> // (we can't pass our Cluster instance to Job, since Job wraps the
> config
> // instance, and the two configs would then diverge)
> cluster = job.getCluster();
> return new NetworkedJob(job);
> } catch (InterruptedException ie) {
> throw new IOException("interrupted", ie);
> }
> }
> {code}
> We will replace the cluster object with the cluster object from Job, but the
> previous old cluster object would never be closed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6554) MRAppMaster servicestart failing with NPE in MRAppMaster#parsePreviousJobHistory
[
https://issues.apache.org/jira/browse/MAPREDUCE-6554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated MAPREDUCE-6554:
--
Target Version/s: 2.6.4
> MRAppMaster servicestart failing with NPE in
> MRAppMaster#parsePreviousJobHistory
> -
>
> Key: MAPREDUCE-6554
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6554
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Critical
> Fix For: 2.7.3, 2.6.4
>
> Attachments: 0001-MAPREDUCE-6554.patch, 0002-MAPREDUCE-6554.patch,
> 0003-MAPREDUCE-6554.patch, MAPREDUCE-6554-branch-2.7.001.patch,
> MAPREDUCE-6554-branch-2.7.patch
>
>
> Create scenario so that MR app master gets preempted.
> On next MRAppMaster launch tried to recover previous job history file
> {{MRAppMaster#parsePreviousJobHistory}}
> {noformat}
> 2015-11-21 13:52:27,722 INFO [main]
> org.apache.hadoop.service.AbstractService: Service
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state STARTED;
> cause: java.lang.NullPointerException
> java.lang.NullPointerException
> at java.io.StringReader.(StringReader.java:50)
> at org.apache.avro.Schema$Parser.parse(Schema.java:917)
> at org.apache.avro.Schema.parse(Schema.java:966)
> at
> org.apache.hadoop.mapreduce.jobhistory.EventReader.(EventReader.java:75)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:139)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.parsePreviousJobHistory(MRAppMaster.java:1256)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.processRecovery(MRAppMaster.java:1225)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1087)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1570)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1673)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1566)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1499)
> 2015-11-21 13:52:27,725 INFO [main]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping
> JobHistoryEventHandler. Size of the outstanding queue size is 0
> {noformat}
> EventReader(EventReader stream)
> {noformat}
> this.version = in.readLine();
> ...
> Schema myschema = new
> SpecificData(Event.class.getClassLoader()).getSchema(Event.class);
> this.schema = Schema.parse(in.readLine());
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6363) [NNBench] Lease mismatch error when running with multiple mappers
[ https://issues.apache.org/jira/browse/MAPREDUCE-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-6363: -- Target Version/s: 2.8.0, 2.6.4 (was: 2.8.0) > [NNBench] Lease mismatch error when running with multiple mappers > - > > Key: MAPREDUCE-6363 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6363 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: benchmarks >Reporter: Brahma Reddy Battula >Assignee: Bibin A Chundatt >Priority: Critical > Fix For: 2.8.0, 2.7.3, 2.6.4 > > Attachments: HDFS4929.patch, MAPREDUCE-6363-001.patch, > MAPREDUCE-6363-002.patch, MAPREDUCE-6363-003.patch, MAPREDUCE-6363-004.patch, > MAPREDUCE-6363-005.patch, MAPREDUCE-6363-006.patch, MAPREDUCE-6363-007.patch, > MAPREDUCE-6363-008.patch, nnbench.log > > > Command : > ./yarn jar > ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.0.1-tests.jar > nnbench -operation create_write -numberOfFiles 1000 -blockSize 268435456 > -bytesToWrite 102400 -baseDir /benchmarks/NNBench`hostname -s` > -replicationFactorPerFile 3 -maps 100 -reduces 10 > Trace : > 013-06-21 10:44:53,763 INFO org.apache.hadoop.ipc.Server: IPC Server handler > 7 on 9005, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from > 192.168.105.214:36320: error: > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: Lease mismatch > on /benchmarks/NNBenchlinux-185/data/file_linux-214__0 owned by > DFSClient_attempt_1371782327901_0001_m_48_0_1383437860_1 but is accessed > by DFSClient_attempt_1371782327901_0001_m_84_0_1880545303_1 > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: Lease mismatch > on /benchmarks/NNBenchlinux-185/data/file_linux-214__0 owned by > DFSClient_attempt_1371782327901_0001_m_48_0_1383437860_1 but is accessed > by DFSClient_attempt_1371782327901_0001_m_84_0_1880545303_1 > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2351) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2098) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2019) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:501) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:213) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:52012) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:435) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:925) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1710) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1706) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (MAPREDUCE-6112) Negative shuffle time on History UI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen reassigned MAPREDUCE-6112: - Assignee: Haibo Chen > Negative shuffle time on History UI > --- > > Key: MAPREDUCE-6112 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6112 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.4.1 >Reporter: Andrew Or >Assignee: Haibo Chen > Attachments: Screen Shot 2014-09-26 at 11.31.21 AM.png > > > Just running a simple "pi" job, with 10 mappers and 100 iterations. See > screenshot. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6621) Memory Leak in JobClient#submitJobInternal()
[
https://issues.apache.org/jira/browse/MAPREDUCE-6621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuan Gong updated MAPREDUCE-6621:
-
Attachment: MAPREDUCE-6621.2.patch
> Memory Leak in JobClient#submitJobInternal()
>
>
> Key: MAPREDUCE-6621
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6621
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: MAPREDUCE-6621.1.patch, MAPREDUCE-6621.2.patch
>
>
> In JobClient:
> {code}
> public RunningJob submitJobInternal(final JobConf conf)
> throws FileNotFoundException, IOException {
> try {
> conf.setBooleanIfUnset("mapred.mapper.new-api", false);
> conf.setBooleanIfUnset("mapred.reducer.new-api", false);
> Job job = clientUgi.doAs(new PrivilegedExceptionAction () {
> @Override
> public Job run() throws IOException, ClassNotFoundException,
> InterruptedException {
> Job job = Job.getInstance(conf);
> job.submit();
> return job;
> }
> });
> // update our Cluster instance with the one created by Job for
> submission
> // (we can't pass our Cluster instance to Job, since Job wraps the
> config
> // instance, and the two configs would then diverge)
> cluster = job.getCluster();
> return new NetworkedJob(job);
> } catch (InterruptedException ie) {
> throw new IOException("interrupted", ie);
> }
> }
> {code}
> We will replace the cluster object with the cluster object from Job, but the
> previous old cluster object would never be closed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5982) Task attempts that fail from the ASSIGNED state can disappear
[ https://issues.apache.org/jira/browse/MAPREDUCE-5982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-5982: -- Target Version/s: 2.6.4 > Task attempts that fail from the ASSIGNED state can disappear > - > > Key: MAPREDUCE-5982 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5982 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 0.23.10, 2.2.1, 2.7.1 >Reporter: Jason Lowe >Assignee: Chang Li > Fix For: 2.7.2, 2.6.4 > > Attachments: MAPREDUCE-5982.2.patch, MAPREDUCE-5982.3.patch, > MAPREDUCE-5982.4.patch, MAPREDUCE-5982.5.patch, MAPREDUCE-5982.6.patch, > MAPREDUCE-5982.branch.2.7.patch, MAPREDUCE-5982.patch > > > If a task attempt fails in the ASSIGNED state, e.g.: container launch fails, > then it can disappear from the job history. The task overview page will show > subsequent attempts but the attempt in question is simply missing. For > example attempt ID 1 appears but the attempt ID 0 is missing. Similarly in > the job overview page the task attempt doesn't appear in any of the > failed/killed/succeeded counts or pages. It's as if the task attempt never > existed, but the AM logs show otherwise. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (MAPREDUCE-6527) Data race on field org.apache.hadoop.mapred.JobConf.credentials
[
https://issues.apache.org/jira/browse/MAPREDUCE-6527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haibo Chen reassigned MAPREDUCE-6527:
-
Assignee: Haibo Chen
> Data race on field org.apache.hadoop.mapred.JobConf.credentials
> ---
>
> Key: MAPREDUCE-6527
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6527
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Ali Kheradmand
>Assignee: Haibo Chen
>
> I am running the test suite against a dynamic race detector called
> RV-Predict. Here is a race report that I got:
> {noformat}
> Data race on field org.apache.hadoop.mapred.JobConf.credentials: {{{
> Concurrent read in thread T327 (locks held: {})
> > at org.apache.hadoop.mapred.JobConf.getCredentials(JobConf.java:505)
> at
> org.apache.hadoop.mapreduce.task.JobContextImpl.(JobContextImpl.java:70)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:524)
> T327 is created by T22
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.(LocalJobRunner.java:218)
> Concurrent write in thread T22 (locks held: {Monitor@496c673a,
> Monitor@496319b0})
> > at org.apache.hadoop.mapred.JobConf.setCredentials(JobConf.java:510)
> at
> org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:787)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:241)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1669)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
> at
> org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335)
> locked Monitor@496319b0 at
> org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:n/a)
>
> at
> org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:245)
> locked Monitor@496c673a at
> org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:229)
>
> T22 is created by T1
> at
> org.apache.hadoop.mapred.jobcontrol.TestJobControl.doJobControlTest(TestJobControl.java:111)
> }}}
> {noformat}
> In the source code of org.apache.hadoop.mapreduce.JobStatus.submitJob
> function, we have the following lines:
> {code}
> Job job = new Job(JobID.downgrade(jobid), jobSubmitDir);
> job.job.setCredentials(credentials);
> {code}
> It looks a bit suspicious: Job extends thread and at the end of its
> constructor it starts a new thread which creates a new instance of
> JobContextImpl which reads credentials. However, the first thread
> concurrently sets credentials after a creating the Job instance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6618) YarnClientProtocolProvider leaking the YarnClient thread.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126449#comment-15126449
]
Jason Lowe commented on MAPREDUCE-6618:
---
+1 lgtm. Committing this.
> YarnClientProtocolProvider leaking the YarnClient thread.
> --
>
> Key: MAPREDUCE-6618
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6618
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: MAPREDUCE-6618.1.patch, MAPREDUCE-6618.2.patch,
> MAPREDUCE-6618.3.patch, MAPREDUCE-6618.4.patch, MAPREDUCE-6618.5.patch,
> MAPREDUCE-6618.6.patch
>
>
> YarnClientProtocolProvider creates YarnRunner which includes
> ResourceMgrDelegate. In ResourceMgrDelegate, we would initiate and start
> yarnclient. The yarnClient thread would be leaked due to
> {code}
> @Override
> public void close(ClientProtocol clientProtocol) throws IOException {
> // nothing to do
> }
> {code} in YarnClientProtocolProvider
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
