[jira] [Commented] (MAPREDUCE-6626) Reuse objectMapper instance in MapReduce
[
https://issues.apache.org/jira/browse/MAPREDUCE-6626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129941#comment-15129941
]
Hadoop QA commented on MAPREDUCE-6626:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 51s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
33s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 26s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 43s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 58s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 27s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 27s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 39s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 39s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 26s
{color} | {color:red} hadoop-mapreduce-project/hadoop-mapreduce-client: patch
generated 2 new + 338 unchanged - 0 fixed = 340 total (was 338) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 52s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 10s {color}
| {color:red} hadoop-mapreduce-client-app in the patch failed with JDK
v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 9s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 40s
{color} | {color:green} hadoop-mapreduce-client-app in the patch passed with
JDK v1.7.0_91. {color} |
| {color:green}+1{color} | {co
[jira] [Updated] (MAPREDUCE-6626) Reuse objectMapper instance in MapReduce
[
https://issues.apache.org/jira/browse/MAPREDUCE-6626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lin Yiqun updated MAPREDUCE-6626:
-
Attachment: MAPREDUCE-6626.002.patch
> Reuse objectMapper instance in MapReduce
>
>
> Key: MAPREDUCE-6626
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6626
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: performance
>Affects Versions: 2.7.1
>Reporter: Lin Yiqun
>Assignee: Lin Yiqun
> Attachments: MAPREDUCE-6626.002.patch, MAPREDUCE.001.patch
>
>
> Now in MapReduce, there are some places creating a new ObjectMapper instance
> every time. In wiki of ObjectMapper, it suggested:
> {code}
> Further: it is beneficial to use just one instance (or small number of
> instances) for data binding; many optimizations for reuse (of symbol tables,
> some buffers) depend on ObjectMapper instances being reused.
> {code}
> http://webcache.googleusercontent.com/search?q=cache:kybMTIJC6F4J:wiki.fasterxml.com/JacksonFAQ+&cd=4&hl=ja&ct=clnk&gl=jp,
> it's similar to HDFS-9724.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6626) Reuse objectMapper instance in MapReduce
[
https://issues.apache.org/jira/browse/MAPREDUCE-6626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129931#comment-15129931
]
Lin Yiqun commented on MAPREDUCE-6626:
--
Thanks [~brahmareddy]. Update the latest patch.
> Reuse objectMapper instance in MapReduce
>
>
> Key: MAPREDUCE-6626
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6626
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: performance
>Affects Versions: 2.7.1
>Reporter: Lin Yiqun
>Assignee: Lin Yiqun
> Attachments: MAPREDUCE.001.patch
>
>
> Now in MapReduce, there are some places creating a new ObjectMapper instance
> every time. In wiki of ObjectMapper, it suggested:
> {code}
> Further: it is beneficial to use just one instance (or small number of
> instances) for data binding; many optimizations for reuse (of symbol tables,
> some buffers) depend on ObjectMapper instances being reused.
> {code}
> http://webcache.googleusercontent.com/search?q=cache:kybMTIJC6F4J:wiki.fasterxml.com/JacksonFAQ+&cd=4&hl=ja&ct=clnk&gl=jp,
> it's similar to HDFS-9724.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6626) Reuse objectMapper instance in MapReduce
[
https://issues.apache.org/jira/browse/MAPREDUCE-6626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129886#comment-15129886
]
Brahma Reddy Battula commented on MAPREDUCE-6626:
-
Moved to MapReduce project..Thanks for reporting and working on this..Patch
looks good to me.
> Reuse objectMapper instance in MapReduce
>
>
> Key: MAPREDUCE-6626
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6626
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: performance
>Affects Versions: 2.7.1
>Reporter: Lin Yiqun
>Assignee: Lin Yiqun
> Attachments: MAPREDUCE.001.patch
>
>
> Now in MapReduce, there are some places creating a new ObjectMapper instance
> every time. In wiki of ObjectMapper, it suggested:
> {code}
> Further: it is beneficial to use just one instance (or small number of
> instances) for data binding; many optimizations for reuse (of symbol tables,
> some buffers) depend on ObjectMapper instances being reused.
> {code}
> http://webcache.googleusercontent.com/search?q=cache:kybMTIJC6F4J:wiki.fasterxml.com/JacksonFAQ+&cd=4&hl=ja&ct=clnk&gl=jp,
> it's similar to HDFS-9724.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Moved] (MAPREDUCE-6626) Reuse objectMapper instance in MapReduce
[
https://issues.apache.org/jira/browse/MAPREDUCE-6626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula moved HDFS-9747 to MAPREDUCE-6626:
---
Affects Version/s: (was: 2.7.1)
2.7.1
Component/s: (was: performance)
performance
Key: MAPREDUCE-6626 (was: HDFS-9747)
Project: Hadoop Map/Reduce (was: Hadoop HDFS)
> Reuse objectMapper instance in MapReduce
>
>
> Key: MAPREDUCE-6626
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6626
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: performance
>Affects Versions: 2.7.1
>Reporter: Lin Yiqun
>Assignee: Lin Yiqun
> Attachments: MAPREDUCE.001.patch
>
>
> Now in MapReduce, there are some places creating a new ObjectMapper instance
> every time. In wiki of ObjectMapper, it suggested:
> {code}
> Further: it is beneficial to use just one instance (or small number of
> instances) for data binding; many optimizations for reuse (of symbol tables,
> some buffers) depend on ObjectMapper instances being reused.
> {code}
> http://webcache.googleusercontent.com/search?q=cache:kybMTIJC6F4J:wiki.fasterxml.com/JacksonFAQ+&cd=4&hl=ja&ct=clnk&gl=jp,
> it's similar to HDFS-9724.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6537) hadoop pipes examples aren't in the mvn package tar ball
[
https://issues.apache.org/jira/browse/MAPREDUCE-6537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129407#comment-15129407
]
Hadoop QA commented on MAPREDUCE-6537:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
45s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 13s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 12s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 17s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 9s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 9s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 10s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 10s
{color} | {color:green} hadoop-assemblies in the patch passed with JDK
v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 9s
{color} | {color:green} hadoop-assemblies in the patch passed with JDK
v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 35s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12763984/HADOOP-12381.00.patch
|
| JIRA Issue | MAPREDUCE-6537 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml |
| uname | Linux a5ddd89510af 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / ccbba4a |
| Default Java | 1.7.0_91 |
| Mult
[jira] [Commented] (MAPREDUCE-6620) Jobs that did not start are shown as starting in 1969 in the JHS web UI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129379#comment-15129379 ] Hudson commented on MAPREDUCE-6620: --- SUCCESS: Integrated in Hadoop-trunk-Commit #9233 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9233/]) MAPREDUCE-6620. Jobs that did not start are shown as starting in 1969 in (rkanter: rev ccbba4a4deb8ade54a04137c993526e461bcb46e) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/dao/JobInfo.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/dao/TestJobInfo.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobsBlock.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobBlock.java * hadoop-mapreduce-project/CHANGES.txt > Jobs that did not start are shown as starting in 1969 in the JHS web UI > --- > > Key: MAPREDUCE-6620 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6620 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 2.7.2 >Reporter: Daniel Templeton >Assignee: Haibo Chen > Fix For: 2.9.0 > > Attachments: MAPREDUCE-6620.001.patch, MAPREDUCE-6620.002.patch, > mapreduce6620.001.patch, mapreduce6620.003.patch > > > If a job fails, its start time is stored as -1. The RM UI correctly handles > negative start times. The JHS UI does not, blindly converting it into a date > in 1969. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6621) Memory Leak in JobClient#submitJobInternal()
[
https://issues.apache.org/jira/browse/MAPREDUCE-6621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated MAPREDUCE-6621:
--
Target Version/s: 2.6.4
> Memory Leak in JobClient#submitJobInternal()
>
>
> Key: MAPREDUCE-6621
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6621
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Fix For: 2.7.3, 2.6.4
>
> Attachments: MAPREDUCE-6621.1.patch, MAPREDUCE-6621.2.patch
>
>
> In JobClient:
> {code}
> public RunningJob submitJobInternal(final JobConf conf)
> throws FileNotFoundException, IOException {
> try {
> conf.setBooleanIfUnset("mapred.mapper.new-api", false);
> conf.setBooleanIfUnset("mapred.reducer.new-api", false);
> Job job = clientUgi.doAs(new PrivilegedExceptionAction () {
> @Override
> public Job run() throws IOException, ClassNotFoundException,
> InterruptedException {
> Job job = Job.getInstance(conf);
> job.submit();
> return job;
> }
> });
> // update our Cluster instance with the one created by Job for
> submission
> // (we can't pass our Cluster instance to Job, since Job wraps the
> config
> // instance, and the two configs would then diverge)
> cluster = job.getCluster();
> return new NetworkedJob(job);
> } catch (InterruptedException ie) {
> throw new IOException("interrupted", ie);
> }
> }
> {code}
> We will replace the cluster object with the cluster object from Job, but the
> previous old cluster object would never be closed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6620) Jobs that did not start are shown as starting in 1969 in the JHS web UI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated MAPREDUCE-6620: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.9.0 Status: Resolved (was: Patch Available) Thanks Haibo. Committed to trunk and branch-2! > Jobs that did not start are shown as starting in 1969 in the JHS web UI > --- > > Key: MAPREDUCE-6620 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6620 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 2.7.2 >Reporter: Daniel Templeton >Assignee: Haibo Chen > Fix For: 2.9.0 > > Attachments: MAPREDUCE-6620.001.patch, MAPREDUCE-6620.002.patch, > mapreduce6620.001.patch, mapreduce6620.003.patch > > > If a job fails, its start time is stored as -1. The RM UI correctly handles > negative start times. The JHS UI does not, blindly converting it into a date > in 1969. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6537) hadoop pipes examples aren't in the mvn package tar ball
[ https://issues.apache.org/jira/browse/MAPREDUCE-6537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129329#comment-15129329 ] Kai Sasaki commented on MAPREDUCE-6537: --- [~aw] Could you check it when you are available? Thank you. > hadoop pipes examples aren't in the mvn package tar ball > > > Key: MAPREDUCE-6537 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6537 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: pipes >Affects Versions: 3.0.0 >Reporter: Allen Wittenauer >Assignee: Kai Sasaki >Priority: Blocker > Attachments: HADOOP-12381.00.patch > > > Hadoop pipes examples are built but never packaged. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6620) Jobs that did not start are shown as starting in 1969 in the JHS web UI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129312#comment-15129312 ] Robert Kanter commented on MAPREDUCE-6620: -- +1 on the latest patch. > Jobs that did not start are shown as starting in 1969 in the JHS web UI > --- > > Key: MAPREDUCE-6620 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6620 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 2.7.2 >Reporter: Daniel Templeton >Assignee: Haibo Chen > Attachments: MAPREDUCE-6620.001.patch, MAPREDUCE-6620.002.patch, > mapreduce6620.001.patch, mapreduce6620.003.patch > > > If a job fails, its start time is stored as -1. The RM UI correctly handles > negative start times. The JHS UI does not, blindly converting it into a date > in 1969. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6507) TestRMNMInfo fails intermittently
[
https://issues.apache.org/jira/browse/MAPREDUCE-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129307#comment-15129307
]
Hadoop QA commented on MAPREDUCE-6507:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
53s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 21s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 28s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
36s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 40s {color}
| {color:red} hadoop-yarn-server-tests in the patch failed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 25s {color}
| {color:red} hadoop-yarn-server-tests in the patch failed with JDK v1.7.0_91.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 35m 17s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.yarn.server.TestMiniYARNClusterForHA |
| | hadoop.yarn.server.TestContainerManagerSecurity |
| | hadoop.yarn.server.TestMiniYarnClusterNodeUtilization |
| JDK v1.7.0_91 Failed junit tests |
hadoop.yarn.server.TestMiniYARNClusterForHA |
| | hadoop.yarn.server.TestContainerManager
[jira] [Updated] (MAPREDUCE-6112) Negative shuffle time on History UI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated MAPREDUCE-6112: -- Assignee: (was: Haibo Chen) > Negative shuffle time on History UI > --- > > Key: MAPREDUCE-6112 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6112 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.4.1 >Reporter: Andrew Or > Attachments: Screen Shot 2014-09-26 at 11.31.21 AM.png > > > Just running a simple "pi" job, with 10 mappers and 100 iterations. See > screenshot. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6620) Jobs that did not start are shown as starting in 1969 in the JHS web UI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129137#comment-15129137 ] Haibo Chen commented on MAPREDUCE-6620: --- seems unrelated > Jobs that did not start are shown as starting in 1969 in the JHS web UI > --- > > Key: MAPREDUCE-6620 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6620 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 2.7.2 >Reporter: Daniel Templeton >Assignee: Haibo Chen > Attachments: MAPREDUCE-6620.001.patch, MAPREDUCE-6620.002.patch, > mapreduce6620.001.patch, mapreduce6620.003.patch > > > If a job fails, its start time is stored as -1. The RM UI correctly handles > negative start times. The JHS UI does not, blindly converting it into a date > in 1969. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6507) TestRMNMInfo fails intermittently
[
https://issues.apache.org/jira/browse/MAPREDUCE-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Badger updated MAPREDUCE-6507:
---
Target Version/s: 2.7.3
I noticed that TestMRTimelineEventHandling is failing in branch-2.7. I believe
that this patch will also fix this failing test.
> TestRMNMInfo fails intermittently
> -
>
> Key: MAPREDUCE-6507
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6507
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Rohith Sharma K S
>Assignee: Eric Badger
> Attachments: MAPREDUCE-6507.001.patch
>
>
> TestRMNMInfo fails intermittently. Below is trace for the failure
> {noformat}
> testRMNMInfo(org.apache.hadoop.mapreduce.v2.TestRMNMInfo) Time elapsed: 0.28
> sec <<< FAILURE!
> java.lang.AssertionError: Unexpected number of live nodes: expected:<4> but
> was:<3>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.mapreduce.v2.TestRMNMInfo.testRMNMInfo(TestRMNMInfo.java:111)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6507) TestRMNMInfo fails intermittently
[
https://issues.apache.org/jira/browse/MAPREDUCE-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Badger updated MAPREDUCE-6507:
---
Status: Patch Available (was: Open)
> TestRMNMInfo fails intermittently
> -
>
> Key: MAPREDUCE-6507
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6507
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Rohith Sharma K S
>Assignee: Eric Badger
> Attachments: MAPREDUCE-6507.001.patch
>
>
> TestRMNMInfo fails intermittently. Below is trace for the failure
> {noformat}
> testRMNMInfo(org.apache.hadoop.mapreduce.v2.TestRMNMInfo) Time elapsed: 0.28
> sec <<< FAILURE!
> java.lang.AssertionError: Unexpected number of live nodes: expected:<4> but
> was:<3>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.mapreduce.v2.TestRMNMInfo.testRMNMInfo(TestRMNMInfo.java:111)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6507) TestRMNMInfo fails intermittently
[
https://issues.apache.org/jira/browse/MAPREDUCE-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Badger updated MAPREDUCE-6507:
---
Attachment: MAPREDUCE-6507.001.patch
The problem is that there is a thread starting the services and the main thread
is asynchronously checking the state. Thus, the main thread checks that the
services are fully up, but they can appear to be started before they have
actually completed their startup (AbstractService sets the state to STARTED
before calling serviceStart()). I think that the easiest fix is to remove the
threads that are spawned from within the main thread that do not appear to
accomplish anything of value.
> TestRMNMInfo fails intermittently
> -
>
> Key: MAPREDUCE-6507
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6507
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Rohith Sharma K S
>Assignee: Eric Badger
> Attachments: MAPREDUCE-6507.001.patch
>
>
> TestRMNMInfo fails intermittently. Below is trace for the failure
> {noformat}
> testRMNMInfo(org.apache.hadoop.mapreduce.v2.TestRMNMInfo) Time elapsed: 0.28
> sec <<< FAILURE!
> java.lang.AssertionError: Unexpected number of live nodes: expected:<4> but
> was:<3>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.mapreduce.v2.TestRMNMInfo.testRMNMInfo(TestRMNMInfo.java:111)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6620) Jobs that did not start are shown as starting in 1969 in the JHS web UI
[
https://issues.apache.org/jira/browse/MAPREDUCE-6620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129081#comment-15129081
]
Hadoop QA commented on MAPREDUCE-6620:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
37s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 24s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
36s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 10s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m 35s
{color} | {color:green} hadoop-mapreduce-client-hs in the patch passed with JDK
v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m 4s
{color} | {color:green} hadoop-mapreduce-client-hs in the patch passed with JDK
v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 18s
{color} | {color:red} Patch generated 14 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 59s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12785856/mapreduce6620.003.patch
|
| JIRA Issue | MAPREDUCE-6620 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 6af1925b8f3b 3.1
[jira] [Updated] (MAPREDUCE-6527) Data race on field org.apache.hadoop.mapred.JobConf.credentials
[
https://issues.apache.org/jira/browse/MAPREDUCE-6527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haibo Chen updated MAPREDUCE-6527:
--
Attachment: mapreduce6527.001.patch
Minor fix
> Data race on field org.apache.hadoop.mapred.JobConf.credentials
> ---
>
> Key: MAPREDUCE-6527
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6527
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.7.1
>Reporter: Ali Kheradmand
>Assignee: Haibo Chen
> Attachments: mapreduce6527.001.patch
>
>
> I am running the test suite against a dynamic race detector called
> RV-Predict. Here is a race report that I got:
> {noformat}
> Data race on field org.apache.hadoop.mapred.JobConf.credentials: {{{
> Concurrent read in thread T327 (locks held: {})
> > at org.apache.hadoop.mapred.JobConf.getCredentials(JobConf.java:505)
> at
> org.apache.hadoop.mapreduce.task.JobContextImpl.(JobContextImpl.java:70)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:524)
> T327 is created by T22
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.(LocalJobRunner.java:218)
> Concurrent write in thread T22 (locks held: {Monitor@496c673a,
> Monitor@496319b0})
> > at org.apache.hadoop.mapred.JobConf.setCredentials(JobConf.java:510)
> at
> org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:787)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:241)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1669)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
> at
> org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:335)
> locked Monitor@496319b0 at
> org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:n/a)
>
> at
> org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:245)
> locked Monitor@496c673a at
> org.apache.hadoop.mapreduce.lib.jobcontrol.JobControl.run(JobControl.java:229)
>
> T22 is created by T1
> at
> org.apache.hadoop.mapred.jobcontrol.TestJobControl.doJobControlTest(TestJobControl.java:111)
> }}}
> {noformat}
> In the source code of org.apache.hadoop.mapreduce.JobStatus.submitJob
> function, we have the following lines:
> {code}
> Job job = new Job(JobID.downgrade(jobid), jobSubmitDir);
> job.job.setCredentials(credentials);
> {code}
> It looks a bit suspicious: Job extends thread and at the end of its
> constructor it starts a new thread which creates a new instance of
> JobContextImpl which reads credentials. However, the first thread
> concurrently sets credentials after a creating the Job instance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6620) Jobs that did not start are shown as starting in 1969 in the JHS web UI
[ https://issues.apache.org/jira/browse/MAPREDUCE-6620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated MAPREDUCE-6620: -- Attachment: mapreduce6620.003.patch Fixed the issued reported by Jenkin report > Jobs that did not start are shown as starting in 1969 in the JHS web UI > --- > > Key: MAPREDUCE-6620 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6620 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 2.7.2 >Reporter: Daniel Templeton >Assignee: Haibo Chen > Attachments: MAPREDUCE-6620.001.patch, MAPREDUCE-6620.002.patch, > mapreduce6620.001.patch, mapreduce6620.003.patch > > > If a job fails, its start time is stored as -1. The RM UI correctly handles > negative start times. The JHS UI does not, blindly converting it into a date > in 1969. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6621) Memory Leak in JobClient#submitJobInternal()
[
https://issues.apache.org/jira/browse/MAPREDUCE-6621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15128882#comment-15128882
]
Hudson commented on MAPREDUCE-6621:
---
FAILURE: Integrated in Hadoop-trunk-Commit #9228 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/9228/])
MAPREDUCE-6621. Memory Leak in JobClient#submitJobInternal(). (jlowe: rev
43e669b22d9ac0b81ab8cc397c5d6d92bac986c6)
* hadoop-mapreduce-project/CHANGES.txt
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
> Memory Leak in JobClient#submitJobInternal()
>
>
> Key: MAPREDUCE-6621
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6621
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Fix For: 2.7.3, 2.6.4
>
> Attachments: MAPREDUCE-6621.1.patch, MAPREDUCE-6621.2.patch
>
>
> In JobClient:
> {code}
> public RunningJob submitJobInternal(final JobConf conf)
> throws FileNotFoundException, IOException {
> try {
> conf.setBooleanIfUnset("mapred.mapper.new-api", false);
> conf.setBooleanIfUnset("mapred.reducer.new-api", false);
> Job job = clientUgi.doAs(new PrivilegedExceptionAction () {
> @Override
> public Job run() throws IOException, ClassNotFoundException,
> InterruptedException {
> Job job = Job.getInstance(conf);
> job.submit();
> return job;
> }
> });
> // update our Cluster instance with the one created by Job for
> submission
> // (we can't pass our Cluster instance to Job, since Job wraps the
> config
> // instance, and the two configs would then diverge)
> cluster = job.getCluster();
> return new NetworkedJob(job);
> } catch (InterruptedException ie) {
> throw new IOException("interrupted", ie);
> }
> }
> {code}
> We will replace the cluster object with the cluster object from Job, but the
> previous old cluster object would never be closed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6621) Memory Leak in JobClient#submitJobInternal()
[
https://issues.apache.org/jira/browse/MAPREDUCE-6621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated MAPREDUCE-6621:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.6.4
2.7.3
Status: Resolved (was: Patch Available)
Thanks, Xuan! I committed this to trunk, branch-2, branch-2.8, branch-2.7, and
branch-2.6.
> Memory Leak in JobClient#submitJobInternal()
>
>
> Key: MAPREDUCE-6621
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6621
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Fix For: 2.7.3, 2.6.4
>
> Attachments: MAPREDUCE-6621.1.patch, MAPREDUCE-6621.2.patch
>
>
> In JobClient:
> {code}
> public RunningJob submitJobInternal(final JobConf conf)
> throws FileNotFoundException, IOException {
> try {
> conf.setBooleanIfUnset("mapred.mapper.new-api", false);
> conf.setBooleanIfUnset("mapred.reducer.new-api", false);
> Job job = clientUgi.doAs(new PrivilegedExceptionAction () {
> @Override
> public Job run() throws IOException, ClassNotFoundException,
> InterruptedException {
> Job job = Job.getInstance(conf);
> job.submit();
> return job;
> }
> });
> // update our Cluster instance with the one created by Job for
> submission
> // (we can't pass our Cluster instance to Job, since Job wraps the
> config
> // instance, and the two configs would then diverge)
> cluster = job.getCluster();
> return new NetworkedJob(job);
> } catch (InterruptedException ie) {
> throw new IOException("interrupted", ie);
> }
> }
> {code}
> We will replace the cluster object with the cluster object from Job, but the
> previous old cluster object would never be closed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6621) Memory Leak in JobClient#submitJobInternal()
[
https://issues.apache.org/jira/browse/MAPREDUCE-6621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15128811#comment-15128811
]
Jason Lowe commented on MAPREDUCE-6621:
---
+1. The same TestCLI failure has been sporadically happening in other
precommit builds. Filed MAPREDUCE-6625.
Committing this.
> Memory Leak in JobClient#submitJobInternal()
>
>
> Key: MAPREDUCE-6621
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6621
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: MAPREDUCE-6621.1.patch, MAPREDUCE-6621.2.patch
>
>
> In JobClient:
> {code}
> public RunningJob submitJobInternal(final JobConf conf)
> throws FileNotFoundException, IOException {
> try {
> conf.setBooleanIfUnset("mapred.mapper.new-api", false);
> conf.setBooleanIfUnset("mapred.reducer.new-api", false);
> Job job = clientUgi.doAs(new PrivilegedExceptionAction () {
> @Override
> public Job run() throws IOException, ClassNotFoundException,
> InterruptedException {
> Job job = Job.getInstance(conf);
> job.submit();
> return job;
> }
> });
> // update our Cluster instance with the one created by Job for
> submission
> // (we can't pass our Cluster instance to Job, since Job wraps the
> config
> // instance, and the two configs would then diverge)
> cluster = job.getCluster();
> return new NetworkedJob(job);
> } catch (InterruptedException ie) {
> throw new IOException("interrupted", ie);
> }
> }
> {code}
> We will replace the cluster object with the cluster object from Job, but the
> previous old cluster object would never be closed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6625) TestCLI#testGetJob fails occasionally
Jason Lowe created MAPREDUCE-6625:
-
Summary: TestCLI#testGetJob fails occasionally
Key: MAPREDUCE-6625
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6625
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Reporter: Jason Lowe
Lately TestCLI has been failing sometimes in precommit builds:
{noformat}
Running org.apache.hadoop.mapreduce.tools.TestCLI
Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.883 sec <<<
FAILURE! - in org.apache.hadoop.mapreduce.tools.TestCLI
testGetJob(org.apache.hadoop.mapreduce.tools.TestCLI) Time elapsed: 0.037 sec
<<< FAILURE!
java.lang.AssertionError: null
at org.junit.Assert.fail(Assert.java:86)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertTrue(Assert.java:52)
at
org.apache.hadoop.mapreduce.tools.TestCLI.testGetJob(TestCLI.java:175)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6608) Work Preserving AM Restart for MapReduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-6608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15128785#comment-15128785 ] Srikanth Sampath commented on MAPREDUCE-6608: - I have attached the high level approach used in the solution that is in the works. I will update with a design patch and a detailed design soon. > Work Preserving AM Restart for MapReduce > > > Key: MAPREDUCE-6608 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6608 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Srikanth Sampath >Assignee: Srikanth Sampath > Attachments: WorkPreservingMRAppMaster-1.pdf, > WorkPreservingMRAppMaster.pdf > > > Providing a framework for work preserving AM is achieved in > [YARN-1489|https://issues.apache.org/jira/browse/YARN-1489]. We would like > to take advantage of this for MapReduce(MR) applications. There are some > challenges which have been described in the attached document and few options > discussed. We solicit feedback from the community. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6608) Work Preserving AM Restart for MapReduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-6608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Srikanth Sampath updated MAPREDUCE-6608: Attachment: WorkPreservingMRAppMaster-1.pdf High Level Approach > Work Preserving AM Restart for MapReduce > > > Key: MAPREDUCE-6608 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6608 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Srikanth Sampath >Assignee: Srikanth Sampath > Attachments: WorkPreservingMRAppMaster-1.pdf, > WorkPreservingMRAppMaster.pdf > > > Providing a framework for work preserving AM is achieved in > [YARN-1489|https://issues.apache.org/jira/browse/YARN-1489]. We would like > to take advantage of this for MapReduce(MR) applications. There are some > challenges which have been described in the attached document and few options > discussed. We solicit feedback from the community. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
