[jira] [Updated] (MAPREDUCE-6905) Fix meaningless operations in TestDFSIO in some situation.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allen Wittenauer updated MAPREDUCE-6905:
Fix Version/s: (was: 2.9)
2.9.0
> Fix meaningless operations in TestDFSIO in some situation.
> --
>
> Key: MAPREDUCE-6905
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6905
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tools/rumen
>Affects Versions: 3.0.0-alpha4
>Reporter: LiXin Ge
>Assignee: LiXin Ge
> Labels: patch
> Fix For: 2.9.0, 3.0.0-beta1, 2.8.2
>
> Attachments: MAPREDUCE-6905.001.patch
>
>
> When run TestDFSIO in write mode with 2 million nrFiles, it will *takes
> hours* to create control files and get IOException as last because of
> directory item limit is exceeded. And, it will leave over 1 million useless
> files which will be deleted when run TestDFSIO again with acceptable nrFiles.
>
> {quote}
> 17/06/21 09:12:16 INFO fs.TestDFSIO: creating control file: 1024 bytes,
> 200 files
> java.io.IOException: The directory item limit of
> /benchmarks/TestDFSIO/io_control is exceeded: limit=1048576 items=1048576
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyMaxDirItems(FSDirectory.java:2033)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.addChild(FSDirectory.java:2084)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.addLastINode(FSDirectory.java:2053)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.addINode(FSDirectory.java:1873)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.addFile(FSDirectory.java:327)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2750)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2632)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2520)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:579)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:394)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2036)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2034)
> at org.apache.hadoop.fs.TestDFSIO.createControlFile(TestDFSIO.java:302)
> at org.apache.hadoop.fs.TestDFSIO.run(TestDFSIO.java:751)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at org.apache.hadoop.fs.TestDFSIO.main(TestDFSIO.java:650)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
> at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
> at
> org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:118)
> at
> org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:126)
> {quote}
> In brief, we'd better check the parameter of nrFiles before it waste our time
> and hurt our feelings.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6304) Specifying node labels when submitting MR jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-6304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-6304: Component/s: job submission > Specifying node labels when submitting MR jobs > -- > > Key: MAPREDUCE-6304 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6304 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: job submission >Reporter: Jian Fang >Assignee: Naganarasimha G R > Labels: mapreduce > Fix For: 2.8.0, 2.7.4, 3.0.0-alpha1 > > Attachments: MAPREDUCE-6304.20150410-1.patch, > MAPREDUCE-6304.20150411-1.patch, MAPREDUCE-6304.20150501-1.patch, > MAPREDUCE-6304.20150510-1.patch, MAPREDUCE-6304.20150511-1.patch, > MAPREDUCE-6304.20150512-1.patch, MAPREDUCE-6304-branch-2.7.patch > > > Per the discussion on YARN-796, we need a mechanism in MAPREDUCE to specify > node labels when submitting MR jobs. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6892) Issues with the count of failed/killed tasks in the jhist file
[
https://issues.apache.org/jira/browse/MAPREDUCE-6892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haibo Chen updated MAPREDUCE-6892:
--
Tags: (was: incom)
> Issues with the count of failed/killed tasks in the jhist file
> --
>
> Key: MAPREDUCE-6892
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6892
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, jobhistoryserver
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: MAPREDUCE-6892-001.patch, MAPREDUCE-6892-002.PATCH,
> MAPREDUCE-6892-003.patch, MAPREDUCE-6892-004.patch, MAPREDUCE-6892-005.patch
>
>
> Recently we encountered some issues with the value of failed tasks. After
> parsing the jhist file, {{JobInfo.getFailedMaps()}} returned 0, but actually
> there were failures.
> Another minor thing is that you cannot get the number of killed tasks
> (although this can be calculated).
> The root cause is that {{JobUnsuccessfulCompletionEvent}} contains only the
> successful map/reduce task counts. Number of failed (or killed) tasks are not
> stored.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6892) Issues with the count of failed/killed tasks in the jhist file
[
https://issues.apache.org/jira/browse/MAPREDUCE-6892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haibo Chen updated MAPREDUCE-6892:
--
Tags: incom
Hadoop Flags: Incompatible change
> Issues with the count of failed/killed tasks in the jhist file
> --
>
> Key: MAPREDUCE-6892
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6892
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, jobhistoryserver
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: MAPREDUCE-6892-001.patch, MAPREDUCE-6892-002.PATCH,
> MAPREDUCE-6892-003.patch, MAPREDUCE-6892-004.patch, MAPREDUCE-6892-005.patch
>
>
> Recently we encountered some issues with the value of failed tasks. After
> parsing the jhist file, {{JobInfo.getFailedMaps()}} returned 0, but actually
> there were failures.
> Another minor thing is that you cannot get the number of killed tasks
> (although this can be calculated).
> The root cause is that {{JobUnsuccessfulCompletionEvent}} contains only the
> successful map/reduce task counts. Number of failed (or killed) tasks are not
> stored.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6937) Backport MAPREDUCE-6870 to branch-2 while preserving compatibility
[
https://issues.apache.org/jira/browse/MAPREDUCE-6937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16144055#comment-16144055
]
Hadoop QA commented on MAPREDUCE-6937:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 14m
6s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.7 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
45s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
12s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} branch-2.7 passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green} branch-2.7 passed with JDK v1.7.0_151 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} branch-2.7 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
51s{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app in
branch-2.7 has 1 extant Findbugs warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
6s{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
in branch-2.7 has 2 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} branch-2.7 passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} branch-2.7 passed with JDK v1.7.0_151 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} the patch passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green} the patch passed with JDK v1.7.0_151 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
35s{color} | {color:red} hadoop-mapreduce-project/hadoop-mapreduce-client: The
patch generated 6 new + 1350 unchanged - 2 fixed = 1356 total (was 1352)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 129 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
0s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} the patch passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green} the patch passed with JDK v1.7.0_151 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 5m 32s{color}
| {color:red} hadoop-mapreduce-client-core in the patch failed with JDK
v1.7.0_151. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 30m 20s{color}
| {color:red} hadoop-mapreduce-client-app
[jira] [Commented] (MAPREDUCE-6892) Issues with the count of failed/killed tasks in the jhist file
[
https://issues.apache.org/jira/browse/MAPREDUCE-6892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16143964#comment-16143964
]
Haibo Chen commented on MAPREDUCE-6892:
---
Thanks for the update [~pbacsko]! The patch is very close to be ready. But I
still think we should give -1 as the default value to stand for unknown in the
schema (Events.avpr).
In TestJobHistoryParsing.testHistoryParsingForKilledAndFailedAttempts(), let's
use // for the comments you've added (/**/ is for class/methods) and remove the
nested for loop to assert RACK NAME for each of the tasks (I don't think they
are relevant). Otherwise, the patch LGTM.
> Issues with the count of failed/killed tasks in the jhist file
> --
>
> Key: MAPREDUCE-6892
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6892
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, jobhistoryserver
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: MAPREDUCE-6892-001.patch, MAPREDUCE-6892-002.PATCH,
> MAPREDUCE-6892-003.patch, MAPREDUCE-6892-004.patch, MAPREDUCE-6892-005.patch
>
>
> Recently we encountered some issues with the value of failed tasks. After
> parsing the jhist file, {{JobInfo.getFailedMaps()}} returned 0, but actually
> there were failures.
> Another minor thing is that you cannot get the number of killed tasks
> (although this can be calculated).
> The root cause is that {{JobUnsuccessfulCompletionEvent}} contains only the
> successful map/reduce task counts. Number of failed (or killed) tasks are not
> stored.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6947) Moving logging APIs over to slf4j in hadoop-mapreduce-examples
[
https://issues.apache.org/jira/browse/MAPREDUCE-6947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16143912#comment-16143912
]
Hadoop QA commented on MAPREDUCE-6947:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
51s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
30s{color} | {color:red} hadoop-mapreduce-project/hadoop-mapreduce-examples:
The patch generated 1 new + 260 unchanged - 2 fixed = 261 total (was 262)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
55s{color} | {color:green} hadoop-mapreduce-examples in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 35m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | MAPREDUCE-6947 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12883715/MAPREDUCE-6947.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 4d062d5db56c 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 02599bd |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7100/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-examples.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7100/testReport/ |
| modules | C: hadoop-mapreduce-project/hadoop-mapreduce-examples U:
hadoop-mapreduce-project/hadoop-mapreduce-examples |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7100/console |
| Powered by | Apache Yetus 0.5.0 http://yetus.apache.org |
This message was automatically generated.
> Moving logging APIs over to slf4j in hadoop-mapreduce-examples
> ---
>
> Key: MAPREDUCE-6947
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6947
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>
[jira] [Updated] (MAPREDUCE-6937) Backport MAPREDUCE-6870 to branch-2 while preserving compatibility
[ https://issues.apache.org/jira/browse/MAPREDUCE-6937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated MAPREDUCE-6937: Status: Patch Available (was: Open) > Backport MAPREDUCE-6870 to branch-2 while preserving compatibility > -- > > Key: MAPREDUCE-6937 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6937 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Zhe Zhang >Assignee: Erik Krogen > Attachments: MAPREDUCE-6870-branch-2.01.patch, > MAPREDUCE-6870-branch-2.7.03.patch, MAPREDUCE-6870_branch2.7.patch, > MAPREDUCE-6870_branch2.7v2.patch, MAPREDUCE-6870-branch-2.8.03.patch, > MAPREDUCE-6870_branch2.8.patch, MAPREDUCE-6870_branch2.8v2.patch > > > To maintain compatibility we need to disable this by default per discussion > on MAPREDUCE-6870. > Using a separate JIRA to correctly track incompatibilities. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6937) Backport MAPREDUCE-6870 to branch-2 while preserving compatibility
[ https://issues.apache.org/jira/browse/MAPREDUCE-6937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated MAPREDUCE-6937: Attachment: MAPREDUCE-6870-branch-2.7.03.patch MAPREDUCE-6870-branch-2.8.03.patch > Backport MAPREDUCE-6870 to branch-2 while preserving compatibility > -- > > Key: MAPREDUCE-6937 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6937 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Zhe Zhang >Assignee: Erik Krogen > Attachments: MAPREDUCE-6870-branch-2.01.patch, > MAPREDUCE-6870-branch-2.7.03.patch, MAPREDUCE-6870_branch2.7.patch, > MAPREDUCE-6870_branch2.7v2.patch, MAPREDUCE-6870-branch-2.8.03.patch, > MAPREDUCE-6870_branch2.8.patch, MAPREDUCE-6870_branch2.8v2.patch > > > To maintain compatibility we need to disable this by default per discussion > on MAPREDUCE-6870. > Using a separate JIRA to correctly track incompatibilities. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6937) Backport MAPREDUCE-6870 to branch-2 while preserving compatibility
[ https://issues.apache.org/jira/browse/MAPREDUCE-6937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated MAPREDUCE-6937: Attachment: MAPREDUCE-6870-branch-2.01.patch > Backport MAPREDUCE-6870 to branch-2 while preserving compatibility > -- > > Key: MAPREDUCE-6937 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6937 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Zhe Zhang >Assignee: Erik Krogen > Attachments: MAPREDUCE-6870-branch-2.01.patch, > MAPREDUCE-6870_branch2.7.patch, MAPREDUCE-6870_branch2.7v2.patch, > MAPREDUCE-6870_branch2.8.patch, MAPREDUCE-6870_branch2.8v2.patch > > > To maintain compatibility we need to disable this by default per discussion > on MAPREDUCE-6870. > Using a separate JIRA to correctly track incompatibilities. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6937) Backport MAPREDUCE-6870 to branch-2 while preserving compatibility
[
https://issues.apache.org/jira/browse/MAPREDUCE-6937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16143888#comment-16143888
]
Peter Bacsko commented on MAPREDUCE-6937:
-
I haven't clicked on Submit Patch yet. There will be a third version for branch
2.7 and branch 2.8 because {{DEFAULT_FINISH_JOB_WHEN_REDUCERS_DONE}} has to be
false as well.
> Backport MAPREDUCE-6870 to branch-2 while preserving compatibility
> --
>
> Key: MAPREDUCE-6937
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6937
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Zhe Zhang
>Assignee: Erik Krogen
> Attachments: MAPREDUCE-6870_branch2.7.patch,
> MAPREDUCE-6870_branch2.7v2.patch, MAPREDUCE-6870_branch2.8.patch,
> MAPREDUCE-6870_branch2.8v2.patch
>
>
> To maintain compatibility we need to disable this by default per discussion
> on MAPREDUCE-6870.
> Using a separate JIRA to correctly track incompatibilities.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6937) Backport MAPREDUCE-6870 to branch-2 while preserving compatibility
[ https://issues.apache.org/jira/browse/MAPREDUCE-6937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16143883#comment-16143883 ] Haibo Chen commented on MAPREDUCE-6937: --- Seems like the patch is not picked up by Jenkins. Can you also rename the patch in the format of [jira name]-[branch name].[patch version].patch, e.g. MAPREDUCE-6870-branch-2.7.02.patch? > Backport MAPREDUCE-6870 to branch-2 while preserving compatibility > -- > > Key: MAPREDUCE-6937 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6937 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Zhe Zhang >Assignee: Erik Krogen > Attachments: MAPREDUCE-6870_branch2.7.patch, > MAPREDUCE-6870_branch2.7v2.patch, MAPREDUCE-6870_branch2.8.patch, > MAPREDUCE-6870_branch2.8v2.patch > > > To maintain compatibility we need to disable this by default per discussion > on MAPREDUCE-6870. > Using a separate JIRA to correctly track incompatibilities. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6937) Backport MAPREDUCE-6870 to branch-2 while preserving compatibility
[ https://issues.apache.org/jira/browse/MAPREDUCE-6937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16143871#comment-16143871 ] Peter Bacsko commented on MAPREDUCE-6937: - Sorry I just realized that we need a patch for branch-2 as well. Will upload that soon. > Backport MAPREDUCE-6870 to branch-2 while preserving compatibility > -- > > Key: MAPREDUCE-6937 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6937 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Zhe Zhang >Assignee: Erik Krogen > Attachments: MAPREDUCE-6870_branch2.7.patch, > MAPREDUCE-6870_branch2.7v2.patch, MAPREDUCE-6870_branch2.8.patch, > MAPREDUCE-6870_branch2.8v2.patch > > > To maintain compatibility we need to disable this by default per discussion > on MAPREDUCE-6870. > Using a separate JIRA to correctly track incompatibilities. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6937) Backport MAPREDUCE-6870 to branch-2 while preserving compatibility
[ https://issues.apache.org/jira/browse/MAPREDUCE-6937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated MAPREDUCE-6937: Attachment: MAPREDUCE-6870_branch2.7v2.patch MAPREDUCE-6870_branch2.8v2.patch > Backport MAPREDUCE-6870 to branch-2 while preserving compatibility > -- > > Key: MAPREDUCE-6937 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6937 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Zhe Zhang >Assignee: Erik Krogen > Attachments: MAPREDUCE-6870_branch2.7.patch, > MAPREDUCE-6870_branch2.7v2.patch, MAPREDUCE-6870_branch2.8.patch, > MAPREDUCE-6870_branch2.8v2.patch > > > To maintain compatibility we need to disable this by default per discussion > on MAPREDUCE-6870. > Using a separate JIRA to correctly track incompatibilities. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6937) Backport MAPREDUCE-6870 to branch-2 while preserving compatibility
[ https://issues.apache.org/jira/browse/MAPREDUCE-6937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16143857#comment-16143857 ] Haibo Chen commented on MAPREDUCE-6937: --- [~pbacsko], the default value for mapreduce.job.finish-when-all-reducers-done should be false to preserve the current behavior in branch-2.7 and 2.8. > Backport MAPREDUCE-6870 to branch-2 while preserving compatibility > -- > > Key: MAPREDUCE-6937 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6937 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Zhe Zhang >Assignee: Erik Krogen > Attachments: MAPREDUCE-6870_branch2.7.patch, > MAPREDUCE-6870_branch2.8.patch > > > To maintain compatibility we need to disable this by default per discussion > on MAPREDUCE-6870. > Using a separate JIRA to correctly track incompatibilities. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6947) Moving logging APIs over to slf4j in hadoop-mapreduce-examples
[ https://issues.apache.org/jira/browse/MAPREDUCE-6947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gergely Novák updated MAPREDUCE-6947: - Status: Patch Available (was: Open) > Moving logging APIs over to slf4j in hadoop-mapreduce-examples > --- > > Key: MAPREDUCE-6947 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6947 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Gergely Novák >Assignee: Gergely Novák > Attachments: MAPREDUCE-6947.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6892) Issues with the count of failed/killed tasks in the jhist file
[
https://issues.apache.org/jira/browse/MAPREDUCE-6892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16143752#comment-16143752
]
Hadoop QA commented on MAPREDUCE-6892:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
42s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m
34s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
15s{color} | {color:red} root: The patch generated 57 new + 1072 unchanged - 32
fixed = 1129 total (was 1104) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
6s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
36s{color} | {color:red}
hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core
generated 50 new + 2496 unchanged - 0 fixed = 2546 total (was 2496) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 5m 21s{color}
| {color:red} hadoop-mapreduce-client-core in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 15m
22s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 6m
52s{color} | {color:green} hadoop-mapreduce-client-hs in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
38s{color} | {color:green} hadoop-rumen in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}106m 15s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.mapred.TestJobEndNotifier |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | MAPREDUCE-6892 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12884030/MAPREDUCE-6892-005.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 0677a3bc5bd0 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 02599bd |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checks
[jira] [Commented] (MAPREDUCE-6937) Backport MAPREDUCE-6870 to branch-2 while preserving compatibility
[ https://issues.apache.org/jira/browse/MAPREDUCE-6937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16143689#comment-16143689 ] Peter Bacsko commented on MAPREDUCE-6937: - [~xkrogen] [~haibochen] I uploaded the two patches for branch 2.7 and 2.8. > Backport MAPREDUCE-6870 to branch-2 while preserving compatibility > -- > > Key: MAPREDUCE-6937 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6937 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Zhe Zhang >Assignee: Erik Krogen > Attachments: MAPREDUCE-6870_branch2.7.patch, > MAPREDUCE-6870_branch2.8.patch > > > To maintain compatibility we need to disable this by default per discussion > on MAPREDUCE-6870. > Using a separate JIRA to correctly track incompatibilities. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6937) Backport MAPREDUCE-6870 to branch-2 while preserving compatibility
[ https://issues.apache.org/jira/browse/MAPREDUCE-6937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated MAPREDUCE-6937: Attachment: MAPREDUCE-6870_branch2.7.patch MAPREDUCE-6870_branch2.8.patch > Backport MAPREDUCE-6870 to branch-2 while preserving compatibility > -- > > Key: MAPREDUCE-6937 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6937 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Zhe Zhang >Assignee: Erik Krogen > Attachments: MAPREDUCE-6870_branch2.7.patch, > MAPREDUCE-6870_branch2.8.patch > > > To maintain compatibility we need to disable this by default per discussion > on MAPREDUCE-6870. > Using a separate JIRA to correctly track incompatibilities. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6892) Issues with the count of failed/killed tasks in the jhist file
[
https://issues.apache.org/jira/browse/MAPREDUCE-6892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated MAPREDUCE-6892:
Attachment: MAPREDUCE-6892-005.patch
> Issues with the count of failed/killed tasks in the jhist file
> --
>
> Key: MAPREDUCE-6892
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6892
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, jobhistoryserver
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: MAPREDUCE-6892-001.patch, MAPREDUCE-6892-002.PATCH,
> MAPREDUCE-6892-003.patch, MAPREDUCE-6892-004.patch, MAPREDUCE-6892-005.patch
>
>
> Recently we encountered some issues with the value of failed tasks. After
> parsing the jhist file, {{JobInfo.getFailedMaps()}} returned 0, but actually
> there were failures.
> Another minor thing is that you cannot get the number of killed tasks
> (although this can be calculated).
> The root cause is that {{JobUnsuccessfulCompletionEvent}} contains only the
> successful map/reduce task counts. Number of failed (or killed) tasks are not
> stored.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6892) Issues with the count of failed/killed tasks in the jhist file
[
https://issues.apache.org/jira/browse/MAPREDUCE-6892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16143654#comment-16143654
]
Peter Bacsko commented on MAPREDUCE-6892:
-
Uploaded patch v5 where I fixed the flaky test + the other failures.
> Issues with the count of failed/killed tasks in the jhist file
> --
>
> Key: MAPREDUCE-6892
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6892
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, jobhistoryserver
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: MAPREDUCE-6892-001.patch, MAPREDUCE-6892-002.PATCH,
> MAPREDUCE-6892-003.patch, MAPREDUCE-6892-004.patch, MAPREDUCE-6892-005.patch
>
>
> Recently we encountered some issues with the value of failed tasks. After
> parsing the jhist file, {{JobInfo.getFailedMaps()}} returned 0, but actually
> there were failures.
> Another minor thing is that you cannot get the number of killed tasks
> (although this can be calculated).
> The root cause is that {{JobUnsuccessfulCompletionEvent}} contains only the
> successful map/reduce task counts. Number of failed (or killed) tasks are not
> stored.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
