[jira] [Commented] (MAPREDUCE-3115) OOM When the value for the property mapred.map.multithreadedrunner.class is set to MultithreadedMapper instance.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525135#comment-14525135
]
Hadoop QA commented on MAPREDUCE-3115:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 41s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 30s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 35s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 20s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 2 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 32s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 56s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 1m 34s | Tests passed in
hadoop-mapreduce-client-core. |
| {color:green}+1{color} | mapreduce tests | 106m 51s | Tests passed in
hadoop-mapreduce-client-jobclient. |
| | | 145m 58s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12555686/MAPREDUCE-3115.2.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| whitespace |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5584/artifact/patchprocess/whitespace.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5584/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| hadoop-mapreduce-client-jobclient test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5584/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5584/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5584/console |
This message was automatically generated.
OOM When the value for the property mapred.map.multithreadedrunner.class is
set to MultithreadedMapper instance.
--
Key: MAPREDUCE-3115
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3115
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv1
Affects Versions: 0.23.0, 1.0.0
Environment: NA
Reporter: Bhallamudi Venkata Siva Kamesh
Attachments: MAPREDUCE-3115.2.patch, MAPREDUCE-3115.patch
When we set the value for the property *mapred.map.multithreadedrunner.class*
as instance of MultithreadedMapper, using
MultithreadedMapper.setMapperClass(), it simply throws
IllegalArgumentException.
But when we set the same property, using job's conf object using
job.getConfiguration().setClass(*mapred.map.multithreadedrunner.class*,
MultithreadedMapper.class, Mapper.class), throws OOM.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-2632) Avoid calling the partitioner when the numReduceTasks is 1.
[
https://issues.apache.org/jira/browse/MAPREDUCE-2632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525167#comment-14525167
]
Hadoop QA commented on MAPREDUCE-2632:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 28s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:red}-1{color} | javac | 7m 29s | The applied patch generated 127
additional warning messages. |
| {color:green}+1{color} | javadoc | 9m 35s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 21s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 48s | The applied patch generated 2
new checkstyle issues (total was 14, now 16). |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 13s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 1m 36s | Tests passed in
hadoop-mapreduce-client-core. |
| | | 37m 39s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12493554/MAPREDUCE-2632-1.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| javac |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5588/artifact/patchprocess/diffJavacWarnings.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5588/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5588/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5588/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5588/console |
This message was automatically generated.
Avoid calling the partitioner when the numReduceTasks is 1.
---
Key: MAPREDUCE-2632
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2632
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: tasktracker
Affects Versions: 0.23.0
Reporter: Ravi Teja Ch N V
Assignee: Ravi Teja Ch N V
Attachments: MAPREDUCE-2632-1.patch, MAPREDUCE-2632.patch
We can avoid the call to the partitioner when the number of reducers is
1.This will avoid the unnecessary computations by the partitioner.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-2632) Avoid calling the partitioner when the numReduceTasks is 1.
[
https://issues.apache.org/jira/browse/MAPREDUCE-2632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525104#comment-14525104
]
Hadoop QA commented on MAPREDUCE-2632:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 15m 6s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:red}-1{color} | javac | 7m 44s | The applied patch generated 127
additional warning messages. |
| {color:green}+1{color} | javadoc | 9m 52s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 47s | The applied patch generated 2
new checkstyle issues (total was 15, now 17). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 37s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 16s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 1m 45s | Tests passed in
hadoop-mapreduce-client-core. |
| | | 39m 8s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12493554/MAPREDUCE-2632-1.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| javac |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5583/artifact/patchprocess/diffJavacWarnings.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5583/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5583/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5583/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5583/console |
This message was automatically generated.
Avoid calling the partitioner when the numReduceTasks is 1.
---
Key: MAPREDUCE-2632
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2632
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: tasktracker
Affects Versions: 0.23.0
Reporter: Ravi Teja Ch N V
Assignee: Ravi Teja Ch N V
Attachments: MAPREDUCE-2632-1.patch, MAPREDUCE-2632.patch
We can avoid the call to the partitioner when the number of reducers is
1.This will avoid the unnecessary computations by the partitioner.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4216) Make MultipleOutputs generic to support non-file output formats
[
https://issues.apache.org/jira/browse/MAPREDUCE-4216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525147#comment-14525147
]
Hadoop QA commented on MAPREDUCE-4216:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 38s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 29s | There were no new javac warning
messages. |
| {color:red}-1{color} | javadoc | 9m 36s | The applied patch generated 1
additional warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 46s | The applied patch generated 3
new checkstyle issues (total was 67, now 70). |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 4 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 36s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 14s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 1m 34s | Tests passed in
hadoop-mapreduce-client-core. |
| | | 37m 52s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12525460/MAPREDUCE-4216.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| javadoc |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5585/artifact/patchprocess/diffJavadocWarnings.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5585/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5585/artifact/patchprocess/whitespace.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5585/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5585/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5585/console |
This message was automatically generated.
Make MultipleOutputs generic to support non-file output formats
---
Key: MAPREDUCE-4216
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4216
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Affects Versions: 1.0.2
Reporter: Robbie Strickland
Labels: Output
Attachments: MAPREDUCE-4216.patch
The current MultipleOutputs implementation is tied to FileOutputFormat in
such a way that it is not extensible to other types of output. It should be
made more generic, such as with an interface that can be implemented for
different outputs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4840) Delete dead code and deprecate public API related to skipping bad records
[
https://issues.apache.org/jira/browse/MAPREDUCE-4840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525127#comment-14525127
]
Hadoop QA commented on MAPREDUCE-4840:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 53s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:red}-1{color} | javac | 7m 33s | The applied patch generated 164
additional warning messages. |
| {color:green}+1{color} | javadoc | 9m 51s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 0s | The applied patch generated 8
new checkstyle issues (total was 613, now 531). |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 2m 2s | The patch appears to introduce 1
new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 1m 36s | Tests passed in
hadoop-mapreduce-client-core. |
| {color:green}+1{color} | mapreduce tests | 101m 49s | Tests passed in
hadoop-mapreduce-client-jobclient. |
| | | 141m 33s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-mapreduce-client-core |
| | Should org.apache.hadoop.mapred.MapTask$TrackedRecordReader be a _static_
inner class? At MapTask.java:inner class? At MapTask.java:[lines 151-228] |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12555682/MAPREDUCE-4840.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| javac |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5582/artifact/patchprocess/diffJavacWarnings.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5582/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5582/artifact/patchprocess/whitespace.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5582/artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5582/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| hadoop-mapreduce-client-jobclient test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5582/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5582/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5582/console |
This message was automatically generated.
Delete dead code and deprecate public API related to skipping bad records
-
Key: MAPREDUCE-4840
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4840
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.0.0-alpha
Reporter: Mostafa Elhemali
Priority: Minor
Attachments: MAPREDUCE-4840.patch
It looks like the decision was made in MAPREDUCE-1932 to remove support for
skipping bad records rather than fix it (it doesn't work right now in trunk).
If that's the case then we should probably delete all the dead code related
to it and deprecate the public API's for it right?
Dead code I'm talking about:
1. Task class: skipping, skipRanges, writeSkipRecs
2. MapTask class: SkippingRecordReader inner class
3. ReduceTask class: SkippingReduceValuesIterator inner class
4. Tests: TestBadRecords
Public API:
1. SkipBadRecords class
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-3115) OOM When the value for the property mapred.map.multithreadedrunner.class is set to MultithreadedMapper instance.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525228#comment-14525228
]
Hadoop QA commented on MAPREDUCE-3115:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 37s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 30s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 34s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 22s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 2 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 57s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 1m 34s | Tests passed in
hadoop-mapreduce-client-core. |
| {color:green}+1{color} | mapreduce tests | 106m 42s | Tests passed in
hadoop-mapreduce-client-jobclient. |
| | | 145m 49s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12555686/MAPREDUCE-3115.2.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| whitespace |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5587/artifact/patchprocess/whitespace.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5587/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| hadoop-mapreduce-client-jobclient test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5587/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5587/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5587/console |
This message was automatically generated.
OOM When the value for the property mapred.map.multithreadedrunner.class is
set to MultithreadedMapper instance.
--
Key: MAPREDUCE-3115
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3115
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv1
Affects Versions: 0.23.0, 1.0.0
Environment: NA
Reporter: Bhallamudi Venkata Siva Kamesh
Attachments: MAPREDUCE-3115.2.patch, MAPREDUCE-3115.patch
When we set the value for the property *mapred.map.multithreadedrunner.class*
as instance of MultithreadedMapper, using
MultithreadedMapper.setMapperClass(), it simply throws
IllegalArgumentException.
But when we set the same property, using job's conf object using
job.getConfiguration().setClass(*mapred.map.multithreadedrunner.class*,
MultithreadedMapper.class, Mapper.class), throws OOM.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6345) Documentation fix for when CRLA is enabled for MRAppMaster logs
[ https://issues.apache.org/jira/browse/MAPREDUCE-6345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525216#comment-14525216 ] Hudson commented on MAPREDUCE-6345: --- FAILURE: Integrated in Hadoop-Yarn-trunk #915 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/915/]) MAPREDUCE-6345. Documentation fix for when CRLA is enabled for MRAppMaster logs. (Rohit Agarwal via gera) (gera: rev f1a152cc0adc071277c80637ea6f5faa0bf06a1a) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * hadoop-mapreduce-project/CHANGES.txt Documentation fix for when CRLA is enabled for MRAppMaster logs --- Key: MAPREDUCE-6345 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6345 Project: Hadoop Map/Reduce Issue Type: Bug Components: documentation Affects Versions: 2.4.0, 2.5.0, 2.6.0, 2.7.0 Reporter: Rohit Agarwal Assignee: Rohit Agarwal Priority: Trivial Fix For: 2.8.0 Attachments: MAPREDUCE-6345.patch CRLA is enabled for the ApplicationMaster when both yarn.app.mapreduce.am.container.log.limit.kb (not mapreduce.task.userlog.limit.kb) and yarn.app.mapreduce.am.container.log.backups are greater than zero. This was changed in MAPREDUCE-5773. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6345) Documentation fix for when CRLA is enabled for MRAppMaster logs
[ https://issues.apache.org/jira/browse/MAPREDUCE-6345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525284#comment-14525284 ] Hudson commented on MAPREDUCE-6345: --- SUCCESS: Integrated in Hadoop-Hdfs-trunk #2113 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2113/]) MAPREDUCE-6345. Documentation fix for when CRLA is enabled for MRAppMaster logs. (Rohit Agarwal via gera) (gera: rev f1a152cc0adc071277c80637ea6f5faa0bf06a1a) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * hadoop-mapreduce-project/CHANGES.txt Documentation fix for when CRLA is enabled for MRAppMaster logs --- Key: MAPREDUCE-6345 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6345 Project: Hadoop Map/Reduce Issue Type: Bug Components: documentation Affects Versions: 2.4.0, 2.5.0, 2.6.0, 2.7.0 Reporter: Rohit Agarwal Assignee: Rohit Agarwal Priority: Trivial Fix For: 2.8.0 Attachments: MAPREDUCE-6345.patch CRLA is enabled for the ApplicationMaster when both yarn.app.mapreduce.am.container.log.limit.kb (not mapreduce.task.userlog.limit.kb) and yarn.app.mapreduce.am.container.log.backups are greater than zero. This was changed in MAPREDUCE-5773. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6345) Documentation fix for when CRLA is enabled for MRAppMaster logs
[ https://issues.apache.org/jira/browse/MAPREDUCE-6345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525199#comment-14525199 ] Hudson commented on MAPREDUCE-6345: --- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #181 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/181/]) MAPREDUCE-6345. Documentation fix for when CRLA is enabled for MRAppMaster logs. (Rohit Agarwal via gera) (gera: rev f1a152cc0adc071277c80637ea6f5faa0bf06a1a) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml Documentation fix for when CRLA is enabled for MRAppMaster logs --- Key: MAPREDUCE-6345 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6345 Project: Hadoop Map/Reduce Issue Type: Bug Components: documentation Affects Versions: 2.4.0, 2.5.0, 2.6.0, 2.7.0 Reporter: Rohit Agarwal Assignee: Rohit Agarwal Priority: Trivial Fix For: 2.8.0 Attachments: MAPREDUCE-6345.patch CRLA is enabled for the ApplicationMaster when both yarn.app.mapreduce.am.container.log.limit.kb (not mapreduce.task.userlog.limit.kb) and yarn.app.mapreduce.am.container.log.backups are greater than zero. This was changed in MAPREDUCE-5773. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6345) Documentation fix for when CRLA is enabled for MRAppMaster logs
[ https://issues.apache.org/jira/browse/MAPREDUCE-6345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525271#comment-14525271 ] Hudson commented on MAPREDUCE-6345: --- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #172 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/172/]) MAPREDUCE-6345. Documentation fix for when CRLA is enabled for MRAppMaster logs. (Rohit Agarwal via gera) (gera: rev f1a152cc0adc071277c80637ea6f5faa0bf06a1a) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml Documentation fix for when CRLA is enabled for MRAppMaster logs --- Key: MAPREDUCE-6345 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6345 Project: Hadoop Map/Reduce Issue Type: Bug Components: documentation Affects Versions: 2.4.0, 2.5.0, 2.6.0, 2.7.0 Reporter: Rohit Agarwal Assignee: Rohit Agarwal Priority: Trivial Fix For: 2.8.0 Attachments: MAPREDUCE-6345.patch CRLA is enabled for the ApplicationMaster when both yarn.app.mapreduce.am.container.log.limit.kb (not mapreduce.task.userlog.limit.kb) and yarn.app.mapreduce.am.container.log.backups are greater than zero. This was changed in MAPREDUCE-5773. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4840) Delete dead code and deprecate public API related to skipping bad records
[
https://issues.apache.org/jira/browse/MAPREDUCE-4840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525227#comment-14525227
]
Hadoop QA commented on MAPREDUCE-4840:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 31s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:red}-1{color} | javac | 7m 29s | The applied patch generated 164
additional warning messages. |
| {color:green}+1{color} | javadoc | 9m 34s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 7s | The applied patch generated 8
new checkstyle issues (total was 613, now 531). |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 1m 59s | The patch appears to introduce 1
new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 1m 36s | Tests passed in
hadoop-mapreduce-client-core. |
| {color:green}+1{color} | mapreduce tests | 104m 6s | Tests passed in
hadoop-mapreduce-client-jobclient. |
| | | 143m 12s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-mapreduce-client-core |
| | Should org.apache.hadoop.mapred.MapTask$TrackedRecordReader be a _static_
inner class? At MapTask.java:inner class? At MapTask.java:[lines 151-228] |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12555682/MAPREDUCE-4840.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| javac |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5586/artifact/patchprocess/diffJavacWarnings.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5586/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5586/artifact/patchprocess/whitespace.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5586/artifact/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5586/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| hadoop-mapreduce-client-jobclient test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5586/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5586/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5586/console |
This message was automatically generated.
Delete dead code and deprecate public API related to skipping bad records
-
Key: MAPREDUCE-4840
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4840
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.0.0-alpha
Reporter: Mostafa Elhemali
Priority: Minor
Attachments: MAPREDUCE-4840.patch
It looks like the decision was made in MAPREDUCE-1932 to remove support for
skipping bad records rather than fix it (it doesn't work right now in trunk).
If that's the case then we should probably delete all the dead code related
to it and deprecate the public API's for it right?
Dead code I'm talking about:
1. Task class: skipping, skipRanges, writeSkipRecs
2. MapTask class: SkippingRecordReader inner class
3. ReduceTask class: SkippingReduceValuesIterator inner class
4. Tests: TestBadRecords
Public API:
1. SkipBadRecords class
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6345) Documentation fix for when CRLA is enabled for MRAppMaster logs
[ https://issues.apache.org/jira/browse/MAPREDUCE-6345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525297#comment-14525297 ] Hudson commented on MAPREDUCE-6345: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #182 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/182/]) MAPREDUCE-6345. Documentation fix for when CRLA is enabled for MRAppMaster logs. (Rohit Agarwal via gera) (gera: rev f1a152cc0adc071277c80637ea6f5faa0bf06a1a) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml Documentation fix for when CRLA is enabled for MRAppMaster logs --- Key: MAPREDUCE-6345 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6345 Project: Hadoop Map/Reduce Issue Type: Bug Components: documentation Affects Versions: 2.4.0, 2.5.0, 2.6.0, 2.7.0 Reporter: Rohit Agarwal Assignee: Rohit Agarwal Priority: Trivial Fix For: 2.8.0 Attachments: MAPREDUCE-6345.patch CRLA is enabled for the ApplicationMaster when both yarn.app.mapreduce.am.container.log.limit.kb (not mapreduce.task.userlog.limit.kb) and yarn.app.mapreduce.am.container.log.backups are greater than zero. This was changed in MAPREDUCE-5773. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6345) Documentation fix for when CRLA is enabled for MRAppMaster logs
[ https://issues.apache.org/jira/browse/MAPREDUCE-6345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525316#comment-14525316 ] Hudson commented on MAPREDUCE-6345: --- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2131 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2131/]) MAPREDUCE-6345. Documentation fix for when CRLA is enabled for MRAppMaster logs. (Rohit Agarwal via gera) (gera: rev f1a152cc0adc071277c80637ea6f5faa0bf06a1a) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * hadoop-mapreduce-project/CHANGES.txt Documentation fix for when CRLA is enabled for MRAppMaster logs --- Key: MAPREDUCE-6345 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6345 Project: Hadoop Map/Reduce Issue Type: Bug Components: documentation Affects Versions: 2.4.0, 2.5.0, 2.6.0, 2.7.0 Reporter: Rohit Agarwal Assignee: Rohit Agarwal Priority: Trivial Fix For: 2.8.0 Attachments: MAPREDUCE-6345.patch CRLA is enabled for the ApplicationMaster when both yarn.app.mapreduce.am.container.log.limit.kb (not mapreduce.task.userlog.limit.kb) and yarn.app.mapreduce.am.container.log.backups are greater than zero. This was changed in MAPREDUCE-5773. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6304) Specifying node labels when submitting MR jobs
[
https://issues.apache.org/jira/browse/MAPREDUCE-6304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525326#comment-14525326
]
Naganarasimha G R commented on MAPREDUCE-6304:
--
Thanks for reviewing [~wangda],
bq. If specify label= in mapred-default.xml, which means the job will always
set label= So I suggest to change the default to be NOT SPECIFIED
(which is not a valid label since it contains space) as default value, and we
will replace it by null.
Good catch, but AFAIK we need not do so much handling and if we just remove
{{value/value}} in mapred-default.xml then the value will be taken as null
by xml parser in the Configuration class. while i am testing in the multi node
cluster will verify this and then update the patch.
bq. could you deploy a few nodes cluster, with 2-3 labels, and run MR job with
am.label, mapper.label, reducer.label, etc. that will be very important to make
sure everything works well.
Will try to get this tested on monday ( currenty only left with laptop :) ).
Test case failures doesn't seem to be related to the patch based on logs.
Specifying node labels when submitting MR jobs
--
Key: MAPREDUCE-6304
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6304
Project: Hadoop Map/Reduce
Issue Type: New Feature
Reporter: Jian Fang
Assignee: Naganarasimha G R
Fix For: 2.8.0
Attachments: MAPREDUCE-6304.20150410-1.patch,
MAPREDUCE-6304.20150411-1.patch, MAPREDUCE-6304.20150501-1.patch
Per the discussion on YARN-796, we need a mechanism in MAPREDUCE to specify
node labels when submitting MR jobs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5374) CombineFileRecordReader does not set map.input.* configuration parameters for first file read
[
https://issues.apache.org/jira/browse/MAPREDUCE-5374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525375#comment-14525375
]
Hadoop QA commented on MAPREDUCE-5374:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 33s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 32s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 31s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 21s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 45s | The applied patch generated 2
new checkstyle issues (total was 17, now 19). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 14s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 1m 34s | Tests passed in
hadoop-mapreduce-client-core. |
| | | 37m 39s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12590890/MAPREDUCE-5374.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5592/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5592/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5592/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5592/console |
This message was automatically generated.
CombineFileRecordReader does not set map.input.* configuration parameters
for first file read
---
Key: MAPREDUCE-5374
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5374
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 1.2.0
Reporter: Dave Beech
Assignee: Dave Beech
Attachments: MAPREDUCE-5374.patch, MAPREDUCE-5374.patch
The CombineFileRecordReader operates on splits consisting of multiple files.
Each time a new record reader is initialised for a chunk, certain
parameters are supposed to be set on the configuration object
(map.input.file, map.input.start and map.input.length)
However, the first reader is initialised in a different way to subsequent
ones (i.e. initialize is called by the MapTask directly rather than from
inside the record reader class). Because of this, these config parameters are
not set properly and are returned as null when you access them from inside a
mapper.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6351) Reducer hung in copy phase.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laxman updated MAPREDUCE-6351:
--
Attachment: thread-dumps.out
reducer-container-partial.log.zip
jstat-gc.log
Attached the logs (container log, thread dumps, jstat output) for reference.
Please note that, my thoughts on threading issue may be premature and
incorrect. Irrespective of this analysis problem exists.
Reducer hung in copy phase.
---
Key: MAPREDUCE-6351
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6351
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.6.0
Reporter: Laxman
Attachments: jstat-gc.log, reducer-container-partial.log.zip,
thread-dumps.out
*Problem*
Reducer gets stuck in copy phase and doesn't make progress for very long
time. After killing this task for couple of times manually, it gets
completed.
*Analysis*
- Verfied gc logs. Found no memory related issues. Attache
- Verified thread dumps. Found no thread related problems.
- On verification of logs, fetcher threads are not copying the map outputs
and they are just waiting for merge to happen.
- Merge thread is alive and in wait state.
On careful observation of logs, thread dumps and code, this looks to me like
a classic case of multi-threading issue. Thread goes to wait state after it
has been notified.
Here is the suspect code flow.
*Thread #1*
Fetcher thread - notification comes first
org.apache.hadoop.mapreduce.task.reduce.MergeThread.startMerge(SetT)
{code}
synchronized(pendingToBeMerged) {
pendingToBeMerged.addLast(toMergeInputs);
pendingToBeMerged.notifyAll();
}
{code}
*Thread #2*
Merge Thread - goes to wait state (Notification goes unconsumed)
org.apache.hadoop.mapreduce.task.reduce.MergeThread.run()
{code}
synchronized (pendingToBeMerged) {
while(pendingToBeMerged.size() = 0) {
pendingToBeMerged.wait();
}
// Pickup the inputs to merge.
inputs = pendingToBeMerged.removeFirst();
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-3517) map.input.path is null at the first split when use CombieFileInputFormat
[
https://issues.apache.org/jira/browse/MAPREDUCE-3517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525419#comment-14525419
]
Hadoop QA commented on MAPREDUCE-3517:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 34s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 29s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 37s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 53s | The applied patch generated 1
new checkstyle issues (total was 17, now 18). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 14s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 1m 35s | Tests passed in
hadoop-mapreduce-client-core. |
| | | 37m 53s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12729949/MAPREDUCE-3517.02.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 6ae2a0d |
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5596/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5596/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5596/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5596/console |
This message was automatically generated.
map.input.path is null at the first split when use CombieFileInputFormat
---
Key: MAPREDUCE-3517
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3517
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: task
Affects Versions: 0.20.203.0
Reporter: wanbin
Attachments: CombineFileRecordReader.diff, MAPREDUCE-3517.02.patch
map.input.path is null at the first split when use CombieFileInputFormat.
because in runNewMapper function, mapContext instead of taskContext which is
set map.input.path. so we need set map.input.path again to mapContext
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5225) SplitSampler in mapreduce.lib should use a SPLIT_STEP to jump around splits
[
https://issues.apache.org/jira/browse/MAPREDUCE-5225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525392#comment-14525392
]
Hadoop QA commented on MAPREDUCE-5225:
--
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 36s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 31s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 32s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 22s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 54s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 1m 34s | Tests passed in
hadoop-mapreduce-client-core. |
| {color:green}+1{color} | mapreduce tests | 108m 31s | Tests passed in
hadoop-mapreduce-client-jobclient. |
| | | 147m 31s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12582421/MAPREDUCE-5225.1.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5589/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| hadoop-mapreduce-client-jobclient test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5589/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5589/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5589/console |
This message was automatically generated.
SplitSampler in mapreduce.lib should use a SPLIT_STEP to jump around splits
---
Key: MAPREDUCE-5225
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5225
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Attachments: MAPREDUCE-5225.1.patch
Now, SplitSampler only samples the first maxSplitsSampled splits, caused by
MAPREDUCE-1820. However, jumping around all splits is in general preferable
than the first N splits.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5227) JobTrackerMetricsSource and QueueMetrics should standardize naming rules
[
https://issues.apache.org/jira/browse/MAPREDUCE-5227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525393#comment-14525393
]
Hadoop QA commented on MAPREDUCE-5227:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12582492/MAPREDUCE-5227-branch-1.1.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | branch-1 / 5f5138e |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5594/console |
This message was automatically generated.
JobTrackerMetricsSource and QueueMetrics should standardize naming rules
Key: MAPREDUCE-5227
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5227
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv1
Affects Versions: 1.1.3, 1.2.1
Reporter: Tsuyoshi Ozawa
Assignee: Tsuyoshi Ozawa
Priority: Minor
Attachments: MAPREDUCE-5227-1.1-branch.1.patch,
MAPREDUCE-5227-branch-1.1.patch, MAPREDUCE-5227.1.patch
JobTrackerMetricsSource and QueueMetrics provides users with some metrics,
but its naming rules( jobs_running, running_maps, running_reduces)
sometimes confuses users. It should be standardized.
One concern is backward compatibility, so one idea is to share
MetricMutableGaugeInt object from old and new property name.
e.g. to share runningMaps from running_maps and maps_running.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6351) Reducer hung in copy phase.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Laxman updated MAPREDUCE-6351:
--
Description:
*Problem*
Reducer gets stuck in copy phase and doesn't make progress for very long time.
After killing this task for couple of times manually, it gets completed.
*Observations*
- Verfied gc logs. Found no memory related issues. Attached the logs.
- Verified thread dumps. Found no thread related problems.
- On verification of logs, fetcher threads are not copying the map outputs and
they are just waiting for merge to happen.
- Merge thread is alive and in wait state.
*Analysis*
On careful observation of logs, thread dumps and code, this looks to me like a
classic case of multi-threading issue. Thread goes to wait state after it has
been notified.
Here is the suspect code flow.
*Thread #1*
Fetcher thread - notification comes first
org.apache.hadoop.mapreduce.task.reduce.MergeThread.startMerge(SetT)
{code}
synchronized(pendingToBeMerged) {
pendingToBeMerged.addLast(toMergeInputs);
pendingToBeMerged.notifyAll();
}
{code}
*Thread #2*
Merge Thread - goes to wait state (Notification goes unconsumed)
org.apache.hadoop.mapreduce.task.reduce.MergeThread.run()
{code}
synchronized (pendingToBeMerged) {
while(pendingToBeMerged.size() = 0) {
pendingToBeMerged.wait();
}
// Pickup the inputs to merge.
inputs = pendingToBeMerged.removeFirst();
}
{code}
was:
*Problem*
Reducer gets stuck in copy phase and doesn't make progress for very long time.
After killing this task for couple of times manually, it gets completed.
*Analysis*
- Verfied gc logs. Found no memory related issues. Attache
- Verified thread dumps. Found no thread related problems.
- On verification of logs, fetcher threads are not copying the map outputs and
they are just waiting for merge to happen.
- Merge thread is alive and in wait state.
On careful observation of logs, thread dumps and code, this looks to me like a
classic case of multi-threading issue. Thread goes to wait state after it has
been notified.
Here is the suspect code flow.
*Thread #1*
Fetcher thread - notification comes first
org.apache.hadoop.mapreduce.task.reduce.MergeThread.startMerge(SetT)
{code}
synchronized(pendingToBeMerged) {
pendingToBeMerged.addLast(toMergeInputs);
pendingToBeMerged.notifyAll();
}
{code}
*Thread #2*
Merge Thread - goes to wait state (Notification goes unconsumed)
org.apache.hadoop.mapreduce.task.reduce.MergeThread.run()
{code}
synchronized (pendingToBeMerged) {
while(pendingToBeMerged.size() = 0) {
pendingToBeMerged.wait();
}
// Pickup the inputs to merge.
inputs = pendingToBeMerged.removeFirst();
}
{code}
Reducer hung in copy phase.
---
Key: MAPREDUCE-6351
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6351
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.6.0
Reporter: Laxman
Attachments: jstat-gc.log, reducer-container-partial.log.zip,
thread-dumps.out
*Problem*
Reducer gets stuck in copy phase and doesn't make progress for very long
time. After killing this task for couple of times manually, it gets
completed.
*Observations*
- Verfied gc logs. Found no memory related issues. Attached the logs.
- Verified thread dumps. Found no thread related problems.
- On verification of logs, fetcher threads are not copying the map outputs
and they are just waiting for merge to happen.
- Merge thread is alive and in wait state.
*Analysis*
On careful observation of logs, thread dumps and code, this looks to me like
a classic case of multi-threading issue. Thread goes to wait state after it
has been notified.
Here is the suspect code flow.
*Thread #1*
Fetcher thread - notification comes first
org.apache.hadoop.mapreduce.task.reduce.MergeThread.startMerge(SetT)
{code}
synchronized(pendingToBeMerged) {
pendingToBeMerged.addLast(toMergeInputs);
pendingToBeMerged.notifyAll();
}
{code}
*Thread #2*
Merge Thread - goes to wait state (Notification goes unconsumed)
org.apache.hadoop.mapreduce.task.reduce.MergeThread.run()
{code}
synchronized (pendingToBeMerged) {
while(pendingToBeMerged.size() = 0) {
pendingToBeMerged.wait();
}
// Pickup the inputs to merge.
inputs = pendingToBeMerged.removeFirst();
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6351) Reducer hung in copy phase.
Laxman created MAPREDUCE-6351:
-
Summary: Reducer hung in copy phase.
Key: MAPREDUCE-6351
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6351
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.6.0
Reporter: Laxman
*Problem*
Reducer gets stuck in copy phase and doesn't make progress for very long time.
After killing this task for couple of times manually, it gets completed.
*Analysis*
- Verfied gc logs. Found no memory related issues. Attache
- Verified thread dumps. Found no thread related problems.
- On verification of logs, fetcher threads are not copying the map outputs and
they are just waiting for merge to happen.
- Merge thread is alive and in wait state.
On careful observation of logs, thread dumps and code, this looks to me like a
classic case of multi-threading issue. Thread goes to wait state after it has
been notified.
Here is the suspect code flow.
*Thread #1*
Fetcher thread - notification comes first
org.apache.hadoop.mapreduce.task.reduce.MergeThread.startMerge(SetT)
{code}
synchronized(pendingToBeMerged) {
pendingToBeMerged.addLast(toMergeInputs);
pendingToBeMerged.notifyAll();
}
{code}
*Thread #2*
Merge Thread - goes to wait state (Notification goes unconsumed)
org.apache.hadoop.mapreduce.task.reduce.MergeThread.run()
{code}
synchronized (pendingToBeMerged) {
while(pendingToBeMerged.size() = 0) {
pendingToBeMerged.wait();
}
// Pickup the inputs to merge.
inputs = pendingToBeMerged.removeFirst();
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5499) Fix synchronization issues of the setters/getters of *PBImpl which take in/return lists
[
https://issues.apache.org/jira/browse/MAPREDUCE-5499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525377#comment-14525377
]
Hadoop QA commented on MAPREDUCE-5499:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 15m 3s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 37s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 47s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 38s | The applied patch generated 2
new checkstyle issues (total was 91, now 92). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 0s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 0m 46s | Tests passed in
hadoop-mapreduce-client-common. |
| | | 37m 25s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12602406/MAPREDUCE-5499.2.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 57d9a97 |
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5593/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-common.txt
|
| hadoop-mapreduce-client-common test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5593/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5593/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5593/console |
This message was automatically generated.
Fix synchronization issues of the setters/getters of *PBImpl which take
in/return lists
---
Key: MAPREDUCE-5499
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5499
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Zhijie Shen
Assignee: Xuan Gong
Attachments: MAPREDUCE-5499.1.patch, MAPREDUCE-5499.2.patch
Similar to YARN-609. There're the following *PBImpls which need to be fixed:
1. GetDiagnosticsResponsePBImpl
2. GetTaskAttemptCompletionEventsResponsePBImpl
3. GetTaskReportsResposnePBImpl
4. CounterGroupPBImpl
5. JobReportPBImpl
6. TaskReportPBImpl
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5577) Allow querying the JobHistoryServer by job arrival time
[ https://issues.apache.org/jira/browse/MAPREDUCE-5577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525416#comment-14525416 ] Hadoop QA commented on MAPREDUCE-5577: -- (!) The patch artifact directory has been removed! This is a fatal error for test-patch.sh. Aborting. Jenkins (node H3) information at https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5595/ may provide some hints. Allow querying the JobHistoryServer by job arrival time --- Key: MAPREDUCE-5577 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5577 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobhistoryserver Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-5577.patch The JobHistoryServer REST APIs currently allow querying by job submit time and finish time. However, jobs don't necessarily arrive in order of their finish time, meaning that a client who wants to stay on top of all completed jobs needs to query large time intervals to make sure they're not missing anything. Exposing functionality to allow querying by the time a job lands at the JobHistoryServer would allow clients to set the start of their query interval to the time of their last query. The arrival time of a job would be defined as the time that it lands in the done directory and can be picked up using the last modified date on history files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6352) test-patch.sh goes into an infinite loop on non-maven builds
Allen Wittenauer created MAPREDUCE-6352: --- Summary: test-patch.sh goes into an infinite loop on non-maven builds Key: MAPREDUCE-6352 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6352 Project: Hadoop Map/Reduce Issue Type: Test Components: test Reporter: Allen Wittenauer Priority: Critical If post HADOOP-11746 test patch is given a non-maven-based build, it goes into an infinite loop looking for modules pom.xml. There should be an escape clause after switching branches to see if it is maven based. If it is not maven based, then test-patch should either abort or re-exec using that version's test-patch script. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4487) Reduce job latency by removing hardcoded sleep statements
[ https://issues.apache.org/jira/browse/MAPREDUCE-4487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525390#comment-14525390 ] Hadoop QA commented on MAPREDUCE-4487: -- (!) The patch artifact directory has been removed! This is a fatal error for test-patch.sh. Aborting. Jenkins (node H3) information at https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5590/ may provide some hints. Reduce job latency by removing hardcoded sleep statements - Key: MAPREDUCE-4487 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4487 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1, mrv2, performance Affects Versions: 1.0.3, 2.0.0-alpha Reporter: Tom White Assignee: Tom White Attachments: MAPREDUCE-4487-mr2.patch, MAPREDUCE-4487.patch There are a few places in MapReduce where there are hardcoded sleep statements. By replacing them with wait/notify or similar it's possible to reduce latency for short running jobs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3517) map.input.path is null at the first split when use CombieFileInputFormat
[ https://issues.apache.org/jira/browse/MAPREDUCE-3517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-3517: Attachment: MAPREDUCE-3517.02.patch This is a re-upload of the current patch in the proper format. It still needs to be reviewed. map.input.path is null at the first split when use CombieFileInputFormat --- Key: MAPREDUCE-3517 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3517 Project: Hadoop Map/Reduce Issue Type: Bug Components: task Affects Versions: 0.20.203.0 Reporter: wanbin Attachments: CombineFileRecordReader.diff, MAPREDUCE-3517.02.patch map.input.path is null at the first split when use CombieFileInputFormat. because in runNewMapper function, mapContext instead of taskContext which is set map.input.path. so we need set map.input.path again to mapContext -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5203) Make AM of M/R Use NMClient
[
https://issues.apache.org/jira/browse/MAPREDUCE-5203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525345#comment-14525345
]
Hadoop QA commented on MAPREDUCE-5203:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 15m 12s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 42s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 55s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 17s | The applied patch generated 1
new checkstyle issues (total was 497, now 498). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 39s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 12s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 9m 20s | Tests passed in
hadoop-mapreduce-client-app. |
| {color:green}+1{color} | mapreduce tests | 1m 38s | Tests passed in
hadoop-mapreduce-client-core. |
| | | 49m 54s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12590068/MAPREDUCE-5203.5.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5591/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
|
| hadoop-mapreduce-client-app test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5591/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5591/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5591/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5591/console |
This message was automatically generated.
Make AM of M/R Use NMClient
---
Key: MAPREDUCE-5203
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5203
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Attachments: MAPREDUCE-5203.1.patch, MAPREDUCE-5203.2.patch,
MAPREDUCE-5203.3.patch, MAPREDUCE-5203.4.patch, MAPREDUCE-5203.5.patch
YARN-422 adds NMClient. AM of mapreduce should use it instead of using the
raw ContainerManager proxy directly. ContainerLauncherImpl needs to be
changed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6279) AM should explicity exit JVM after all services have stopped
[
https://issues.apache.org/jira/browse/MAPREDUCE-6279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne updated MAPREDUCE-6279:
--
Attachment: MAPREDUCE-6279.v2.txt
[~jlowe], thank you very much for your help on this patch.
The way the first patch was written, it caused many of the tests in
hadoop-mapreduce-client-app to fail because they are instantiating the
{{MRAppMaster}} object and invoking {{shutDownJob}} (either directly or
inderectly through {{JobFinishEventHandler}}) without actually starting the
main thread of {{MRAppMaster}}.
In order to alleviate the need to modify several tests, version 2 of the patch
puts the ExitUtil object into test mode if the {{MRAppMaster}} main thread
has not been started.
AM should explicity exit JVM after all services have stopped
Key: MAPREDUCE-6279
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6279
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 2.5.0
Reporter: Jason Lowe
Assignee: Eric Payne
Attachments: MAPREDUCE-6279.v1.txt, MAPREDUCE-6279.v2.txt
Occasionally the MapReduce AM can get stuck trying to shut down.
MAPREDUCE-6049 and MAPREDUCE-5888 were specific instances that have been
fixed, but this can also occur with uber jobs if the task code inadvertently
leaves non-daemon threads lingering.
We should explicitly shutdown the JVM after the MapReduce AM has unregistered
and all services have been stopped.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5700) historyServer can't show container's log when aggregation is not enabled
[
https://issues.apache.org/jira/browse/MAPREDUCE-5700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525480#comment-14525480
]
Hadoop QA commented on MAPREDUCE-5700:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 34s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 35s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 53s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 26s | The applied patch generated 4
new checkstyle issues (total was 41, now 44). |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 36s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 0m 43s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 5m 43s | Tests passed in
hadoop-mapreduce-client-hs. |
| | | 41m 28s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12620602/yarn-647-2.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 6ae2a0d |
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5598/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-hs.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5598/artifact/patchprocess/whitespace.txt
|
| hadoop-mapreduce-client-hs test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5598/artifact/patchprocess/testrun_hadoop-mapreduce-client-hs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5598/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5598/console |
This message was automatically generated.
historyServer can't show container's log when aggregation is not enabled
Key: MAPREDUCE-5700
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5700
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 0.23.7, 2.0.4-alpha, 2.2.0
Environment: yarn.log-aggregation-enable=false , HistoryServer will
show like this:
Aggregation is not enabled. Try the nodemanager at hd13-vm1:34669
Reporter: Hong Shen
Assignee: Hong Shen
Attachments: yarn-647-2.patch, yarn-647.patch
When yarn.log-aggregation-enable is seted to false, after a MR_App complete,
we can't view the container's log from the HistoryServer, it shows message
like:
Aggregation is not enabled. Try the nodemanager at hd13-vm1:34669
Since we don't want to aggregate the container's log, because it will be a
pressure to namenode. but sometimes we also want to take a look at
container's log.
Should we show the container's log across HistoryServer even if
yarn.log-aggregation-enable is seted to false.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5248) Let NNBenchWithoutMR specify the replication factor for its test
[ https://issues.apache.org/jira/browse/MAPREDUCE-5248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-5248: Attachment: MAPREDUCE-5248.patch Let NNBenchWithoutMR specify the replication factor for its test Key: MAPREDUCE-5248 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5248 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client, test Affects Versions: 3.0.0 Reporter: Erik Paulson Assignee: Erik Paulson Priority: Minor Attachments: MAPREDUCE-5248.patch, MAPREDUCE-5248.txt Original Estimate: 1h Remaining Estimate: 1h The NNBenchWithoutMR test creates files with a replicationFactorPerFile hard-coded to 1. It'd be nice to be able to specify that on the commandline. Also, it'd be great if MAPREDUCE-4750 was merged along with this fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5708) Duplicate String.format in getSpillFileForWrite
[
https://issues.apache.org/jira/browse/MAPREDUCE-5708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525483#comment-14525483
]
Hadoop QA commented on MAPREDUCE-5708:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 39s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 29s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 35s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 35s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 0m 57s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 9m 12s | Tests passed in
hadoop-mapreduce-client-app. |
| | | 44m 58s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12621475/0001-Removed-duplicate-String.format.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 6ae2a0d |
| hadoop-mapreduce-client-app test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5599/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5599/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5599/console |
This message was automatically generated.
Duplicate String.format in getSpillFileForWrite
---
Key: MAPREDUCE-5708
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5708
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Konstantin Weitz
Priority: Minor
Attachments: 0001-Removed-duplicate-String.format.patch
Original Estimate: 10m
Remaining Estimate: 10m
The code responsible for formatting the spill file name (namely
_getSpillFileForWrite_) unnecessarily calls _String.format_ twice. This does
not only affect performance, but leads to the weird requirement that task
attempt ids cannot contain _%_ characters (because these would be interpreted
as format specifiers in the outside _String.format_ call).
I assume this was done by mistake, as it could only be useful if task attempt
ids contained _%n_.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-1380) Adaptive Scheduler
[
https://issues.apache.org/jira/browse/MAPREDUCE-1380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525548#comment-14525548
]
Hadoop QA commented on MAPREDUCE-1380:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12630631/MAPREDUCE-1380-branch-1.2.patch
|
| Optional Tests | shellcheck javadoc javac unit findbugs checkstyle |
| git revision | branch-1 / 5f5138e |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5607/console |
This message was automatically generated.
Adaptive Scheduler
--
Key: MAPREDUCE-1380
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1380
Project: Hadoop Map/Reduce
Issue Type: New Feature
Affects Versions: 2.4.1
Reporter: Jordà Polo
Priority: Minor
Attachments: MAPREDUCE-1380-branch-1.2.patch,
MAPREDUCE-1380_0.1.patch, MAPREDUCE-1380_1.1.patch, MAPREDUCE-1380_1.1.pdf
The Adaptive Scheduler is a pluggable Hadoop scheduler that automatically
adjusts the amount of used resources depending on the performance of jobs and
on user-defined high-level business goals.
Existing Hadoop schedulers are focused on managing large, static clusters in
which nodes are added or removed manually. On the other hand, the goal of
this scheduler is to improve the integration of Hadoop and the applications
that run on top of it with environments that allow a more dynamic
provisioning of resources.
The current implementation is quite straightforward. Users specify a deadline
at job submission time, and the scheduler adjusts the resources to meet that
deadline (at the moment, the scheduler can be configured to either minimize
or maximize the amount of resources). If multiple jobs are run
simultaneously, the scheduler prioritizes them by deadline. Note that the
current approach to estimate the completion time of jobs is quite simplistic:
it is based on the time it takes to finish each task, so it works well with
regular jobs, but there is still room for improvement for unpredictable jobs.
The idea is to further integrate it with cloud-like and virtual environments
(such as Amazon EC2, Emotive, etc.) so that if, for instance, a job isn't
able to meet its deadline, the scheduler automatically requests more
resources.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5876) SequenceFileRecordReader NPE if close() is called before initialize()
[
https://issues.apache.org/jira/browse/MAPREDUCE-5876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525547#comment-14525547
]
Hadoop QA commented on MAPREDUCE-5876:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 15m 11s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:red}-1{color} | javac | 7m 46s | The applied patch generated 124
additional warning messages. |
| {color:green}+1{color} | javadoc | 9m 44s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 23s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 35s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 57s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 1m 39s | Tests passed in
hadoop-mapreduce-client-core. |
| {color:red}-1{color} | mapreduce tests | 91m 4s | Tests failed in
hadoop-mapreduce-client-jobclient. |
| | | 131m 19s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.mapreduce.v2.TestMRAppWithCombiner |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12643448/MAPREDUCE-5876.1.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 6ae2a0d |
| javac |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5602/artifact/patchprocess/diffJavacWarnings.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5602/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| hadoop-mapreduce-client-jobclient test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5602/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5602/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5602/console |
This message was automatically generated.
SequenceFileRecordReader NPE if close() is called before initialize()
-
Key: MAPREDUCE-5876
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5876
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 2.3.0, 2.4.0
Reporter: Reinis Vicups
Assignee: Tsuyoshi Ozawa
Attachments: MAPREDUCE-5876.1.patch
org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader extends
org.apache.hadoop.mapreduce.RecordReader which in turn implements
java.io.Closeable.
According to java spec the java.io.Closeable#close() has to be idempotent
(http://docs.oracle.com/javase/7/docs/api/java/io/Closeable.html) which is
not.
An NPE is being thrown if close() method is invoked without previously
calling initialize() method. This happens because SequenceFile.Reader in is
null.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6003) Resource Estimator suggests huge map output in some cases
[
https://issues.apache.org/jira/browse/MAPREDUCE-6003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525549#comment-14525549
]
Hadoop QA commented on MAPREDUCE-6003:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12657803/MAPREDUCE-6003-branch-1.2.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | branch-1 / 5f5138e |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5608/console |
This message was automatically generated.
Resource Estimator suggests huge map output in some cases
-
Key: MAPREDUCE-6003
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6003
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Affects Versions: 1.2.1
Reporter: Chengbing Liu
Assignee: Chengbing Liu
Attachments: MAPREDUCE-6003-branch-1.2.patch
In some cases, ResourceEstimator can return way too large map output
estimation. This happens when input size is not correctly calculated.
A typical case is when joining two Hive tables (one in HDFS and the other in
HBase). The maps that process the HBase table finish first, which has a 0
length of inputs due to its TableInputFormat. Then for a map that processes
HDFS table, the estimated output size is very large because of the wrong
input size, causing the map task not possible to be assigned.
There are two possible solutions to this problem:
(1) Make input size correct for each case, e.g. HBase, etc.
(2) Use another algorithm to estimate the map output, or at least make it
closer to reality.
I prefer the second way, since the first would require all possibilities to
be taken care of. It is not easy for some inputs such as URIs.
In my opinion, we could make a second estimation which is independent of the
input size:
estimationB = (completedMapOutputSize / completedMaps) * totalMaps * 10
Here, multiplying by 10 makes the estimation more conservative, so that it
will be less likely to assign it to some where not big enough.
The former estimation goes like this:
estimationA = (inputSize * completedMapOutputSize * 2.0) /
completedMapInputSize
My suggestion is to take minimum of the two estimations:
estimation = min(estimationA, estimationB)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5905) CountersStrings.toEscapedCompactStrings outputs unnecessary null strings
[
https://issues.apache.org/jira/browse/MAPREDUCE-5905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525552#comment-14525552
]
Hadoop QA commented on MAPREDUCE-5905:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 30s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 27s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 35s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 51s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 32s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 15s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 1m 36s | Tests passed in
hadoop-mapreduce-client-core. |
| | | 37m 46s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12646838/MAPREDUCE-5905.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 6ae2a0d |
| whitespace |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5605/artifact/patchprocess/whitespace.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5605/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5605/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5605/console |
This message was automatically generated.
CountersStrings.toEscapedCompactStrings outputs unnecessary null strings
--
Key: MAPREDUCE-5905
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5905
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA
Priority: Minor
Attachments: MAPREDUCE-5905.patch
CountersStrings.toEscapedCompactStrings outputs null strings if a
CounterGroup has more than one Counter.
That way there are some null strings in MRv1(CDH) job history log.
https://issues.cloudera.org/browse/DISTRO-598
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5845) TestShuffleHandler failing intermittently on windows
[
https://issues.apache.org/jira/browse/MAPREDUCE-5845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525498#comment-14525498
]
Hadoop QA commented on MAPREDUCE-5845:
--
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 5m 7s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 24s | There were no new javac warning
messages. |
| {color:green}+1{color} | release audit | 0m 19s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 20s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 32s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 31s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 0m 33s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 0m 18s | Tests passed in
hadoop-mapreduce-client-shuffle. |
| | | 16m 8s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12640707/apache-mapreduce-5845.0.patch
|
| Optional Tests | javac unit findbugs checkstyle |
| git revision | trunk / 6ae2a0d |
| hadoop-mapreduce-client-shuffle test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5601/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5601/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5601/console |
This message was automatically generated.
TestShuffleHandler failing intermittently on windows
Key: MAPREDUCE-5845
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5845
Project: Hadoop Map/Reduce
Issue Type: Test
Reporter: Varun Vasudev
Assignee: Varun Vasudev
Attachments: apache-mapreduce-5845.0.patch
TestShuffleHandler fails intermittently on Windows - specifically,
testClientClosesConnection.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5649) Reduce cannot use more than 2G memory for the final merge
[ https://issues.apache.org/jira/browse/MAPREDUCE-5649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gera Shegalov updated MAPREDUCE-5649: - Attachment: MAPREDUCE-5649.003.patch whitespace fix Reduce cannot use more than 2G memory for the final merge -- Key: MAPREDUCE-5649 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5649 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: stanley shi Assignee: Gera Shegalov Attachments: MAPREDUCE-5649.001.patch, MAPREDUCE-5649.002.patch, MAPREDUCE-5649.003.patch In the org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.java file, in the finalMerge method: int maxInMemReduce = (int)Math.min( Runtime.getRuntime().maxMemory() * maxRedPer, Integer.MAX_VALUE); This means no matter how much memory user has, reducer will not retain more than 2G data in memory before the reduce phase starts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5264) FileAlreadyExistsException is assumed to be thrown by FileSystem#mkdirs or FileContext#mkdir in the codebase
[
https://issues.apache.org/jira/browse/MAPREDUCE-5264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525500#comment-14525500
]
Hadoop QA commented on MAPREDUCE-5264:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 15m 6s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 39s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 48s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 49s | The applied patch generated
11 new checkstyle issues (total was 341, now 352). |
| {color:red}-1{color} | checkstyle | 1m 2s | The applied patch generated 8
new checkstyle issues (total was 17, now 25). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 43s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 9m 27s | Tests passed in
hadoop-mapreduce-client-app. |
| {color:green}+1{color} | mapreduce tests | 5m 43s | Tests passed in
hadoop-mapreduce-client-hs. |
| | | 52m 58s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12586811/MAPREDUCE-5264.20130607.1.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 6ae2a0d |
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5600/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-app.txt
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5600/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-hs.txt
|
| hadoop-mapreduce-client-app test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5600/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
|
| hadoop-mapreduce-client-hs test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5600/artifact/patchprocess/testrun_hadoop-mapreduce-client-hs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5600/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5600/console |
This message was automatically generated.
FileAlreadyExistsException is assumed to be thrown by FileSystem#mkdirs or
FileContext#mkdir in the codebase
Key: MAPREDUCE-5264
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5264
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Rémy SAISSY
Attachments: MAPREDUCE-5264.20130607.1.patch
According to https://issues.apache.org/jira/browse/HADOOP-9438,
FileSystem#mkdirs and FileContext#mkdir do not throw
FileAlreadyExistsException if the directory already exist.
Some places in the mapreduce codebase assumes FileSystem#mkdirs or
FileContext#mkdir throw FileAlreadyExistsException.
At least the following files are concerned:
- YarnChild.java
- JobHistoryEverntHandler.java
- HistoryFileManager.java
It would be good to re-review and patch this if needed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5216) While using TextSplitter in DataDrivenDBInputformat, the lower limit (split start) always remains the same, for all splits.
[
https://issues.apache.org/jira/browse/MAPREDUCE-5216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525536#comment-14525536
]
Hadoop QA commented on MAPREDUCE-5216:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 34s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 31s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 37s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 46s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 13s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:red}-1{color} | mapreduce tests | 1m 32s | Tests failed in
hadoop-mapreduce-client-core. |
| | | 37m 46s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.mapreduce.lib.db.TestSplitters |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12582090/MAPREDUCE-5216.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 6ae2a0d |
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5603/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5603/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5603/console |
This message was automatically generated.
While using TextSplitter in DataDrivenDBInputformat, the lower limit (split
start) always remains the same, for all splits.
---
Key: MAPREDUCE-5216
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5216
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Gelesh
Attachments: MAPREDUCE-5216.patch
Original Estimate: 1h
Remaining Estimate: 1h
While using TextSplitter in DataDrivenDBInputformat, the lower limit (split
start) always remains the same, for all splits.
ie,
Split 1 Start =A, End = M, Split 2 Start =A, End = P, Split 3 Start =A, End =
S,
instead of
Split 1 Start =A, End = M, Split 2 Start =M, End = P, Split 3 Start =P, End =
S,
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5883) Total megabyte-seconds in job counters is slightly misleading
[
https://issues.apache.org/jira/browse/MAPREDUCE-5883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525544#comment-14525544
]
Hadoop QA commented on MAPREDUCE-5883:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 34s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 27s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 32s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | mapreduce tests | 1m 36s | Tests passed in
hadoop-mapreduce-client-core. |
| | | 35m 42s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12643763/MAPREDUCE-5883.patch |
| Optional Tests | javadoc javac unit |
| git revision | trunk / 6ae2a0d |
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5604/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5604/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5604/console |
This message was automatically generated.
Total megabyte-seconds in job counters is slightly misleading
---
Key: MAPREDUCE-5883
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5883
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 3.0.0, 2.4.0
Reporter: Nathan Roberts
Assignee: Nathan Roberts
Priority: Minor
Attachments: MAPREDUCE-5883.patch
The following counters are in milliseconds so megabyte-seconds might be
better stated as megabyte-milliseconds
MB_MILLIS_MAPS.name= Total megabyte-seconds taken by all map
tasks
MB_MILLIS_REDUCES.name=Total megabyte-seconds taken by all reduce
tasks
VCORES_MILLIS_MAPS.name= Total vcore-seconds taken by all map tasks
VCORES_MILLIS_REDUCES.name=Total vcore-seconds taken by all reduce
tasks
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6146) Reduce tar ball size for MR over distributed cache
[
https://issues.apache.org/jira/browse/MAPREDUCE-6146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525646#comment-14525646
]
Hadoop QA commented on MAPREDUCE-6146:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 29s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 28s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 30s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| | | 33m 59s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12679094/MAPREDUCE-6146-v2.patch
|
| Optional Tests | javadoc javac unit |
| git revision | trunk / e8d0ee5 |
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5621/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5621/console |
This message was automatically generated.
Reduce tar ball size for MR over distributed cache
--
Key: MAPREDUCE-6146
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6146
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: build
Reporter: Junping Du
Assignee: Junping Du
Priority: Critical
Attachments: HADOOP-11177.patch, MAPREDUCE-6146-v2.patch,
MAPREDUCE-6146.patch
The current tar ball built from mvn package -Pdist -DskipTests -Dtar is
over 160M in size. We need more smaller tar ball pieces for feature like MR
over distributed cache to support Rolling update of cluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6164) mapreduce.reduce.shuffle.fetch.retry.timeout-ms should be set to 3 minutes instead of 30 seconds by default to be consistent with other retry timeout
[
https://issues.apache.org/jira/browse/MAPREDUCE-6164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525657#comment-14525657
]
Hadoop QA commented on MAPREDUCE-6164:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 39s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 29s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 41s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | mapreduce tests | 1m 34s | Tests passed in
hadoop-mapreduce-client-core. |
| | | 35m 55s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12681914/MAPREDUCE-6164.patch |
| Optional Tests | javadoc javac unit |
| git revision | trunk / e8d0ee5 |
| whitespace |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5622/artifact/patchprocess/whitespace.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5622/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5622/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5622/console |
This message was automatically generated.
mapreduce.reduce.shuffle.fetch.retry.timeout-ms should be set to 3 minutes
instead of 30 seconds by default to be consistent with other retry timeout
Key: MAPREDUCE-6164
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6164
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Junping Du
Assignee: Junping Du
Attachments: MAPREDUCE-6164.patch
In MAPREDUCE-5891, we are adding retry logic to MAPREDUCE shuffle stage for
fetcher can be survival during NM downtime (with shuffle service down as
well). In many places, we are setting the default timeout to be 3 minutes
(connection timeout, etc.) to tolerant possible more time for NM down, but we
are making mapreduce.reduce.shuffle.fetch.retry.timeout-ms to be 30 seconds
which is not consistent here. We should change this to 180 seconds.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5954) Optional exclusion of counters from getTaskReports
[
https://issues.apache.org/jira/browse/MAPREDUCE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525664#comment-14525664
]
Hadoop QA commented on MAPREDUCE-5954:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 33s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 31s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 36s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 16s | The applied patch generated 3
new checkstyle issues (total was 15, now 18). |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 6 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 35s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 36s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 8m 58s | Tests passed in
hadoop-mapreduce-client-app. |
| {color:green}+1{color} | mapreduce tests | 0m 46s | Tests passed in
hadoop-mapreduce-client-common. |
| {color:green}+1{color} | mapreduce tests | 105m 56s | Tests passed in
hadoop-mapreduce-client-jobclient. |
| | | 153m 53s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12654187/MAPREDUCE-5954.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / e8d0ee5 |
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5617/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-common.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5617/artifact/patchprocess/whitespace.txt
|
| hadoop-mapreduce-client-app test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5617/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
|
| hadoop-mapreduce-client-common test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5617/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
|
| hadoop-mapreduce-client-jobclient test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5617/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5617/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5617/console |
This message was automatically generated.
Optional exclusion of counters from getTaskReports
--
Key: MAPREDUCE-5954
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5954
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Maysam Yabandeh
Assignee: Maysam Yabandeh
Attachments: MAPREDUCE-5954.patch
MRClientService#getTaskReports returns the set of map or reduce tasks along
with their counters, which are quite large. For big jobs, the response could
be as large as 0.5 GB. This has a negative impact both on MRAppMaster and the
monitoring tool that invokes getTaskReports. This problem has led Pig users
to entirely disable getTaskReports for big jobs:
https://issues.apache.org/jira/browse/PIG-4043
Many monitoring tools, including ours, do not need the task counters when
invoking getTaskReports. Pig also does not make any use of task counters.
Here are the usages of Tasks in pig:
{code}
protected void getErrorMessages(TaskReport reports[], String type,
String msgs[] = reports[i].getDiagnostics();
if (HadoopShims.isJobFailed(reports[i])) {
{code}
and
{code}
protected long computeTimeSpent(TaskReport[] taskReports) {
long timeSpent = 0;
for (TaskReport r : taskReports) {
timeSpent += (r.getFinishTime() - r.getStartTime());
}
return timeSpent;
}
{code}
GetTaskReportsRequest can be augmented with an optional boolean with which
the monitoring tool can request excluding the counters form the response.
This minor change is very simple and yet makes many
[jira] [Commented] (MAPREDUCE-6111) Hadoop users' staging directories should be under a user folder
[
https://issues.apache.org/jira/browse/MAPREDUCE-6111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525670#comment-14525670
]
Hadoop QA commented on MAPREDUCE-6111:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 32s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 26s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 34s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 32s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 38s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:red}-1{color} | mapreduce tests | 9m 33s | Tests failed in
hadoop-mapreduce-client-app. |
| {color:green}+1{color} | mapreduce tests | 0m 45s | Tests passed in
hadoop-mapreduce-client-common. |
| {color:green}+1{color} | mapreduce tests | 5m 43s | Tests passed in
hadoop-mapreduce-client-hs. |
| | | 54m 14s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.mapreduce.v2.app.TestMRAppMaster |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12672704/MAPREDUCE-6111.2.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / e8d0ee5 |
| hadoop-mapreduce-client-app test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5623/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
|
| hadoop-mapreduce-client-common test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5623/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
|
| hadoop-mapreduce-client-hs test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5623/artifact/patchprocess/testrun_hadoop-mapreduce-client-hs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5623/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5623/console |
This message was automatically generated.
Hadoop users' staging directories should be under a user folder
---
Key: MAPREDUCE-6111
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6111
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.5.0, 2.4.1
Reporter: Jian Fang
Attachments: MAPREDUCE-6111.2.patch
Right now, Hadoop puts all users' staging directories under
/tmp/hadoop-yarn/staging/, for example /tmp/hadoop-yarn/staging/hadoop for
user hadoop, but the directory /tmp/hadoop-yarn/staging is also used for
other purpose. For example, /tmp/hadoop-yarn/staging/history/ is used to hold
finished JHS files. The shared parent /tmp/hadoop-yarn/staging makes it
difficult to track all users' folders without adding extra logic to exclude
other known folders.
As a result, we should move all users' folders to a user sub-folder, i.e.,
/tmp/hadoop-yarn/staging/user/. In this case, user hadoop's staging folder
becomes /tmp/hadoop-yarn/staging/user/hadoop/.staging.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-2094) org.apache.hadoop.mapreduce.lib.input.FileInputFormat: isSplitable implements unsafe default behaviour that is different from the documented behaviour.
[
https://issues.apache.org/jira/browse/MAPREDUCE-2094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525562#comment-14525562
]
Hadoop QA commented on MAPREDUCE-2094:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 31s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | javac | 7m 27s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 37s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 3s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 1s | The patch has 9 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 35s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 15s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 1m 34s | Tests passed in
hadoop-mapreduce-client-core. |
| | | 38m 0s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12658039/MAPREDUCE-2094-20140727-svn.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 6ae2a0d |
| whitespace |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5609/artifact/patchprocess/whitespace.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5609/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5609/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5609/console |
This message was automatically generated.
org.apache.hadoop.mapreduce.lib.input.FileInputFormat: isSplitable implements
unsafe default behaviour that is different from the documented behaviour.
---
Key: MAPREDUCE-2094
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2094
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: task
Reporter: Niels Basjes
Assignee: Niels Basjes
Attachments: MAPREDUCE-2094-2011-05-19.patch,
MAPREDUCE-2094-20140727-svn.patch, MAPREDUCE-2094-20140727.patch,
MAPREDUCE-2094-FileInputFormat-docs-v2.patch
When implementing a custom derivative of FileInputFormat we ran into the
effect that a large Gzipped input file would be processed several times.
A near 1GiB file would be processed around 36 times in its entirety. Thus
producing garbage results and taking up a lot more CPU time than needed.
It took a while to figure out and what we found is that the default
implementation of the isSplittable method in
[org.apache.hadoop.mapreduce.lib.input.FileInputFormat |
http://svn.apache.org/viewvc/hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java?view=markup
] is simply return true;.
This is a very unsafe default and is in contradiction with the JavaDoc of the
method which states: Is the given filename splitable? Usually, true, but if
the file is stream compressed, it will not be. . The actual implementation
effectively does Is the given filename splitable? Always true, even if the
file is stream compressed using an unsplittable compression codec.
For our situation (where we always have Gzipped input) we took the easy way
out and simply implemented an isSplittable in our class that does return
false;
Now there are essentially 3 ways I can think of for fixing this (in order of
what I would find preferable):
# Implement something that looks at the used compression of the file (i.e. do
migrate the implementation from TextInputFormat to FileInputFormat). This
would make the method do what the JavaDoc describes.
# Force developers to think about it and make this method abstract.
# Use a safe default (i.e. return false)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6027) mr jobs with relative paths can fail
[
https://issues.apache.org/jira/browse/MAPREDUCE-6027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525576#comment-14525576
]
Hadoop QA commented on MAPREDUCE-6027:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 38s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 30s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 37s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 27s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 2 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 13s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 1m 35s | Tests passed in
hadoop-mapreduce-client-core. |
| | | 38m 31s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12660521/MAPREDUCE-6027.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 6ae2a0d |
| whitespace |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5612/artifact/patchprocess/whitespace.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5612/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5612/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5612/console |
This message was automatically generated.
mr jobs with relative paths can fail
Key: MAPREDUCE-6027
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6027
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: job submission
Reporter: Wing Yew Poon
Assignee: Wing Yew Poon
Attachments: MAPREDUCE-6027.patch
I built hadoop from branch-2 and tried to run terasort as follows:
{noformat}
wypoon$ bin/hadoop jar
share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-SNAPSHOT.jar terasort
sort-input sort-output
14/08/07 08:57:55 INFO terasort.TeraSort: starting
2014-08-07 08:57:56.229 java[36572:1903] Unable to load realm info from
SCDynamicStore
14/08/07 08:57:56 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
14/08/07 08:57:57 INFO input.FileInputFormat: Total input paths to process : 2
Spent 156ms computing base-splits.
Spent 2ms computing TeraScheduler splits.
Computing input splits took 159ms
Sampling 2 splits of 2
Making 1 from 10 sampled records
Computing parititions took 626ms
Spent 789ms computing partitions.
14/08/07 08:57:57 INFO client.RMProxy: Connecting to ResourceManager at
localhost/127.0.0.1:8032
14/08/07 08:57:58 INFO mapreduce.JobSubmitter: Cleaning up the staging area
/tmp/hadoop-yarn/staging/wypoon/.staging/job_1407426900134_0001
java.lang.IllegalArgumentException: Can not create a Path from an empty URI
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:140)
at org.apache.hadoop.fs.Path.init(Path.java:192)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.checkPermissionOfOther(ClientDistributedCacheManager.java:275)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.ancestorsHaveExecutePermissions(ClientDistributedCacheManager.java:256)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.isPublic(ClientDistributedCacheManager.java:243)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineCacheVisibilities(ClientDistributedCacheManager.java:162)
at
[jira] [Commented] (MAPREDUCE-5871) Estimate Job Endtime
[
https://issues.apache.org/jira/browse/MAPREDUCE-5871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525578#comment-14525578
]
Hadoop QA commented on MAPREDUCE-5871:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 29s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 30s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 35s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 41s | The applied patch generated
25 new checkstyle issues (total was 78, now 103). |
| {color:red}-1{color} | whitespace | 0m 2s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 0m 56s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 8m 51s | Tests passed in
hadoop-mapreduce-client-app. |
| | | 44m 35s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12642692/MAPREDUCE-5871.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 6ae2a0d |
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5611/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-app.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5611/artifact/patchprocess/whitespace.txt
|
| hadoop-mapreduce-client-app test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5611/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5611/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5611/console |
This message was automatically generated.
Estimate Job Endtime
Key: MAPREDUCE-5871
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5871
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Maysam Yabandeh
Assignee: Maysam Yabandeh
Attachments: MAPREDUCE-5871.patch
YARN-1969 adds a new earliest-endtime-first policy to the fair scheduler. As
a prerequisite step, the AppMaster should estimate its end time and send it
to the RM via the heartbeat. This jira focuses on how the AppMaster performs
this estimation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-3182) loadgen ignores -m command line when writing random data
[
https://issues.apache.org/jira/browse/MAPREDUCE-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525588#comment-14525588
]
Hadoop QA commented on MAPREDUCE-3182:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 5m 18s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 41s | There were no new javac warning
messages. |
| {color:green}+1{color} | release audit | 0m 19s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 34s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 31s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 0m 42s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:red}-1{color} | mapreduce tests | 97m 2s | Tests failed in
hadoop-mapreduce-client-jobclient. |
| | | 113m 44s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.mapreduce.v2.TestMRJobsWithProfiler |
| | hadoop.mapred.TestMRIntermediateDataEncryption |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12640090/MAPREDUCE-3182.patch |
| Optional Tests | javac unit findbugs checkstyle |
| git revision | trunk / 6ae2a0d |
| hadoop-mapreduce-client-jobclient test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5606/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5606/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5606/console |
This message was automatically generated.
loadgen ignores -m command line when writing random data
Key: MAPREDUCE-3182
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3182
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, test
Affects Versions: 0.23.0, 2.3.0
Reporter: Jonathan Eagles
Assignee: Chen He
Attachments: MAPREDUCE-3182.patch
If no input directories are specified, loadgen goes into a special mode where
random data is generated and written. In that mode, setting the number of
mappers (-m command line option) is overridden by a calculation. Instead, it
should take into consideration the user specified number of mappers and fall
back to the calculation. In addition, update the documentation as well to
match the new behavior in the code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5221) Reduce side Combiner is not used when using the new API
[
https://issues.apache.org/jira/browse/MAPREDUCE-5221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525590#comment-14525590
]
Hadoop QA commented on MAPREDUCE-5221:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 34s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 4 new or modified test files. |
| {color:red}-1{color} | javac | 4m 4s | The patch appears to cause the
build to fail. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12666386/MAPREDUCE-5221.10.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 6ae2a0d |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5614/console |
This message was automatically generated.
Reduce side Combiner is not used when using the new API
---
Key: MAPREDUCE-5221
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5221
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.0.4-alpha
Reporter: Siddharth Seth
Assignee: Tsuyoshi Ozawa
Attachments: MAPREDUCE-5221.1.patch, MAPREDUCE-5221.10.patch,
MAPREDUCE-5221.2.patch, MAPREDUCE-5221.3.patch, MAPREDUCE-5221.4.patch,
MAPREDUCE-5221.5.patch, MAPREDUCE-5221.6.patch, MAPREDUCE-5221.7-2.patch,
MAPREDUCE-5221.7.patch, MAPREDUCE-5221.8.patch, MAPREDUCE-5221.9.patch
If a combiner is specified using o.a.h.mapreduce.Job.setCombinerClass - this
will silently ignored on the reduce side since the reduce side usage is only
aware of the old api combiner.
This doesn't fail the job - since the new combiner key does not deprecate the
old key.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5937) hadoop/mapred job -history history_file shows the counters twice in the output.
[
https://issues.apache.org/jira/browse/MAPREDUCE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525601#comment-14525601
]
Hadoop QA commented on MAPREDUCE-5937:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 38s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 29s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 31s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 48s | The applied patch generated 7
new checkstyle issues (total was 207, now 209). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 15s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 1m 34s | Tests passed in
hadoop-mapreduce-client-core. |
| | | 37m 45s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12651727/MAPREDUCE-5937.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 6ae2a0d |
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5613/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5613/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5613/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5613/console |
This message was automatically generated.
hadoop/mapred job -history history_file shows the counters twice in the
output.
-
Key: MAPREDUCE-5937
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5937
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Jinghui Wang
Assignee: Jinghui Wang
Attachments: MAPREDUCE-5937.patch, job_history_cli_sample.out
HiistoryView#printCounters method uses AbstractCounter#getGroupNames, which
includes legacy groups can cause duplicates on CLI output.
See attached example output.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6079) Renaming JobImpl#username to reporterUserName
[
https://issues.apache.org/jira/browse/MAPREDUCE-6079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525613#comment-14525613
]
Hadoop QA commented on MAPREDUCE-6079:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 37s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 29s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 36s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 37s | The applied patch generated 2
new checkstyle issues (total was 107, now 108). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 0m 56s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 8m 45s | Tests passed in
hadoop-mapreduce-client-app. |
| | | 44m 30s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12667367/MAPREDUCE-6079.1.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 6ae2a0d |
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5615/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-app.txt
|
| hadoop-mapreduce-client-app test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5615/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5615/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5615/console |
This message was automatically generated.
Renaming JobImpl#username to reporterUserName
-
Key: MAPREDUCE-6079
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6079
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Tsuyoshi Ozawa
Assignee: Tsuyoshi Ozawa
Attachments: MAPREDUCE-6079.1.patch
On MAPREDUCE-6033, we found the bug because of confusing field names
{{userName}} and {{username}}. We should change the names to distinguish them
easily.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6117) Hadoop ignores yarn.nodemanager.hostname for RPC listeners
[
https://issues.apache.org/jira/browse/MAPREDUCE-6117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525637#comment-14525637
]
Hadoop QA commented on MAPREDUCE-6117:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 15m 10s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 44s | There were no new javac warning
messages. |
| {color:red}-1{color} | javadoc | 9m 58s | The applied patch generated 8
additional warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 36s | The applied patch generated 8
new checkstyle issues (total was 45, now 53). |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 36s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 0m 55s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 9m 17s | Tests passed in
hadoop-mapreduce-client-app. |
| | | 46m 14s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12675078/MapReduce-534.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / e8d0ee5 |
| javadoc |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5619/artifact/patchprocess/diffJavadocWarnings.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5619/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-app.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5619/artifact/patchprocess/whitespace.txt
|
| hadoop-mapreduce-client-app test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5619/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5619/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5619/console |
This message was automatically generated.
Hadoop ignores yarn.nodemanager.hostname for RPC listeners
--
Key: MAPREDUCE-6117
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6117
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client, task
Affects Versions: 2.2.1, 2.4.1, 2.5.1
Environment: Any mapreduce example with standard cluster. In our
case each node has four networks. It is important that all internode
communication be done on a specific network.
Reporter: Waldyn Benbenek
Assignee: Waldyn Benbenek
Fix For: 2.5.1
Attachments: MapReduce-534.patch
Original Estimate: 48h
Time Spent: 384h
Remaining Estimate: 0h
The RPC listeners for an application are using the hostname of the node as
the binding address of the listener, They ignore yarn.nodemanager.hostname
for this. In our setup we want all communication between nodes to be done
via the network addresses we specify in yarn.nodemanager.hostname on each
node.
TaskAttemptListenerImpl.java and MRClientService.java are two places I have
found where the default address is used rather that NM_host. The node
Manager hostname should be used for all communication between nodes including
the RPC listeners.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5747) Potential null pointer deference in HsTasksBlock#render()
[
https://issues.apache.org/jira/browse/MAPREDUCE-5747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525636#comment-14525636
]
Hadoop QA commented on MAPREDUCE-5747:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 31s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 29s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 32s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 26s | The applied patch generated 2
new checkstyle issues (total was 10, now 11). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 32s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 0m 42s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 5m 43s | Tests passed in
hadoop-mapreduce-client-hs. |
| | | 40m 55s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12676205/MAPREDUCE-5747-1.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / e8d0ee5 |
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5620/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-hs.txt
|
| hadoop-mapreduce-client-hs test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5620/artifact/patchprocess/testrun_hadoop-mapreduce-client-hs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5620/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5620/console |
This message was automatically generated.
Potential null pointer deference in HsTasksBlock#render()
-
Key: MAPREDUCE-5747
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5747
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
Labels: newbie, patch
Attachments: MAPREDUCE-5747-1.patch
At line 140:
{code}
} else {
ta = new TaskAttemptInfo(successful, type, false);
{code}
There is no check for type against null.
TaskAttemptInfo ctor deferences type:
{code}
public TaskAttemptInfo(TaskAttempt ta, TaskType type, Boolean isRunning) {
final TaskAttemptReport report = ta.getReport();
this.type = type.toString();
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6023) Fix SuppressWarnings from unchecked to rawtypes in O.A.H.mapreduce.lib.input.TaggedInputSplit
[
https://issues.apache.org/jira/browse/MAPREDUCE-6023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525575#comment-14525575
]
Hadoop QA commented on MAPREDUCE-6023:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 33s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:red}-1{color} | javac | 7m 27s | The applied patch generated 129
additional warning messages. |
| {color:green}+1{color} | javadoc | 9m 35s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 52s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 35s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 14s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 1m 35s | Tests passed in
hadoop-mapreduce-client-core. |
| | | 37m 52s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12660364/MAPREDUCE-6023.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 6ae2a0d |
| javac |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5610/artifact/patchprocess/diffJavacWarnings.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5610/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5610/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5610/console |
This message was automatically generated.
Fix SuppressWarnings from unchecked to rawtypes in
O.A.H.mapreduce.lib.input.TaggedInputSplit
-
Key: MAPREDUCE-6023
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6023
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Junping Du
Assignee: Abhilash Srimat Tirumala Pallerlamudi
Priority: Minor
Labels: newbie
Attachments: MAPREDUCE-6023.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6085) Facilitate processing of text files without key/value split
[
https://issues.apache.org/jira/browse/MAPREDUCE-6085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525614#comment-14525614
]
Hadoop QA commented on MAPREDUCE-6085:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 32s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 31s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 41s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 24s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 0m 38s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | tools/hadoop tests | 6m 8s | Tests passed in
hadoop-streaming. |
| | | 41m 26s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12668101/IdentifierResolver1.java.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / e8d0ee5 |
| whitespace |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5616/artifact/patchprocess/whitespace.txt
|
| hadoop-streaming test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5616/artifact/patchprocess/testrun_hadoop-streaming.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5616/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5616/console |
This message was automatically generated.
Facilitate processing of text files without key/value split
---
Key: MAPREDUCE-6085
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6085
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 2.4.1
Reporter: Dmitry Sivachenko
Attachments: IdentifierResolver1.java.patch
There is a rather popular type of task: processing of text files line by line
without splitting line to key/value pair in streaming mode. (UNIX commands
like grep, awk, etc, any filter scripts).
By default, Hadoop streaming interface uses TextInputFormat which suites well
for this task: it passes the input line itself to streaming job stdin.
TextOutputReader class, which receives streaming job's output, splits it for
key and value pair, and TextOutputFormat tries to merge this pair with
separator.
This results in extra separator appearing in the output in some cases.
KeyOnlyTextOutputReader solves this problem: it passes the whole line as a
key with null value, and TextOutputFormat correctly writes it without any
separators inserted.
I propose to add another IdentifierResolver: keyonlytextoutput, which uses
standard TextInputWriter but replaces TextOutputReader with
KeyOnlyTextOutputReader).
As a result, lines of text are never split into key/value pair and never
joined back, so lines appear in the output unmodified.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5865) RecordReader is not closed in TeraInputFormat#writePartitionFile()
[
https://issues.apache.org/jira/browse/MAPREDUCE-5865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14525631#comment-14525631
]
Hadoop QA commented on MAPREDUCE-5865:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 15m 12s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:red}-1{color} | javac | 7m 42s | The applied patch generated 140
additional warning messages. |
| {color:green}+1{color} | javadoc | 9m 59s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 21s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 27s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 32s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 0m 44s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 0m 34s | Tests passed in
hadoop-mapreduce-examples. |
| | | 37m 8s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12668711/MAPREDUCE-5865.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / e8d0ee5 |
| javac |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5618/artifact/patchprocess/diffJavacWarnings.txt
|
| hadoop-mapreduce-examples test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5618/artifact/patchprocess/testrun_hadoop-mapreduce-examples.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5618/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5618/console |
This message was automatically generated.
RecordReader is not closed in TeraInputFormat#writePartitionFile()
--
Key: MAPREDUCE-5865
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5865
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv1
Reporter: Ted Yu
Attachments: MAPREDUCE-5865.patch
Here is related code:
{code}
RecordReaderText, Text reader =
inFormat.createRecordReader(splits.get(sampleStep * idx),
context);
reader.initialize(splits.get(sampleStep * idx), context);
while (reader.nextKeyValue()) {
sampler.addKey(new Text(reader.getCurrentKey()));
records += 1;
if (recordsPerSample = records) {
break;
}
}
{code}
reader should be closed using finally block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
