[jira] [Commented] (MAPREDUCE-6674) configure parallel tests for mapreduce-client-jobclient
[
https://issues.apache.org/jira/browse/MAPREDUCE-6674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16112081#comment-16112081
]
Hadoop QA commented on MAPREDUCE-6674:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
36s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 27m 17s{color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
22s{color} | {color:red} The patch generated 6 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 47m 40s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.fs.slive.TestSlive |
| | hadoop.mapreduce.TestMapReduce |
| | hadoop.mapred.TestUserDefinedCounters |
| | hadoop.mapred.TestReporter |
| | hadoop.mapreduce.v2.TestMRJobs |
| | hadoop.fs.TestFileSystem |
| | hadoop.mapred.lib.TestChainMapReduce |
| | hadoop.mapreduce.TestMapCollection |
| | hadoop.mapreduce.lib.chain.TestSingleElementChain |
| | hadoop.mapreduce.lib.input.TestLineRecordReaderJobs |
| | hadoop.mapred.lib.TestKeyFieldBasedComparator |
| | hadoop.mapred.lib.TestMultithreadedMapRunner |
| | hadoop.mapred.TestMROpportunisticMaps |
| | hadoop.mapreduce.TestLargeSort |
| | hadoop.mapreduce.lib.jobcontrol.TestMapReduceJobControl |
| | hadoop.mapred.TestCollect |
| | hadoop.fs.TestDFSIO |
| | hadoop.mapreduce.TestMapperReducerCleanup |
| | hadoop.mapreduce.lib.fieldsel.TestMRFieldSelection |
| | hadoop.mapreduce.v2.TestMRJobsWithProfiler |
| | hadoop.mapred.TestSpecialCharactersInOutputPath |
| | hadoop.conf.TestNoDefaultsJobConf |
| | hadoop.mapred.TestJobCounters |
| | hadoop.mapred.TestJobCleanup |
| | hadoop.mapreduce.lib.output.TestMRMultipleOutputs |
| | hadoop.mapreduce.TestMROutputFormat |
| | hadoop.mapreduce.TestMRJobClient |
| | hadoop.mapred.lib.aggregate.TestAggregates |
| | hadoop.mapred.TestComparators |
| | hadoop.mapreduce.v2.TestSpeculativeExecution |
| | hadoop.mapreduce.lib.input.TestMultipleInputs |
| | hadoop.mapred.TestMapRed |
| | hadoop.mapred.TestFileOutputFormat |
| | hadoop.mapreduce.TestValueIterReset |
| | hadoop.mapred.pipes.TestPipeApplication |
| | hadoop.mapred.TestLineRecordReaderJobs |
| | hadoop.mapred.TestOldCombinerGrouping |
| | hadoop.mapreduce.TestNewCombinerGrouping |
| | hadoop.mapreduce.lib.chain.TestChainErrors |
| | hadoop.hdfs.TestNNBench |
| | hadoop.mapreduce.lib.aggregate.TestMapReduceAggregates |
| | hadoop.mapreduce
[jira] [Updated] (MAPREDUCE-6674) configure parallel tests for mapreduce-client-jobclient
[ https://issues.apache.org/jira/browse/MAPREDUCE-6674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated MAPREDUCE-6674: Attachment: MAPREDUCE-6674.01.patch -01: * rebase to see where we are at > configure parallel tests for mapreduce-client-jobclient > --- > > Key: MAPREDUCE-6674 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6674 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Affects Versions: 3.0.0-alpha1 >Reporter: Allen Wittenauer >Priority: Critical > Attachments: MAPREDUCE-6674.00.patch, MAPREDUCE-6674.01.patch > > > mapreduce-client-jobclient takes almost an hour and a half. Configuring > parallel-tests would greatly reduce this run time. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6674) configure parallel tests for mapreduce-client-jobclient
[
https://issues.apache.org/jira/browse/MAPREDUCE-6674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16111996#comment-16111996
]
Hadoop QA commented on MAPREDUCE-6674:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} MAPREDUCE-6674 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | MAPREDUCE-6674 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12798342/MAPREDUCE-6674.00.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7040/console |
| Powered by | Apache Yetus 0.5.0 http://yetus.apache.org |
This message was automatically generated.
> configure parallel tests for mapreduce-client-jobclient
> ---
>
> Key: MAPREDUCE-6674
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6674
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: test
>Affects Versions: 3.0.0-alpha1
>Reporter: Allen Wittenauer
>Priority: Critical
> Attachments: MAPREDUCE-6674.00.patch
>
>
> mapreduce-client-jobclient takes almost an hour and a half. Configuring
> parallel-tests would greatly reduce this run time.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6931) Fix TestDFSIO "Total Throughput" calculation
[
https://issues.apache.org/jira/browse/MAPREDUCE-6931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16111987#comment-16111987
]
ASF GitHub Bot commented on MAPREDUCE-6931:
---
GitHub user dennishuo opened a pull request:
https://github.com/apache/hadoop/pull/259
MAPREDUCE-6931. Fix TestDFSIO "Total Throughput" calculation.
Previously it failed to convert ms to seconds and thus reports aggregate
throughput as 1/1000x actual numbers. Also, make all the bytes-to-mb
and milliseconds-to-seconds conversions consistent in the reporting
messages to help avoid this type of error in the future.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dennishuo/hadoop trunk
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/hadoop/pull/259.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #259
commit 23a374d6e818004daa0b1cf16c4fedb439ab4943
Author: Dennis Huo
Date: 2017-08-02T21:30:51Z
MAPREDUCE-6931. Fix TestDFSIO "Total Throughput" calculation.
Previously it failed to convert ms to seconds and thus reports aggregate
throughput as 1/1000x actual numbers. Also, make all the bytes-to-mb
and milliseconds-to-seconds conversions consistent in the reporting
messages to help avoid this type of error in the future.
> Fix TestDFSIO "Total Throughput" calculation
>
>
> Key: MAPREDUCE-6931
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6931
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: benchmarks, test
>Affects Versions: 2.8.0
>Reporter: Dennis Huo
>Priority: Trivial
>
> The new "Total Throughput" line added in
> https://issues.apache.org/jira/browse/HDFS-9153 is currently calculated as
> {{toMB(size) / ((float)execTime)}} and claims to be in units of "MB/s", but
> {{execTime}} is in milliseconds; thus, the reported number is 1/1000x the
> actual value:
> {code:java}
> String resultLines[] = {
> "- TestDFSIO - : " + testType,
> "Date & time: " + new Date(System.currentTimeMillis()),
> "Number of files: " + tasks,
> " Total MBytes processed: " + df.format(toMB(size)),
> " Throughput mb/sec: " + df.format(size * 1000.0 / (time *
> MEGA)),
> "Total Throughput mb/sec: " + df.format(toMB(size) /
> ((float)execTime)),
> " Average IO rate mb/sec: " + df.format(med),
> " IO rate std deviation: " + df.format(stdDev),
> " Test exec time sec: " + df.format((float)execTime / 1000),
> "" };
> {code}
> The different calculated fields can also use toMB and a shared
> milliseconds-to-seconds conversion to make it easier to keep units consistent.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6931) Fix TestDFSIO "Total Throughput" calculation
Dennis Huo created MAPREDUCE-6931:
-
Summary: Fix TestDFSIO "Total Throughput" calculation
Key: MAPREDUCE-6931
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6931
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: benchmarks, test
Affects Versions: 2.8.0
Reporter: Dennis Huo
Priority: Trivial
The new "Total Throughput" line added in
https://issues.apache.org/jira/browse/HDFS-9153 is currently calculated as
{{toMB(size) / ((float)execTime)}} and claims to be in units of "MB/s", but
{{execTime}} is in milliseconds; thus, the reported number is 1/1000x the
actual value:
{code:java}
String resultLines[] = {
"- TestDFSIO - : " + testType,
"Date & time: " + new Date(System.currentTimeMillis()),
"Number of files: " + tasks,
" Total MBytes processed: " + df.format(toMB(size)),
" Throughput mb/sec: " + df.format(size * 1000.0 / (time * MEGA)),
"Total Throughput mb/sec: " + df.format(toMB(size) / ((float)execTime)),
" Average IO rate mb/sec: " + df.format(med),
" IO rate std deviation: " + df.format(stdDev),
" Test exec time sec: " + df.format((float)execTime / 1000),
"" };
{code}
The different calculated fields can also use toMB and a shared
milliseconds-to-seconds conversion to make it easier to keep units consistent.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6930) mapreduce.map.cpu.vcores and mapreduce.reduce.cpu.vcores are both present twice in mapred-default.xml
Daniel Templeton created MAPREDUCE-6930: --- Summary: mapreduce.map.cpu.vcores and mapreduce.reduce.cpu.vcores are both present twice in mapred-default.xml Key: MAPREDUCE-6930 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6930 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 3.0.0-alpha4 Reporter: Daniel Templeton The second set should be deleted. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6929) TimelineV2Client hangs MR AM
[ https://issues.apache.org/jira/browse/MAPREDUCE-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vrushali C updated MAPREDUCE-6929: -- Summary: TimelineV2Client hangs MR AM (was: TimelineClient hangs MR AM) > TimelineV2Client hangs MR AM > > > Key: MAPREDUCE-6929 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6929 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am >Reporter: Haibo Chen > > I happened to misconfigure ATSv2 settings, that is, I enabled emitting to > ATSv2 in MR and did not start YARN with ATSv2. The job was stuck after it > finished all its work. > Noticed that in MRAppMaster, TimelineClient is never added as a service, > which I think is why the AM was hanging. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6929) TimelineClient hangs MR AM
[ https://issues.apache.org/jira/browse/MAPREDUCE-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16111769#comment-16111769 ] Haibo Chen commented on MAPREDUCE-6929: --- Yeah, that what I expected also. I did not collect the logs, unfortunately. I did remember seeing TimelineClient related failures in MRAM log. I can upload the logs later if I manage to reproduce it. In any case, we should add TimelineClient as a service under the AM service, right? That way, when AM receives a signal to exit, TimelineClient can also be stopped in serviceStop() > TimelineClient hangs MR AM > -- > > Key: MAPREDUCE-6929 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6929 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am >Reporter: Haibo Chen > > I happened to misconfigure ATSv2 settings, that is, I enabled emitting to > ATSv2 in MR and did not start YARN with ATSv2. The job was stuck after it > finished all its work. > Noticed that in MRAppMaster, TimelineClient is never added as a service, > which I think is why the AM was hanging. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-6923) YARN Shuffle I/O for small partitions
[
https://issues.apache.org/jira/browse/MAPREDUCE-6923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Prakash reassigned MAPREDUCE-6923:
---
Assignee: Robert Schmidtke
> YARN Shuffle I/O for small partitions
> -
>
> Key: MAPREDUCE-6923
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6923
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Environment: Observed in Hadoop 2.7.3 and above (judging from the
> source code of future versions), and Ubuntu 16.04.
>Reporter: Robert Schmidtke
>Assignee: Robert Schmidtke
>
> When a job configuration results in small partitions read by each reducer
> from each mapper (e.g. 65 kilobytes as in my setup: a
> [TeraSort|https://github.com/apache/hadoop/blob/branch-2.7.3/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/TeraSort.java]
> of 256 gigabytes using 2048 mappers and reducers each), and setting
> {code:xml}
>
> mapreduce.shuffle.transferTo.allowed
> false
>
> {code}
> then the default setting of
> {code:xml}
>
> mapreduce.shuffle.transfer.buffer.size
> 131072
>
> {code}
> results in almost 100% overhead in reads during shuffle in YARN, because for
> each 65K needed, 128K are read.
> I propose a fix in
> [FadvisedFileRegion.java|https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/FadvisedFileRegion.java#L114]
> as follows:
> {code:java}
> ByteBuffer byteBuffer = ByteBuffer.allocate(Math.min(this.shuffleBufferSize,
> trans > Integer.MAX_VALUE ? Integer.MAX_VALUE : (int) trans));
> {code}
> e.g.
> [here|https://github.com/apache/hadoop/compare/branch-2.7.3...robert-schmidtke:adaptive-shuffle-buffer].
> This sets the shuffle buffer size to the minimum value of the shuffle buffer
> size specified in the configuration (128K by default), and the actual
> partition size (65K on average in my setup). In my benchmarks this reduced
> the read overhead in YARN from about 100% (255 additional gigabytes as
> described above) down to about 18% (an additional 45 gigabytes). The runtime
> of the job remained the same in my setup.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6923) YARN Shuffle I/O for small partitions
[
https://issues.apache.org/jira/browse/MAPREDUCE-6923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16111759#comment-16111759
]
Ravi Prakash commented on MAPREDUCE-6923:
-
Hi Robert! Thanks for filing the JIRA and your contribution! I'm adding you as
a contributor and assigning the JIRA to you. Could you please post a patch file
to this JIRA? You can name the patch file MAPREDUCE\-6923.00.patch . One minor
nit is that we limit lines to 80 characters. Could you please fix that in the
patch file?
Also since {{trans}} is [guaranteed to be
positive|https://github.com/apache/hadoop/blob/12e44e7bdaf53d3720a89d32f0cc2717241bd6b2/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/FadvisedFileRegion.java#L103]
and {{shuffleBufferSize}} is an integer, maybe we don't really need the
ternary operator condition? Up to you to keep it though.
I'm not surprised that there isn't an improvement in job performance but the
read overhead improvement is great. Do you know where the 18% overhead is going?
The patch sounds reasonable to me. [~jlowe], [~nikola.vujic], [~cnauroth] do
you have any comments? The diff he's proposing is in the link to the word "e.g.
[here|https://github.com/apache/hadoop/compare/branch-2.7.3...robert-schmidtke:adaptive-shuffle-buffer]";
.
> YARN Shuffle I/O for small partitions
> -
>
> Key: MAPREDUCE-6923
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6923
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Environment: Observed in Hadoop 2.7.3 and above (judging from the
> source code of future versions), and Ubuntu 16.04.
>Reporter: Robert Schmidtke
>
> When a job configuration results in small partitions read by each reducer
> from each mapper (e.g. 65 kilobytes as in my setup: a
> [TeraSort|https://github.com/apache/hadoop/blob/branch-2.7.3/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/TeraSort.java]
> of 256 gigabytes using 2048 mappers and reducers each), and setting
> {code:xml}
>
> mapreduce.shuffle.transferTo.allowed
> false
>
> {code}
> then the default setting of
> {code:xml}
>
> mapreduce.shuffle.transfer.buffer.size
> 131072
>
> {code}
> results in almost 100% overhead in reads during shuffle in YARN, because for
> each 65K needed, 128K are read.
> I propose a fix in
> [FadvisedFileRegion.java|https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/FadvisedFileRegion.java#L114]
> as follows:
> {code:java}
> ByteBuffer byteBuffer = ByteBuffer.allocate(Math.min(this.shuffleBufferSize,
> trans > Integer.MAX_VALUE ? Integer.MAX_VALUE : (int) trans));
> {code}
> e.g.
> [here|https://github.com/apache/hadoop/compare/branch-2.7.3...robert-schmidtke:adaptive-shuffle-buffer].
> This sets the shuffle buffer size to the minimum value of the shuffle buffer
> size specified in the configuration (128K by default), and the actual
> partition size (65K on average in my setup). In my benchmarks this reduced
> the read overhead in YARN from about 100% (255 additional gigabytes as
> described above) down to about 18% (an additional 45 gigabytes). The runtime
> of the job remained the same in my setup.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6927) MR job should only set tracking url if history was successfully written
[ https://issues.apache.org/jira/browse/MAPREDUCE-6927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16111744#comment-16111744 ] Jason Lowe commented on MAPREDUCE-6927: --- Thanks for the report and patch! It would be helpful if you elaborated a bit more in the JIRA description about the problem this is fixing and how it manifests in practice. I think we should set the tracking URL as soon as the .jhist file is successfully put in the done folder which is a bit earlier than where it is now in the patch. Even if we fail to get the conf to done, there should be some MapReduce-specific history once the .jhist file is there. Also as written in the patch, there could be a case where we "successfully" complete processDoneFiles but nothing was actually placed in the done directory (i.e.: for whatever reason mi.getHistoryFile() == null). It would be nice if the unit test verified the job history URL was correctly rather than just any string at all. I think the test should also verify that even if the job history event handler receives a job finished event but is unable to complete moving the history to the done directory that it does not set the tracking URL. > MR job should only set tracking url if history was successfully written > --- > > Key: MAPREDUCE-6927 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6927 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Eric Badger >Assignee: Eric Badger > Attachments: MAPREDUCE-6927.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6929) TimelineClient hangs MR AM
[ https://issues.apache.org/jira/browse/MAPREDUCE-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16111724#comment-16111724 ] Vrushali C commented on MAPREDUCE-6929: --- Hmm, so the NMs did not have timeline collector as an auxiliary service but the AM was trying to connect to timeline service? It should retry for a few times (30 I think by default) and then just "complete". Do you have any logs? > TimelineClient hangs MR AM > -- > > Key: MAPREDUCE-6929 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6929 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am >Reporter: Haibo Chen > > I happened to misconfigure ATSv2 settings, that is, I enabled emitting to > ATSv2 in MR and did not start YARN with ATSv2. The job was stuck after it > finished all its work. > Noticed that in MRAppMaster, TimelineClient is never added as a service, > which I think is why the AM was hanging. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6929) TimelineClient hangs MR AM
Haibo Chen created MAPREDUCE-6929: - Summary: TimelineClient hangs MR AM Key: MAPREDUCE-6929 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6929 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mr-am Reporter: Haibo Chen I happened to misconfigure ATSv2 settings, that is, I enabled emitting to ATSv2 in MR and did not start YARN with ATSv2. The job was stuck after it finished all its work. Noticed that in MRAppMaster, TimelineClient is never added as a service, which I think is why the AM was hanging. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6914) Tests use assertTrue(....equals(...)) instead of assertEquals()
[ https://issues.apache.org/jira/browse/MAPREDUCE-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16111690#comment-16111690 ] Maya Wexler commented on MAPREDUCE-6914: +1 (non-binding) I verified that all of the test clauses are still valid. Code looks cleaner with assertEquals(). > Tests use assertTrue(equals(...)) instead of assertEquals() > --- > > Key: MAPREDUCE-6914 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6914 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: test >Affects Versions: 2.8.1, 3.0.0-alpha4 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Minor > Attachments: MAPREDUCE-6914.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6892) Issues with the count of failed/killed tasks in the jhist file
[
https://issues.apache.org/jira/browse/MAPREDUCE-6892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16111628#comment-16111628
]
Haibo Chen commented on MAPREDUCE-6892:
---
bq. 3. Looking at the toTimelineEvent() methods, those are a bit messed up too.
As I mentioned above, it has prop. keys like NUM_MAPS and FINISHED_MAPS, but
both contain the same value. So either we rename NUM_MAPS to SUCCEEDED_MAPS and
delete FINISHED_MAPS or NUM_MAPS should be succeeded+failed+killed and rename
FINISHED_MAPS to SUCCEEDED_MAPS.
TimelineServiceV2 related changes (new fields and renames) are fine as long as
we do it before 3.0 beta1, from which we'll need to consider compatibility. So
let's also include the changes needed here for ATSv2.
bq. 1. I can add new fields to the Avro schema and keep the old ones, but it
adds to the complexity, because we have to support both pairs
bq. 2. We probably also have to check whether "finished" or "succeeded" values
are defined in the jhist file, but not both
we can keep the old names and only add new fields, so we continue to use
'finished' as succeeded. In this case, there won't be a new field, succeeded.
A few more minor comments on the 2rd patch:
1) In JobHistoryEventHandler when we handles Job_KILL event, can we also add
killedMappers + killedReducers in the summary as well?
2) there is one line above JobImpl.unsuccessfulFinish() which is unnecessary.
3) In JobImpl.InternalTerminationTransition, can we include all killed/failed
counters instead of zeroes?
4) JobImpl#L1959 is more than 80 characters
5) In Job20LineHistoryEventEmitter, can we also parse the killed + failed
counters instead of passing -1s?
6) CompletedJob: instead of returning 0s for the new counters, we can return
jobInfo.get*() that we get from parsing .jhist files.
7) In TestJobHistoryParser.testHistoryParsingForKilledAndFailedAttempts(),
verification of noOffailedAttempts seems unnecessary.
In fact, the total # depends on job configuration. Can we remove the related
code?
> Issues with the count of failed/killed tasks in the jhist file
> --
>
> Key: MAPREDUCE-6892
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6892
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, jobhistoryserver
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: MAPREDUCE-6892-001.patch, MAPREDUCE-6892-002.PATCH
>
>
> Recently we encountered some issues with the value of failed tasks. After
> parsing the jhist file, {{JobInfo.getFailedMaps()}} returned 0, but actually
> there were failures.
> Another minor thing is that you cannot get the number of killed tasks
> (although this can be calculated).
> The root cause is that {{JobUnsuccessfulCompletionEvent}} contains only the
> successful map/reduce task counts. Number of failed (or killed) tasks are not
> stored.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6928) BackupStore Should Checksum Files Persisted To Disk
BELUGA BEHR created MAPREDUCE-6928:
--
Summary: BackupStore Should Checksum Files Persisted To Disk
Key: MAPREDUCE-6928
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6928
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Affects Versions: 3.0.0-alpha4, 2.8.1
Reporter: BELUGA BEHR
Priority: Minor
The class {{org.apache.hadoop.mapred.BackupStore}} persists data to memory or
disk. Like the {{IFile}} format that this class works in tandem with, the data
in {{BackupStore}} should be under a check sum to prevent otherwise good data
from becoming corrupt.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6870) Add configuration for MR job to finish when all reducers are complete (even with unfinished mappers)
[
https://issues.apache.org/jira/browse/MAPREDUCE-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16111487#comment-16111487
]
Haibo Chen commented on MAPREDUCE-6870:
---
bq. 1. Variable names (is preemptMappersOnReduceFinish good?)
I thingk we can rename JobImplpreemptMappersOnReduceFinish,
MRJobConfig.PREEMPT_MAPPERS_ON_REDUCE_FINISH and
MRJobConfig.DEFAULT_PREEMPT_MAPPERS_ON_REDUCE_FINISH to convey more clearly
that our intention is to finish the job from the users' perspective (because
the configuration is user-facing). Also the description of
mapreduce.job.finish-when-all-reducers-done in mapred-default.xml.
bq. 2. Added a new method to MapTaskImpl with locking, which is probably not
necessary but I felt it's better to have it anyway
What I meant previously is that we are creating duplicated TA_KILL events in
JobImpl.checkReadyForCompletionWhenAllReducersDone(). I am not sure adding
MapTaskImp.preemptedDueToReducerFinish is going to help in this case, because
we will still be sending duplicated TA_KILL events. I was proposing something
like the following
{code}
private void checkReadyForCompletionWhenAllReducersDone(JobImpl job) {
if (job.preemptRestartedMappersOnReduceFinish) {
int totalReduces = job.getTotalReduces();
int completedReduces = job.getCompletedReduces();
// only if all reducers have finished and we have not sent TA_KILL
events to running
// map tasks before
if (totalReduces > 0 && totalReduces == completedReduces &&
!inCompletingJob) {
for (TaskId mapTaskId : job.mapTasks) {
MapTaskImpl task = (MapTaskImpl) job.tasks.get(mapTaskId);
if (!task.isFinished() && !task.isPreemptedDueToReducerFinish()) {
LOG.info("Killing map task " + task.getID());
task.setPreemptedDueToReducerFinish(true);
job.eventHandler.handle(
new TaskEvent(task.getID(), TaskEventType.T_KILL));
}
}
inCompletingJob = true;
}
}
{code}
In TestJobImpl, can you explain to me what this code is doing
{code}
// finish mappers - if the mapper is preempted, its state will be
// KILLED, set by KillWaitAttemptKilledTransition
for (TaskId taskId: job.tasks.keySet()) {
TaskState state = killMappers ? TaskState.KILLED : TaskState.SUCCEEDED;
if (taskId.getTaskType() == TaskType.MAP) {
job.handle(new JobTaskEvent(taskId, state));
}
}
{code}
Also, do we expect the job the succeed even when killMappers is set to false?
I'd expect if killMappers is false,
The mappers will stay in RUNNING state, and thus hanging the job. In other
words, the job should stay in
RUNNING state (after we drain events in the dispatcher event queue). No?
> Add configuration for MR job to finish when all reducers are complete (even
> with unfinished mappers)
>
>
> Key: MAPREDUCE-6870
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6870
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Affects Versions: 2.6.1
>Reporter: Zhe Zhang
>Assignee: Peter Bacsko
> Attachments: MAPREDUCE-6870-001.patch, MAPREDUCE-6870-002.patch,
> MAPREDUCE-6870-003.patch
>
>
> Even with MAPREDUCE-5817, there could still be cases where mappers get
> scheduled before all reducers are complete, but those mappers run for long
> time, even after all reducers are complete. This could hurt the performance
> of large MR jobs.
> In some cases, mappers don't have any materialize-able outcome other than
> providing intermediate data to reducers. In that case, the job owner should
> have the config option to finish the job once all reducers are complete.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6927) MR job should only set tracking url if history was successfully written
[
https://issues.apache.org/jira/browse/MAPREDUCE-6927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16111262#comment-16111262
]
Hadoop QA commented on MAPREDUCE-6927:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
9s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
14s{color} | {color:green} hadoop-mapreduce-client-hs in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 39m 51s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | MAPREDUCE-6927 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12880066/MAPREDUCE-6927.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux ff869078caaa 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 4889913 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7039/testReport/ |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs U:
hadoop-mapreduce-project/hadoop-mapreduce-client |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7039/console |
| Powered by | Apache Yetus 0.4.0 http://yetus.apache.org |
This message was automatically generated.
[jira] [Updated] (MAPREDUCE-6927) MR job should only set tracking url if history was successfully written
[ https://issues.apache.org/jira/browse/MAPREDUCE-6927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Badger updated MAPREDUCE-6927: --- Attachment: MAPREDUCE-6927.001.patch > MR job should only set tracking url if history was successfully written > --- > > Key: MAPREDUCE-6927 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6927 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Eric Badger >Assignee: Eric Badger > Attachments: MAPREDUCE-6927.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6927) MR job should only set tracking url if history was successfully written
[ https://issues.apache.org/jira/browse/MAPREDUCE-6927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Badger updated MAPREDUCE-6927: --- Status: Patch Available (was: Open) > MR job should only set tracking url if history was successfully written > --- > > Key: MAPREDUCE-6927 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6927 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Eric Badger >Assignee: Eric Badger > Attachments: MAPREDUCE-6927.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-6892) Issues with the count of failed/killed tasks in the jhist file
[
https://issues.apache.org/jira/browse/MAPREDUCE-6892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110785#comment-16110785
]
Peter Bacsko edited comment on MAPREDUCE-6892 at 8/2/17 12:28 PM:
--
[~haibochen] I have been thinking about this renaming and IMO we're better off
solve it in a separate JIRA.
Some problems:
1. I can add new fields to the Avro schema and keep the old ones, but it adds
to the complexity, because we have to support both pairs
2. We probably also have to check whether "finished" or "succeeded" values are
defined in the jhist file, but not both
3. Looking at the {{toTimelineEvent()}} methods, those are a bit messed up too.
As I mentioned above, it has prop. keys like {{NUM_MAPS}} and
{{FINISHED_MAPS}}, but both contain the same value. So either we rename
{{NUM_MAPS}} to {{SUCCEEDED_MAPS}} and delete {{FINISHED_MAPS}} or {{NUM_MAPS}}
should be succeeded+failed+killed and rename {{FINISHED_MAPS}} to
{{SUCCEEDED_MAPS}}.
So it's a bit more complicated. It looks more beneficial to us if we create a
separate JIRA where we try to consolidate the naming inconsistencies (and
possibly involve other upstream devs) and leave these fields alone in this one.
What do you think?
was (Author: pbacsko):
[~haibochen] I have been thinking about this renaming and IMO we're better off
solve it in a separate JIRA.
Some problems:
1. I can add new fields to the Avro schema and keep the old ones, but it adds
to the complexity, because we have to support both pairs
2. We probably also have to check whether "finished" or "succeeded" values are
defined in the jhist file, but not both
3. Looking at the {{toTimelineEvent()}} methods, those are a bit messed up too.
As I mentioned above, it has prop. keys like {{NUM_MAPS}} and
{{FINISHED_MAPS}}, but both contain the same value. So either we rename
{{NUM_MAPS}} to {{SUCCEEDED_MAPS}} and delete {{FINISHED_MAPS}} or {{NUM_MAPS}}
should be succeeded+failed+killed and rename {{FINISHED_MAPS}} to
{{SUCCEEDED_MAPS}}.
So it's a bit more complicated. It looks more beneficial to us if we create a
separate JIRA where we try to consolidate the naming inconsistencies (and
possibly involve other upstream devs) and leave them alone in this one. What do
you think?
> Issues with the count of failed/killed tasks in the jhist file
> --
>
> Key: MAPREDUCE-6892
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6892
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, jobhistoryserver
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: MAPREDUCE-6892-001.patch, MAPREDUCE-6892-002.PATCH
>
>
> Recently we encountered some issues with the value of failed tasks. After
> parsing the jhist file, {{JobInfo.getFailedMaps()}} returned 0, but actually
> there were failures.
> Another minor thing is that you cannot get the number of killed tasks
> (although this can be calculated).
> The root cause is that {{JobUnsuccessfulCompletionEvent}} contains only the
> successful map/reduce task counts. Number of failed (or killed) tasks are not
> stored.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6892) Issues with the count of failed/killed tasks in the jhist file
[
https://issues.apache.org/jira/browse/MAPREDUCE-6892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110785#comment-16110785
]
Peter Bacsko commented on MAPREDUCE-6892:
-
[~haibochen] I have been thinking about this renaming and IMO we're better off
solve it in a separate JIRA.
Some problems:
1. I can add new fields to the Avro schema and keep the old ones, but it adds
to the complexity, because we have to support both pairs
2. We probably also have to check whether "finished" or "succeeded" values are
defined in the jhist file, but not both
3. Looking at the {{toTimelineEvent()}} methods, those are a bit messed up too.
As I mentioned above, it has prop. keys like {{NUM_MAPS}} and
{{FINISHED_MAPS}}, but both contain the same value. So either we rename
{{NUM_MAPS}} to {{SUCCEEDED_MAPS}} and delete {{FINISHED_MAPS}} or {{NUM_MAPS}}
should be succeeded+failed+killed and rename {{FINISHED_MAPS}} to
{{SUCCEEDED_MAPS}}.
So it's a bit more complicated. It looks more beneficial to us if we create a
separate JIRA where we try to consolidate the naming inconsistencies (and
possibly involve other upstream devs) and leave them alone in this one. What do
you think?
> Issues with the count of failed/killed tasks in the jhist file
> --
>
> Key: MAPREDUCE-6892
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6892
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client, jobhistoryserver
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: MAPREDUCE-6892-001.patch, MAPREDUCE-6892-002.PATCH
>
>
> Recently we encountered some issues with the value of failed tasks. After
> parsing the jhist file, {{JobInfo.getFailedMaps()}} returned 0, but actually
> there were failures.
> Another minor thing is that you cannot get the number of killed tasks
> (although this can be calculated).
> The root cause is that {{JobUnsuccessfulCompletionEvent}} contains only the
> successful map/reduce task counts. Number of failed (or killed) tasks are not
> stored.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6870) Add configuration for MR job to finish when all reducers are complete (even with unfinished mappers)
[
https://issues.apache.org/jira/browse/MAPREDUCE-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110695#comment-16110695
]
Hadoop QA commented on MAPREDUCE-6870:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
54s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
45s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
6s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 41m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | MAPREDUCE-6870 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12880006/MAPREDUCE-6870-003.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle xml |
| uname | Linux 9d9cb5dc9e02 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7
11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / f9139ac |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7038/testReport/ |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U:
hadoop-mapreduce-project/hadoop-mapreduce-client |
| Console output |
https://builds.apache
[jira] [Commented] (MAPREDUCE-6870) Add configuration for MR job to finish when all reducers are complete (even with unfinished mappers)
[
https://issues.apache.org/jira/browse/MAPREDUCE-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110667#comment-16110667
]
Peter Bacsko commented on MAPREDUCE-6870:
-
Thanks [~haibochen], I addressed your comments.
Things to check / consider:
1. Variable names (is {{preemptMappersOnReduceFinish}} good?)
2. Added a new method to {{MapTaskImpl}} with locking, which is probably not
necessary but I felt it's better to have it anyway
> Add configuration for MR job to finish when all reducers are complete (even
> with unfinished mappers)
>
>
> Key: MAPREDUCE-6870
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6870
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Affects Versions: 2.6.1
>Reporter: Zhe Zhang
>Assignee: Peter Bacsko
> Attachments: MAPREDUCE-6870-001.patch, MAPREDUCE-6870-002.patch,
> MAPREDUCE-6870-003.patch
>
>
> Even with MAPREDUCE-5817, there could still be cases where mappers get
> scheduled before all reducers are complete, but those mappers run for long
> time, even after all reducers are complete. This could hurt the performance
> of large MR jobs.
> In some cases, mappers don't have any materialize-able outcome other than
> providing intermediate data to reducers. In that case, the job owner should
> have the config option to finish the job once all reducers are complete.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6870) Add configuration for MR job to finish when all reducers are complete (even with unfinished mappers)
[ https://issues.apache.org/jira/browse/MAPREDUCE-6870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated MAPREDUCE-6870: Attachment: MAPREDUCE-6870-003.patch > Add configuration for MR job to finish when all reducers are complete (even > with unfinished mappers) > > > Key: MAPREDUCE-6870 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6870 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 2.6.1 >Reporter: Zhe Zhang >Assignee: Peter Bacsko > Attachments: MAPREDUCE-6870-001.patch, MAPREDUCE-6870-002.patch, > MAPREDUCE-6870-003.patch > > > Even with MAPREDUCE-5817, there could still be cases where mappers get > scheduled before all reducers are complete, but those mappers run for long > time, even after all reducers are complete. This could hurt the performance > of large MR jobs. > In some cases, mappers don't have any materialize-able outcome other than > providing intermediate data to reducers. In that case, the job owner should > have the config option to finish the job once all reducers are complete. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
