[jira] [Commented] (MAPREDUCE-6953) Skip the testcase testJobWithChangePriority if FairScheduler is used
[
https://issues.apache.org/jira/browse/MAPREDUCE-6953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16159259#comment-16159259
]
Hudson commented on MAPREDUCE-6953:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12826 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/12826/])
MAPREDUCE-6953. Skip the testcase testJobWithChangePriority if (rkanter: rev
a323f73bae65483652de290ef819268850803a17)
* (edit)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/v2/TestMRJobs.java
> Skip the testcase testJobWithChangePriority if FairScheduler is used
>
>
> Key: MAPREDUCE-6953
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6953
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: client
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Fix For: 2.9.0, 3.0.0, 3.1.0
>
> Attachments: MAPREDUCE-6953-001.patch
>
>
> We run the unit tests with Fair Scheduler downstream. FS does not support
> priorities at the moment, so TestMRJobs#testJobWithChangePriority fails.
> Just add {{Assume.assumeFalse(usingFairScheduler);}} and JUnit will skip the
> test.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6953) Skip the testcase testJobWithChangePriority if FairScheduler is used
[
https://issues.apache.org/jira/browse/MAPREDUCE-6953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Kanter updated MAPREDUCE-6953:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.1.0
3.0.0
2.9.0
Status: Resolved (was: Patch Available)
Thanks [~pbacsko]. Committed to trunk, branch-3.0, and branch-2!
> Skip the testcase testJobWithChangePriority if FairScheduler is used
>
>
> Key: MAPREDUCE-6953
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6953
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: client
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Fix For: 2.9.0, 3.0.0, 3.1.0
>
> Attachments: MAPREDUCE-6953-001.patch
>
>
> We run the unit tests with Fair Scheduler downstream. FS does not support
> priorities at the moment, so TestMRJobs#testJobWithChangePriority fails.
> Just add {{Assume.assumeFalse(usingFairScheduler);}} and JUnit will skip the
> test.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6956) FileOutputCommitter to gain abstract superclass PathOutputCommitter
[
https://issues.apache.org/jira/browse/MAPREDUCE-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16159200#comment-16159200
]
Hadoop QA commented on MAPREDUCE-6956:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
39s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 24s{color}
| {color:red}
hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core
generated 3 new + 162 unchanged - 0 fixed = 165 total (was 162) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
17s{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:
The patch generated 8 new + 94 unchanged - 12 fixed = 102 total (was 106)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
39s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 2s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | MAPREDUCE-6956 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886146/MAPREDUCE-6956-001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux e94ff8f0656f 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / c35510a |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| javac |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7125/artifact/patchprocess/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7125/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7125/testReport/ |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7125/console |
| Powered by | Apache Yetus
[jira] [Commented] (MAPREDUCE-6953) Skip the testcase testJobWithChangePriority if FairScheduler is used
[
https://issues.apache.org/jira/browse/MAPREDUCE-6953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16159163#comment-16159163
]
Robert Kanter commented on MAPREDUCE-6953:
--
+1
> Skip the testcase testJobWithChangePriority if FairScheduler is used
>
>
> Key: MAPREDUCE-6953
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6953
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: client
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: MAPREDUCE-6953-001.patch
>
>
> We run the unit tests with Fair Scheduler downstream. FS does not support
> priorities at the moment, so TestMRJobs#testJobWithChangePriority fails.
> Just add {{Assume.assumeFalse(usingFairScheduler);}} and JUnit will skip the
> test.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6956) FileOutputCommitter to gain abstract superclass PathOutputCommitter
[
https://issues.apache.org/jira/browse/MAPREDUCE-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated MAPREDUCE-6956:
--
Priority: Minor (was: Major)
> FileOutputCommitter to gain abstract superclass PathOutputCommitter
> ---
>
> Key: MAPREDUCE-6956
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6956
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
> Attachments: MAPREDUCE-6956-001.patch
>
>
> This is the initial step of MAPREDUCE-6823, which proposes a factory behind
> {{FileOutputFormat}} to create different committers for different
> filesystems, if so configured..
> This patch simply adds the new abstract superclass of
> {{FileOutputCommitter}}, {{PathOutputCommitter extends OutputCommitter}}.
> This abstract class adds the {{getWorkPath()}} method as an abstract method,
> with {{FIleOutputCommitter}} being the implementation..
> {{FileOutputFormat}} then relaxes its requirement of any committer returned
> by {{getOutputCommitter()}}, so that instead of requiring a
> {{FileOutputCommitter}} or subclass, it only needs a {{PathOutputCommitter}},
> using {{PathOutputCommitter.getWorkPath()}} to get the work path.
> What does that do?
> It allows people to implement subclasses of {{FileOutputFormat}} which can
> provide their own committers *which don't need to inherit the complexity that
> FileOutputCommitter has acquired over time*
> Currently anyone implementing a new committer (example: Netflix S3 committer)
> needs to subclass {{FileOutputCommitter}}, which is too complex to understand
> except under a debugger with co-recursive routines, lots of methods which
> need to be overwritten to guarantee a safe subclass, and, because of its
> critical role and known subclassing, something which isn't ever going to be
> cleaned up.
> A new, lean, parent class which {{FileOutputFormat}} can handle allows people
> to write new committers which don't have to worry about implementation
> details of {{FileOutputCommitter}}, but instead how well they implement the
> semantics of committing work.
> The full MAPREDUCE-6823 goes beyond this with a change to
> {{FileOutputFormat}} for a factory for creating FS-specific
> {{PathOutputCommitter}} instances. This patch doesn't include that, as that
> is something which needs to be reviewed in the context of HADOOP-13786 and
> ideally 1+ committer for another store, so people can say "this factory model
> works".
> All I'm proposing here is: tune the committer class hierarchy in MRv2 so that
> people can more easily implement committers, and when that factory is done,
> for it to be switched to easily. And I'd like this in branch-3 from the
> outset, so existing code which calls {{FileOutputFormat.getCommitter()}} to
> get a {{FileOutputCommitter}} *just to call getWorkPath()* can move to the
> new interface across all of Hadoop 3.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6956) FileOutputCommitter to gain abstract superclass PathOutputCommitter
[
https://issues.apache.org/jira/browse/MAPREDUCE-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated MAPREDUCE-6956:
--
Status: Patch Available (was: Open)
> FileOutputCommitter to gain abstract superclass PathOutputCommitter
> ---
>
> Key: MAPREDUCE-6956
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6956
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: MAPREDUCE-6956-001.patch
>
>
> This is the initial step of MAPREDUCE-6823, which proposes a factory behind
> {{FileOutputFormat}} to create different committers for different
> filesystems, if so configured..
> This patch simply adds the new abstract superclass of
> {{FileOutputCommitter}}, {{PathOutputCommitter extends OutputCommitter}}.
> This abstract class adds the {{getWorkPath()}} method as an abstract method,
> with {{FIleOutputCommitter}} being the implementation..
> {{FileOutputFormat}} then relaxes its requirement of any committer returned
> by {{getOutputCommitter()}}, so that instead of requiring a
> {{FileOutputCommitter}} or subclass, it only needs a {{PathOutputCommitter}},
> using {{PathOutputCommitter.getWorkPath()}} to get the work path.
> What does that do?
> It allows people to implement subclasses of {{FileOutputFormat}} which can
> provide their own committers *which don't need to inherit the complexity that
> FileOutputCommitter has acquired over time*
> Currently anyone implementing a new committer (example: Netflix S3 committer)
> needs to subclass {{FileOutputCommitter}}, which is too complex to understand
> except under a debugger with co-recursive routines, lots of methods which
> need to be overwritten to guarantee a safe subclass, and, because of its
> critical role and known subclassing, something which isn't ever going to be
> cleaned up.
> A new, lean, parent class which {{FileOutputFormat}} can handle allows people
> to write new committers which don't have to worry about implementation
> details of {{FileOutputCommitter}}, but instead how well they implement the
> semantics of committing work.
> The full MAPREDUCE-6823 goes beyond this with a change to
> {{FileOutputFormat}} for a factory for creating FS-specific
> {{PathOutputCommitter}} instances. This patch doesn't include that, as that
> is something which needs to be reviewed in the context of HADOOP-13786 and
> ideally 1+ committer for another store, so people can say "this factory model
> works".
> All I'm proposing here is: tune the committer class hierarchy in MRv2 so that
> people can more easily implement committers, and when that factory is done,
> for it to be switched to easily. And I'd like this in branch-3 from the
> outset, so existing code which calls {{FileOutputFormat.getCommitter()}} to
> get a {{FileOutputCommitter}} *just to call getWorkPath()* can move to the
> new interface across all of Hadoop 3.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6956) FileOutputCommitter to gain abstract superclass PathOutputCommitter
[
https://issues.apache.org/jira/browse/MAPREDUCE-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated MAPREDUCE-6956:
--
Attachment: MAPREDUCE-6956-001.patch
Patch 001
This is the code from MAPREDUC-6823 excluding the factory itself and adding a
unit test of assignment and resolution.
As well as the new interface,
* it adds toString() values of FileOutputCommitter, JobContextImpl,
TaskAttemptContextImpl. Invaluable during debugging sessions, logs and
assertions.
* FileOutputCommitter adds an explicit check for task attempts having a null
task attempt path.
* makes changes to FileOutputFormat so as to stop checkstyle complaining (tabs,
some line endings, javadocs).
I've been using this code in HADOOP-13768 for a while, this is just the core
visible change which simplifes the work for anyone trying to implement
committers. FileOutputFormat and downstream (e.g.
{{org.apache.spark.internal.io.HadoopMapReduceCommitProtocol}} are casting to
FileOutputCommitter when they need a workpath, and they benefit from the
ability to use something a bit more abstract
> FileOutputCommitter to gain abstract superclass PathOutputCommitter
> ---
>
> Key: MAPREDUCE-6956
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6956
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Affects Versions: 3.0.0-beta1
>Reporter: Steve Loughran
>Assignee: Steve Loughran
> Attachments: MAPREDUCE-6956-001.patch
>
>
> This is the initial step of MAPREDUCE-6823, which proposes a factory behind
> {{FileOutputFormat}} to create different committers for different
> filesystems, if so configured..
> This patch simply adds the new abstract superclass of
> {{FileOutputCommitter}}, {{PathOutputCommitter extends OutputCommitter}}.
> This abstract class adds the {{getWorkPath()}} method as an abstract method,
> with {{FIleOutputCommitter}} being the implementation..
> {{FileOutputFormat}} then relaxes its requirement of any committer returned
> by {{getOutputCommitter()}}, so that instead of requiring a
> {{FileOutputCommitter}} or subclass, it only needs a {{PathOutputCommitter}},
> using {{PathOutputCommitter.getWorkPath()}} to get the work path.
> What does that do?
> It allows people to implement subclasses of {{FileOutputFormat}} which can
> provide their own committers *which don't need to inherit the complexity that
> FileOutputCommitter has acquired over time*
> Currently anyone implementing a new committer (example: Netflix S3 committer)
> needs to subclass {{FileOutputCommitter}}, which is too complex to understand
> except under a debugger with co-recursive routines, lots of methods which
> need to be overwritten to guarantee a safe subclass, and, because of its
> critical role and known subclassing, something which isn't ever going to be
> cleaned up.
> A new, lean, parent class which {{FileOutputFormat}} can handle allows people
> to write new committers which don't have to worry about implementation
> details of {{FileOutputCommitter}}, but instead how well they implement the
> semantics of committing work.
> The full MAPREDUCE-6823 goes beyond this with a change to
> {{FileOutputFormat}} for a factory for creating FS-specific
> {{PathOutputCommitter}} instances. This patch doesn't include that, as that
> is something which needs to be reviewed in the context of HADOOP-13786 and
> ideally 1+ committer for another store, so people can say "this factory model
> works".
> All I'm proposing here is: tune the committer class hierarchy in MRv2 so that
> people can more easily implement committers, and when that factory is done,
> for it to be switched to easily. And I'd like this in branch-3 from the
> outset, so existing code which calls {{FileOutputFormat.getCommitter()}} to
> get a {{FileOutputCommitter}} *just to call getWorkPath()* can move to the
> new interface across all of Hadoop 3.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6750) TestHSAdminServer.testRefreshSuperUserGroups is failing
[
https://issues.apache.org/jira/browse/MAPREDUCE-6750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko updated MAPREDUCE-6750:
---
Fix Version/s: 2.7.5
> TestHSAdminServer.testRefreshSuperUserGroups is failing
> ---
>
> Key: MAPREDUCE-6750
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6750
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Kihwal Lee
>Assignee: Kihwal Lee
>Priority: Minor
> Fix For: 2.8.0, 3.0.0-alpha1, 2.7.5
>
> Attachments: MAPREDUCE-6750.patch
>
>
> HADOOP-13442 changed {{AccessControlList}} to call {{getGroups()}} instead of
> {{getGroupNames()}}. It should work if the mocks are updated to stub the
> right method and return the right type.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6956) FileOutputCommitter to gain abstract superclass PathOutputCommitter
Steve Loughran created MAPREDUCE-6956:
-
Summary: FileOutputCommitter to gain abstract superclass
PathOutputCommitter
Key: MAPREDUCE-6956
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6956
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Affects Versions: 3.0.0-beta1
Reporter: Steve Loughran
Assignee: Steve Loughran
This is the initial step of MAPREDUCE-6823, which proposes a factory behind
{{FileOutputFormat}} to create different committers for different filesystems,
if so configured..
This patch simply adds the new abstract superclass of {{FileOutputCommitter}},
{{PathOutputCommitter extends OutputCommitter}}. This abstract class adds the
{{getWorkPath()}} method as an abstract method, with {{FIleOutputCommitter}}
being the implementation..
{{FileOutputFormat}} then relaxes its requirement of any committer returned by
{{getOutputCommitter()}}, so that instead of requiring a
{{FileOutputCommitter}} or subclass, it only needs a {{PathOutputCommitter}},
using {{PathOutputCommitter.getWorkPath()}} to get the work path.
What does that do?
It allows people to implement subclasses of {{FileOutputFormat}} which can
provide their own committers *which don't need to inherit the complexity that
FileOutputCommitter has acquired over time*
Currently anyone implementing a new committer (example: Netflix S3 committer)
needs to subclass {{FileOutputCommitter}}, which is too complex to understand
except under a debugger with co-recursive routines, lots of methods which need
to be overwritten to guarantee a safe subclass, and, because of its critical
role and known subclassing, something which isn't ever going to be cleaned up.
A new, lean, parent class which {{FileOutputFormat}} can handle allows people
to write new committers which don't have to worry about implementation details
of {{FileOutputCommitter}}, but instead how well they implement the semantics
of committing work.
The full MAPREDUCE-6823 goes beyond this with a change to {{FileOutputFormat}}
for a factory for creating FS-specific {{PathOutputCommitter}} instances. This
patch doesn't include that, as that is something which needs to be reviewed in
the context of HADOOP-13786 and ideally 1+ committer for another store, so
people can say "this factory model works".
All I'm proposing here is: tune the committer class hierarchy in MRv2 so that
people can more easily implement committers, and when that factory is done, for
it to be switched to easily. And I'd like this in branch-3 from the outset, so
existing code which calls {{FileOutputFormat.getCommitter()}} to get a
{{FileOutputCommitter}} *just to call getWorkPath()* can move to the new
interface across all of Hadoop 3.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Resolved] (MAPREDUCE-6955) remove unnecessary dependency from hadoop-mapreduce-client-app to hadoop-mapreduce-client-shuffle
[ https://issues.apache.org/jira/browse/MAPREDUCE-6955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen resolved MAPREDUCE-6955. --- Resolution: Not A Problem > remove unnecessary dependency from hadoop-mapreduce-client-app to > hadoop-mapreduce-client-shuffle > - > > Key: MAPREDUCE-6955 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6955 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Haibo Chen >Assignee: Haibo Chen > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6955) remove unnecessary dependency from hadoop-mapreduce-client-app to hadoop-mapreduce-client-shuffle
Haibo Chen created MAPREDUCE-6955: - Summary: remove unnecessary dependency from hadoop-mapreduce-client-app to hadoop-mapreduce-client-shuffle Key: MAPREDUCE-6955 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6955 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Haibo Chen Assignee: Haibo Chen -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6953) Skip the testcase testJobWithChangePriority if FairScheduler is used
[
https://issues.apache.org/jira/browse/MAPREDUCE-6953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158729#comment-16158729
]
Hadoop QA commented on MAPREDUCE-6953:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}115m 34s{color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}140m 28s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.mapreduce.TestMRJobClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | MAPREDUCE-6953 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886058/MAPREDUCE-6953-001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux a879390eb7cc 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 5bbca80 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7123/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7123/testReport/ |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
|
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7123/console |
| Powered by | Apache Yetus 0.5.0 http://yetus.apache.org |
This message was automatically generated.
> Skip the testcase testJobWithChangePriority if FairScheduler is used
>
>
> Key: MAPREDUCE-6953
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6953
> Project:
[jira] [Commented] (MAPREDUCE-6954) Disable erasure coding for files that are uploaded to the MR staging area
[
https://issues.apache.org/jira/browse/MAPREDUCE-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158646#comment-16158646
]
Hadoop QA commented on MAPREDUCE-6954:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
21s{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:
The patch generated 2 new + 530 unchanged - 0 fixed = 532 total (was 530)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
55s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | MAPREDUCE-6954 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886071/MAPREDUCE-6954-001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit xml findbugs checkstyle |
| uname | Linux 122d57defd22 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 5bbca80 |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7124/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7124/testReport/ |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7124/console |
| Powered by | Apache Yetus 0.5.0 http://yetus.apache.org |
[jira] [Updated] (MAPREDUCE-6954) Disable erasure coding for files that are uploaded to the MR staging area
[ https://issues.apache.org/jira/browse/MAPREDUCE-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated MAPREDUCE-6954: Status: Patch Available (was: Open) > Disable erasure coding for files that are uploaded to the MR staging area > - > > Key: MAPREDUCE-6954 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6954 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Affects Versions: 3.0.0-alpha4 >Reporter: Peter Bacsko >Assignee: Peter Bacsko > Attachments: MAPREDUCE-6954-001.patch > > > Depending on the encoder/decoder used and the type or MR workload, EC might > negatively affect the performance of an MR job if too many files are > localized. > In such a scenario, users might want to disable EC in the staging area to > speed up the execution. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6954) Disable erasure coding for files that are uploaded to the MR staging area
[
https://issues.apache.org/jira/browse/MAPREDUCE-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158602#comment-16158602
]
Peter Bacsko commented on MAPREDUCE-6954:
-
Forgot to update {{mapred-default.xml}}, will do it in the next patch.
> Disable erasure coding for files that are uploaded to the MR staging area
> -
>
> Key: MAPREDUCE-6954
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6954
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client
>Affects Versions: 3.0.0-alpha4
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: MAPREDUCE-6954-001.patch
>
>
> Depending on the encoder/decoder used and the type or MR workload, EC might
> negatively affect the performance of an MR job if too many files are
> localized.
> In such a scenario, users might want to disable EC in the staging area to
> speed up the execution.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6954) Disable erasure coding for files that are uploaded to the MR staging area
[ https://issues.apache.org/jira/browse/MAPREDUCE-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated MAPREDUCE-6954: Attachment: MAPREDUCE-6954-001.patch > Disable erasure coding for files that are uploaded to the MR staging area > - > > Key: MAPREDUCE-6954 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6954 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Affects Versions: 3.0.0-alpha4 >Reporter: Peter Bacsko >Assignee: Peter Bacsko > Attachments: MAPREDUCE-6954-001.patch > > > Depending on the encoder/decoder used and the type or MR workload, EC might > negatively affect the performance of an MR job if too many files are > localized. > In such a scenario, users might want to disable EC in the staging area to > speed up the execution. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6954) Disable erasure coding for files that are uploaded to the MR staging area
[ https://issues.apache.org/jira/browse/MAPREDUCE-6954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated MAPREDUCE-6954: Affects Version/s: 3.0.0-alpha4 > Disable erasure coding for files that are uploaded to the MR staging area > - > > Key: MAPREDUCE-6954 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6954 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Affects Versions: 3.0.0-alpha4 >Reporter: Peter Bacsko >Assignee: Peter Bacsko > Attachments: MAPREDUCE-6954-001.patch > > > Depending on the encoder/decoder used and the type or MR workload, EC might > negatively affect the performance of an MR job if too many files are > localized. > In such a scenario, users might want to disable EC in the staging area to > speed up the execution. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6954) Disable erasure coding for files that are uploaded to the MR staging area
Peter Bacsko created MAPREDUCE-6954: --- Summary: Disable erasure coding for files that are uploaded to the MR staging area Key: MAPREDUCE-6954 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6954 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Reporter: Peter Bacsko Assignee: Peter Bacsko Depending on the encoder/decoder used and the type or MR workload, EC might negatively affect the performance of an MR job if too many files are localized. In such a scenario, users might want to disable EC in the staging area to speed up the execution. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6953) Skip the testcase testJobWithChangePriority if FairScheduler is used
[
https://issues.apache.org/jira/browse/MAPREDUCE-6953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158523#comment-16158523
]
Peter Bacsko commented on MAPREDUCE-6953:
-
[~rkanter] please review.
> Skip the testcase testJobWithChangePriority if FairScheduler is used
>
>
> Key: MAPREDUCE-6953
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6953
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: client
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: MAPREDUCE-6953-001.patch
>
>
> We run the unit tests with Fair Scheduler downstream. FS does not support
> priorities at the moment, so TestMRJobs#testJobWithChangePriority fails.
> Just add {{Assume.assumeFalse(usingFairScheduler);}} and JUnit will skip the
> test.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6953) Skip the testcase testJobWithChangePriority if FairScheduler is used
[
https://issues.apache.org/jira/browse/MAPREDUCE-6953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated MAPREDUCE-6953:
Status: Patch Available (was: Open)
> Skip the testcase testJobWithChangePriority if FairScheduler is used
>
>
> Key: MAPREDUCE-6953
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6953
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: client
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: MAPREDUCE-6953-001.patch
>
>
> We run the unit tests with Fair Scheduler downstream. FS does not support
> priorities at the moment, so TestMRJobs#testJobWithChangePriority fails.
> Just add {{Assume.assumeFalse(usingFairScheduler);}} and JUnit will skip the
> test.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6953) Skip the testcase testJobWithChangePriority if FairScheduler is used
[
https://issues.apache.org/jira/browse/MAPREDUCE-6953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated MAPREDUCE-6953:
Attachment: MAPREDUCE-6953-001.patch
> Skip the testcase testJobWithChangePriority if FairScheduler is used
>
>
> Key: MAPREDUCE-6953
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6953
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: client
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
> Attachments: MAPREDUCE-6953-001.patch
>
>
> We run the unit tests with Fair Scheduler downstream. FS does not support
> priorities at the moment, so TestMRJobs#testJobWithChangePriority fails.
> Just add {{Assume.assumeFalse(usingFairScheduler);}} and JUnit will skip the
> test.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6953) Skip the testcase testJobWithChangePriority if FairScheduler is used
Peter Bacsko created MAPREDUCE-6953:
---
Summary: Skip the testcase testJobWithChangePriority if
FairScheduler is used
Key: MAPREDUCE-6953
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6953
Project: Hadoop Map/Reduce
Issue Type: Test
Components: client
Reporter: Peter Bacsko
Assignee: Peter Bacsko
We run the unit tests with Fair Scheduler downstream. FS does not support
priorities at the moment, so TestMRJobs#testJobWithChangePriority fails.
Just add {{Assume.assumeFalse(usingFairScheduler);}} and JUnit will skip the
test.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
